8.4 The posture ladder

Included from Basic tier

After this you can: climb the posture ladder — dispatch, node, site and system quiesce, the emergency halt — and read how each rung audits.

Before you start — tick each one:

• Lesson 5.6 — Maintenance windows (opschain learn 5.6)

In the GUI: the Overview's emergency controls in the Operations area show the halt postures with their reasons and lift them (reason required); the System quiesce panel on the Server and Database pages starts the estate-wide quiesce (reason required), aborts a drain and resumes it; the Runners panel's backlog names the posture on every parked entry.

OpsChain has a ladder of ways to stop work, each with a deliberately different scope and blast radius. Reaching for the right rung is the difference between a quiet maintenance window and a needless outage — and between containing an incident and watching it keep executing. This lesson climbs the ladder, lightest to heaviest: what each posture stops, what keeps running, how each is resumed, and how every posture change lands in the audit trail. The component reference is opschain docs platform-halt.

1. The ladder at a glance

Posture	Verb	Scope	What it stops	What keeps running
Dispatch quiesce	opschain admin server quiesce	New job offers, estate-wide	New offers to runners	Everything else: API writes, in-flight steps, triggers (their work queues)
Node quiesce	opschain admin server node quiesce <name>	One server node	That node serving anything	The rest of the fleet, seamlessly
Site quiesce	opschain admin server site quiesce <site>	Every server node of one site	That site's servers	The other site carries the whole estate
System quiesce	opschain admin platform quiesce --reason …	The whole estate	Every write, after draining running work	Reads, and existing sessions
Halt	opschain admin platform halt --plane … --reason …	A plane: runners, agents or both	Everything on the plane, immediately — in-flight steps included	Reads, writes, the other plane

The first three are routine maintenance instruments. The system quiesce is the ceremony for backups, exports and database windows. The halt is the emergency brake — break-glass, reason-required, audited as such.

Maintenance windows are a different instrument again: they pause automation (trigger-driven work) for part of the estate on a schedule, while operators keep working normally — that is lesson 5.6, and nothing on this ladder replaces it.

2. Dispatch quiesce — hold the offers

opschain admin server quiesce holds the dispatch plane: no new offers to runners, queued work stays queued, runners stay parked, existing leases untouched — in-flight steps run to their natural end. It is the gentle instrument for "nothing new starts while we watch something".

opschain admin server quiesce

{
  "quiesced": true
}

Expect the queue to grow while it holds — every parked entry reads dispatch is quiesced … as its reason (lesson 6.1) — and opschain admin server resume releases it. The posture is a durable row every replica honours, so a restart changes nothing.

3. Node and site quiesce — maintenance without a window

The node and site verbs drain servers, not the estate: the named node (or every node of the named site) leaves readiness, stops offering work, hands its scheduler duties back and detaches its in-flight runs for peers to adopt — then is safe to kill, kill -9 included. The workload never notices: peers drive the runs, and the site's runners fail over on their ordered server list and keep working through the surviving site.

opschain admin server node quiesce node-a

server node node-a quiesced: it drains, boots held after a restart, and takes no work until 'opschain admin server node resume node-a'

Two semantics worth internalising:

• Held-at-boot. The posture is durable, so a binary booting under it comes up held — healthy and inspectable, taking no work — and a name that is not currently serving can be quiesced in advance to pre-hold its next boot.
• Human-gated resume. Nothing lifts a hold but the explicit verb — opschain admin server node resume <name> / opschain admin server site resume <site>. No timer, no sweep, no restart.

opschain admin server node list and … site list show what is held. These are the building blocks of rolling upgrades and site maintenance — lessons 8.6 and 8.7.

4. System quiesce — the read-only estate

opschain admin platform quiesce --reason "monthly backup" takes the whole estate to a verified standstill, in two honest phases:

opschain admin platform quiesce --reason "monthly backup"

draining: 2 attempt(s) still running
draining: 1 attempt(s) still running
system quiesced: the write barrier is closed — resume with 'opschain admin platform resume'

1. Drain first. The command reports the attempts still running and waits for them to complete — never cancels or suspends them. Other writes still succeed while draining. Bound the wait with --timeout: on expiry the attempt aborts cleanly back to fully active — never a half-state — and a warn:platform:quiesce_aborted event records what was still running.
2. The write barrier closes. Every mutating request — change creation, settings edits, GUI actions, new credential logins — is refused with a named 503 carrying your --reason. Reads with an existing session still answer; token refresh still works, so logged-in operators are not locked out; /readyz stays green and reports the quiesced phase. Triggers, retention and database maintenance idle.

The posture is durable: a replica restarted mid-quiesce boots straight back into the closed barrier. There is no auto-resume — the estate stays read-only until opschain admin platform resume, so never quiesce a shared estate and walk away. This is the state the backup manifest (lesson 8.1) and the estate export (lesson 8.2) require.

5. Halt — the emergency brake

opschain admin platform halt is for incidents, not maintenance:

opschain admin platform halt --plane runners --reason "incident 4821"

{
  "postures": [
    {"plane": "runners", "id": "01999b2c-…", "reason": "incident 4821", "halted_by": "admin", "halted_at": "2026-10-15T11:02:40Z"}
  ]
}

Immediately: dispatch freezes, queued work parks naming the posture, leases are held — a halt never ages a step into runner_lost, never fires a retry, however long it lasts — and in-flight steps are suspended in place (process-stopped, container-paused), their memory and disk state left intact for forensics. --in-flight complete is the softer variant that lets running steps finish while dispatching nothing new. --plane agents severs and refuses every agent surface instead; --plane both is full lockdown.

While a halt holds you can dispose of suspended attempts individually — opschain admin platform attempt <run-id> <step-id> --disposition resume|kill — continuing one in place, or cancelling it cleanly (the run finalises cancelled, never a kill signal into frozen state), with the halt still in force for everything else. Narrower scalpels exist for the agent plane: opschain admin agent halt <agent-id> --reason … and opschain admin agent halt-resume <agent-id> for one agent — reversible, unlike revocation (lesson 12.5).

Resume is deliberately harder than halt. A halt is lifted per plane, naming the posture id it created — a stale or wrong id is refused with the halt left in force:

opschain admin platform resume --plane runners --posture 01999b2c-…

Re-halting an active plane is idempotent (the same posture, never a toggle), and rapid halt–resume cycling rate-limits further resumes while halt itself always stays available: under pressure, the platform prefers staying stopped. Suspended steps continue exactly where they stopped — no retry, no lost output. Where agents are entitled to operate platform verbs at all, an agent may be trusted to impose a halt autonomously, but a halt can never be agent-lifted: resume is a human verb, and an agent's attempt is refused and recorded (lesson 12.1).

6. Orthogonality — postures compose

Each rung is independent and they compose predictably: a halt can be imposed and lifted while a system quiesce stays in force (the halt family is carved out of the write barrier precisely so the brake works on a quiesced estate); a node quiesce during a site quiesce holds under both; a system quiesce entered during a dispatch quiesce drains what was already running. Resuming one posture never lifts another — opschain admin status and opschain admin platform status always show what remains in force.

7. How posture changes audit

Every entry and exit is an audit event, with the actor and the reason, searchable in the event feed (opschain events, or the Audit history in the GUI):

• info:server:node_quiesced / info:server:node_resumed — node verbs.
• info:server:site_quiesced / info:server:site_resumed — site verbs.
• api:platform:quiesced / api:platform:resumed — the system quiesce barrier closing and reopening; warn:platform:quiesce_aborted for a timed-out drain.
• warn:platform:halted (actor, plane, reason) / info:platform:resumed — the halt bracket; per-attempt dispositions and rate-limited resumes are evented too.

A reason is required on the system quiesce and the halt — it is shown to every refused caller and recorded on the event, so the estate always knows why it is stopped and who stopped it.

Check yourself

• Which rung stops new dispatch but lets running steps finish, and which one is the only one that suspends work in place?
• What must a resume of a halted plane name, and what happens when it names the wrong one?

What you learned

• Five rungs, lightest to heaviest: dispatch quiesce, node, site, system quiesce, halt — each with its own scope.
• Node and site holds are durable and human-gated; a rolled binary boots held.
• The system quiesce drains first, then closes the write barrier, and never resumes itself.
• The halt freezes a plane and preserves state for forensics; its resume names the posture id and is rate-limited.
• Every rung audits its entry and exit with actor and reason.

Next: lesson 8.5 — Database care (opschain learn 8.5), or back to your track (opschain learn).