Platform postures

Included from Basic tier

OpsChain has three ways to stop work, with deliberately different scope, durability and blast radius. From lightest to heaviest:

Posture	Scope	Durability	What it stops	Enter / exit
Replica dispatch quiesce	One replica's offers	In-memory — a restart clears it	New job offers from that replica; leases, queued work and runners stay put	POST /api/fleet/quiesce / POST /api/fleet/resume (opschain admin server quiesce|resume); POST /api/fleet/drain is the heavier sibling that also directs in-flight runners to finish and stop acquiring
System quiesce	The entire estate — API, schedulers, triggers, runner and agent planes, GUI	Durable posture row — survives restarts and failovers	Drains running work first, then holds the whole system read-only	POST /api/platform/quiesce (opschain admin platform quiesce) / bare opschain admin platform resume
Emergency halt	A plane — runners, agents or both	Durable posture row — survives restarts and failovers	Everything on the plane, now: dispatch freezes, leases are held, in-flight runner steps suspend in place	POST /api/platform/halt (opschain admin platform halt) / per-plane, posture-named opschain admin platform resume --plane … --posture …

All three are admin verbs (admin bearer token). The halt and resume verbs are additionally reachable over the agent API lane only under an AUTONOMOUS trust grant for the verb; a SUPERVISED grant is a named refusal recorded as warn:platform:agent_verb_escalation — a platform verb cannot wait on an approval gate, so a human operator performs it. See opschain docs agent-plane.

Emergency halt (break-glass)

POST /api/platform/halt flips a durable per-plane posture row in fleet_postures — LOGGED, read by every replica's offer, proxy and request path, surviving restarts and failovers. A reason is mandatory (a reason is required: the halt verb is break-glass audited); both is the full-lockdown composition of the two planes. The halt is idempotent while active — a repeat never toggles, it can only widen the plane set — and state is deliberately preserved for forensics (a kill mode was rejected by design: cancel the run when destruction is the intent).

While a runner-plane halt is in force:

• Acquire polls answer empty — nothing dispatches anywhere, and queued entries on the fleet queue read runner plane halted (posture <id>): <reason>.
• Lease expiry is held: the lost-runner sweep is skipped entirely, so nothing ages into runner_lost during the halt, and a lease whose TTL lapsed mid-halt renews by token match alone. The first sweep after resume reconciles any runner that genuinely died during the halt.
• In-flight steps follow the halt's in-flight mode: suspend (the default — suspend in place, preserving forensic state) or complete (run to completion). Suspended attempts are then disposed one by one with opschain admin platform attempt (POST /api/platform/halt/attempts): resume a clean one in place while the lockdown holds, or kill a compromised one. Dispositions outside a halt are refused: no runner-plane halt is active: dispositions apply to suspended attempts during a halt (cancel the change otherwise).

An agent-plane halt severs every agent channel and refuses every agent credential, including service-account calls over plain HTTPS (agent plane halted (posture <id>): <reason>). Each channel is closed WITH that reason (typed AGENT_PLANE_HALTED, carrying the posture id), so every connected agent logs one line naming the posture and reason the moment the halt lands, waits with its identity kept, and logs platform: channel restored after <duration> once the plane resumes — see opschain docs agent-plane ("What the agent sees").

Resume is deliberately narrow: per-plane, and it must name the active posture row (posture_id is required: resume must name the active posture row (a stale resume is refused)), so a resume drafted against a posture that has since been replaced fails (the named posture row is not the active halt for this plane) instead of silently reopening a newer halt. Halt↔resume cycling by one subject — 3 resumes inside 10 minutes — trips a rate limit (halt/resume cycling detected for this subject: further resumes are rate-limited (halt stays available)), read from the durable posture history so it holds across replicas and restarts.

GET /api/platform/posture (opschain admin platform status) is the loud state surface: each plane's active posture (null when normal), the recent history, and the system-quiesce posture and phase. Resume stamps resumed_by/resumed_at in place, so the full history stays queryable. The maintenance planes that share the posture table — the system-quiesce, fleet-dispatch and server-node/site rows — are excluded: a node or site maintenance cycle never appears in halt history and never counts toward the resume-cycling rate limit.

Audit events (searchable via /api/events):

• warn:platform:halted — per plane, with actor, reason, in-flight mode and (on the runner plane) the in-flight job inventory at halt time.
• info:platform:resumed — the per-plane resume.
• warn:platform:resume_rate_limited — the cycling guard fired.
• warn:platform:attempt_disposition — a per-attempt disposition.
• info:platform:attempt_suspended, warn:platform:halt_fallback — the runner plane's enforcement-seam events for suspended attempts.
• warn:platform:agent_verb_escalation — a SUPERVISED agent asked for a platform verb.
• warn:agent:group_halted, info:agent:group_halt_resumed — the per-member/per-group scalpel on the agent plane.

Replica dispatch quiesce (the fleet verbs)

POST /api/fleet/quiesce holds dispatch on the replica that answers the request without telling runners anything: leases, queued work and runners all stay put, and work resumes exactly where it was on POST /api/fleet/resume. POST /api/fleet/drain is heavier — it also directs in-flight runners to finish and stop acquiring — but still lighter than a halt. Both are in-memory and per-replica: other replicas keep dispatching, and a restart clears the flag (by contrast, the two durable postures survive restarts).

The flag's "by whom/when" half is remembered per replica and surfaced on the fleet queue response (quiesced, quiesced_by, quiesced_at), and each verb emits its lifecycle event with the actor and replica id: info:runner:fleet_quiesced, warn:runner:fleet_drained, info:runner:fleet_resumed. Queued entries on a quiesced replica read dispatch is quiesced on this replica (resume with POST /api/fleet/resume).

This is the routine instrument for single-replica maintenance and controlled drains; see opschain docs cli/admin-server-quiesce.

System quiesce (estate-wide read-only)

Landing with M2 — this section documents the ratified behaviour as implemented at the time of writing.

POST /api/platform/quiesce (opschain admin platform quiesce --reason … [--timeout …]) begins the estate-wide maintenance posture: a durable row in fleet_postures under the reserved plane system, honoured by every replica, created idempotently (a repeat reports the active posture's progress; concurrent entries across replicas converge on one row). A reason is mandatory (a reason is required: it names the maintenance window on the 503 every refused write answers). The quiesced state is the precondition the backup-info and export/import lanes verify before they run — see opschain docs database-maintenance.

Three phases, reported everywhere as active → draining → quiesced:

1. Draining — from entry: the runner and agent planes take no new leases (acquire polls answer empty), trigger firing stops (opschain docs triggers), and the leader-run background loops (retention, database maintenance) stand down, so the estate settles while running attempts run to completion — never cancelled or suspended. Ordinary API writes still succeed during the drain; only the runner-RPC registry mutations are gated later, by the closed barrier. The remaining work is the union of live scheduler leases and runner job bindings — both shared-store facts, so the count is estate-wide.
2. Quiesced — when the last running attempt finishes, the write barrier closes (quiesced_at is stamped by exactly one closer — the polling CLI, a peer's watcher — via a guarded update). The barrier wraps the whole composed mux: every mutating request (POST/PUT/PATCH/DELETE under /api/ or /admin/) answers a 503 with system quiesced for maintenance: <reason>; new credential logins are refused the same way, while existing tokens keep reading and the session refresh rotation (AuthService Refresh) still works. Registry-mutating runner/agent RPCs (registration among them) answer the same named unavailability. The only exempt mutating routes are the quiesce family itself, the orthogonal emergency-halt family (/api/platform/halt, /api/platform/halt/attempts, /api/platform/resume) and the token refresh route.
3. Back to active — only by explicit verb. There is no auto-resume : the estate stays read-only until a bare opschain admin platform resume (POST /api/platform/quiesce/resume), which lifts the posture from either phase — so an abandoned drain is recoverable by the same resume.

Progress and the abandoned drain: GET /api/platform/quiesce reports the posture, phase, and the still-running attempts (run, step, attempt, action, target path); reading it also advances a finished drain, so the polling CLI is what usually closes the barrier. A watcher on every replica (each minute) closes a drain whose operator went away, so the estate never holds half-stopped forever.

Timeout abort: POST /api/platform/quiesce/abort — the CLI's --timeout expiry path — aborts the attempt cleanly back to fully active, never a half-state, naming the attempts that were still running. A barrier already closed refuses it with 409: the system is already quiesced: the drain completed — resume with 'opschain admin platform resume' (the CLI treats that race as success). The abort returns the estate to fully active.

Where the state shows: GET /readyz reports checks.system_quiesce with the phase; GET /api/info carries the quiesce attribute (phase, reason, since) — the GUI banner's source of truth; GET /api/platform/posture includes system and system_phase; queued entries on the fleet queue read system quiesce in force (<draining|quiesced>): <reason>. Resume is refused with no system quiesce is active when nothing is in force.

Audit events — deliberately the only writes a quiesced estate performs, at the barrier's edges:

• api:platform:quiesced — the barrier closed (posture id, reason, who started it, when).
• api:platform:resumed — the explicit resume, with the phase it resumed from. (Distinct from the halt family's info:platform:resumed.)
• warn:platform:quiesce_aborted — a timeout abort, naming the attempts still running.

The emergency halt is orthogonal: halting and resuming a plane works against a quiesced estate (the halt family is carved out of the barrier), and the quiesce stays in force throughout. The hands-on drill for all of this is opschain docs cli/admin-platform-quiesce.

Choosing a verb

• A runner misbehaving → drain it (opschain docs runner-registry).
• One replica needs maintenance → opschain admin server quiesce.
• Database maintenance, backup, export → opschain admin platform quiesce: finish everything, then read-only.
• Suspected compromise, stop everything now and keep the forensic state → opschain admin platform halt.

Failure modes

• A restart changes nothing durable: a halted or quiesced estate boots back into its posture (the gauges are re-armed from the rows at boot); a replica-quiesced server boots dispatching again — that flag is the in-memory one.
• A stale resume cannot reopen a newer halt — it must name the active posture row, and a mismatch is refused.
• A posture read failing never locks the estate: the gates fail open for availability (the durable row is re-read on every pass), so a degraded database degrades enforcement, not service.
• A drain that never finishes is bounded by the operator: abort (timeout) back to fully active, or cancel the offending run and let the drain complete. Nothing auto-resumes.

Related pages

• opschain docs cli/admin-platform-halt, opschain docs cli/admin-platform-resume, opschain docs cli/admin-platform-status, opschain docs cli/admin-platform-attempt, opschain docs cli/admin-platform-quiesce — the platform CLI family.
• opschain docs cli/admin-server-quiesce — the per-replica verb.
• opschain docs dispatch — what the postures stop.
• opschain docs fleet-queue — how each posture reads on queued work.
• opschain docs database-maintenance — the lanes a quiesced estate unlocks.
• opschain docs events — the audit trail all of these write.