The fleet queue

Included from Basic tier

The fleet queue is the dispatch plane's work inventory: every step the scheduler currently carries, either parked ready (waiting for a runner) or held by a live job lease, each entry annotated with a placement verdict so unsatisfiable work reads as queued-with-a-reason instead of a silent hang.

It is served at GET /api/fleet/queue (admin token) and surfaced in the GUI's Operations area. The neighbouring opschain admin queue (GET /api/admin/queue) is the aggregate view of the same plane — ready-depth per queue/lane, the oldest waiting age and the live leases — without per-entry verdicts.

Where entries come from

When a run's step becomes ready, the orchestrator enqueues it with the scheduler. Enqueue is idempotent per (run, step, attempt): a step already ready, leased or completed at that attempt is left alone. The entry stays in the queue until a runner's parked acquire call is handed a lease for it; the lease then holds the entry (leased: true) until the attempt completes, is released, or the lease expires and is reclaimed. Cancelling the run clears its entry from both states: a parked entry is removed from the queue (and stops counting toward group demand), and a leased entry's runner is directed to abort on its next renewal, releasing the lease.

Runners never poll the queue directly: each replica's dispatcher claims batches from its rendezvous-owned shards, orders them by priority lane and stride fair-share, checks concurrency-pool admission and placement targeting, and hands leases to the parked callers. Ordering is (priority, fair-share, ready_at) — never plain FIFO.

The response envelope

Field	Meaning
entries	The queue entries, documented below. Ready and leased entries appear together; leased tells them apart.
halted	The runner plane's active emergency posture (id, reason, halted_by, halted_at, …), or null when the plane is normal. While non-null, nothing dispatches anywhere.
quiesced	Whether dispatch is held: by the durable fleet quiesce/drain posture (one fleet_postures row every replica honours), or by this replica's own shutdown hold. The fleet verbs survive restarts and apply fleet-wide; only the shutdown hold is per-replica.
quiesced_by / quiesced_at	Who set the fleet posture and when, read from the posture row — the same answer on every replica. Present only while the posture is active.
fleet_verb	Which fleet posture is in force: quiesce (offers held, in-flight runners untouched) or drain (offers held AND in-flight runners directed to finish and stop acquiring). Present only while a fleet posture is active.

The entry, attribute by attribute

Attribute	Meaning
run_id	The owning run (a change or a workflow run).
step_id	The step this entry dispatches.
attempt	The attempt number — a retried step re-enters the queue as a new attempt.
queue	The scheduler's internal dispatch-queue name. Always default today; the field is reserved so additional named queues can be introduced without changing the entry shape.
ready_at	When the entry became (or becomes) eligible for dispatch. Retries with backoff park with a future ready_at; the dispatcher never offers an entry before it.
leased	false: parked ready, waiting for an eligible runner. true: a runner holds the job lease for it right now.
requirement	The step's placement demand, resolved once at enqueue (see below). An empty object demands nothing and matches every non-exclusive runner.
satisfiable	Whether anything can take the entry right now. Always true on leased entries (someone took it); on ready entries it is true exactly when reason is empty.
reason	Why the entry is not dispatching, from the verdict ladder below. Empty when the entry is satisfiable.
pinned_runner	The runner id this entry's run holds (run-scoped execution): the run's first delivered step pinned the run, and every later entry is offered only to that runner — a pinned ready entry is simply queued behind the runner (which holds at most the run's runner.max_parallel_steps jobs at once — default 1, opschain docs dispatch). Omitted when the run is unpinned. A pinned entry whose runner is offline or forgotten reads satisfiable: false with a reason naming it — the run will fail rather than re-spread.
run_scoped	Whether this entry's run is run-scoped — one runner and one shared workspace for the whole run, so its first delivered step pins it. Omitted when false.
scope_source	Why the run's execution scope resolved as it did: the runner.execution_scope override, or the derivation's deciding fact (derived: ruby-lane steps present on a declared tree the creation census found an actions.rb#/mintpress# step in, derived: actions.rb-backed tree, derived: declarative tree, derived: workflow runs are never pinned). The answer to "why did (or didn't) this change pin?". The ruby-lane values are only applicable to existing MintPress customers and those using the MintPress SDK or the packs:mintpress feature.
owner_type / owner_status / owner_terminal / target_path / correlation_id	The owning work item's context: every entry's run_id is a change, a workflow run, or a node background task (a Regenerate Actions / MintModel concretise job, whose synthetic job carries the task id as its run id — the entry the activities feed cannot show). Resolved per read across the three owning tables in one indexed statement; all omitted when the run id matches no owning row. Field semantics in opschain docs resources/fleet_queue_entry.

The requirement object

Resolved once, at enqueue — later settings edits do not reclassify queued work (cancel and re-run to re-resolve):

Field	Demand
group	The runner's group must equal this name.
groups	The runner's group must be one of this set — the node-path-binding shape: work bound to a path may land on any bound group, and never escapes the set.
labels	Every listed label must be present on the runner.
executors	Every listed executor kind (hostexec, containerexec) must be advertised by the runner. The step's own executor kind is folded in automatically at enqueue.
selectors	Group-attribute demands: each key must resolve to a runner-group attribute whose value is in the key's list (equality and in-set only), e.g. {"location": ["dc1"], "network": ["secure"]}.

Resolution order for the group and selectors:

1. The explicit request — the change/run creation payload's metadata.runner block ({"runner": {"group": …, "selectors": …}}). Malformed or unregistered placement is refused at create, never parked as a silently unsatisfiable entry.
2. The target node's settings — the runner block in the settings documents down the target path's ancestry, nearest node winning per field (group and each selector key independently).
3. Node-path bindings — every runner group whose binding covers the target path is added to groups, so bound work stays in its zone even when nothing else was demanded.
4. Otherwise unconstrained.

A runner in an exclusive group only ever receives work targeted at it — work naming the group, bound to it, or carrying selectors its attributes satisfy — never unconstrained overflow. An all-zero requirement therefore shows satisfiable: false when the only online runners sit in exclusive groups.

The verdict ladder

reason is computed per ready entry, first match wins — plane-wide causes before per-entry placement:

1. Halt — the runner plane carries an active emergency posture (opschain admin platform halt): runner plane halted (posture <id>): <reason>. Durable (a posture row); survives restarts; cleared only by a supervised opschain admin platform resume.
2. System quiesce — the estate-wide maintenance posture is in force (opschain admin platform quiesce): system quiesce in force (<draining|quiesced>): <reason>. Durable and topology-wide; running attempts finish (drain), nothing new dispatches anywhere, and only the bare opschain admin platform resume lifts it.
3. Licence — the licence lifecycle reached work-stop: licence expired: the server is not dispatching new work — apply a renewed licence (opschain admin licence apply). Queued work parks; nothing is lost.
4. Fleet quiesce/drain — dispatch is held by the durable fleet posture (POST /api/fleet/quiesce|drain): dispatch is quiesced on this replica (resume with POST /api/fleet/resume). Durable and fleet-wide: every replica reads the one posture row, a restart changes nothing, and POST /api/fleet/resume lifts it everywhere.
5. Placement — no runner can take the requirement right now:
   • no runners are registered
   • no registered runner satisfies the placement requirement {…} — the requirement is echoed so the unmet demand is readable
   • every eligible runner is draining
   • every matching runner is offline (stopped, or silent past the liveness window) — a runner is offline once stopped, or silent for 3 minutes
   • every eligible runner is draining or offline

Offline and draining are separate truths: an eligible-but-silent fleet reads as offline, never as satisfiable.

What changes the queue's behaviour, and who owns it

Lever	Authority	Effect
metadata.runner on the creation payload	The caller, per run	Explicit group/selector demand; wins wholesale per field.
runner block in node settings documents	DB-authoritative (edited via PATCH /api/settings/{id} / the GUI), inheritable down the node tree	Default placement for work targeting the subtree, resolved leaf-most-wins at enqueue.
Runner group attributes, bindings and exclusive	Fleet admin API (opschain admin runner-group set, opschain admin runner-attributes set)	What selectors can match, which paths are zone-bound, and whether a group refuses untargeted work.
Drain (runner, group, or fleet-wide)	Fleet admin API (opschain admin runner drain, runner-group drain, POST /api/fleet/drain)	Eligible runners stop taking new work; queued entries read draining.
Quiesce / resume	Fleet admin API (`opschain admin server quiesce	resume`, durable posture)
Server-node quiesce / resume	Fleet admin API (`opschain admin server node quiesce	resume `, durable per-node posture)
Platform halt / resume	`opschain admin platform halt	resume` (break-glass, durable posture)
System quiesce / resume	`opschain admin platform quiesce	resume` (durable posture)
Licence state	The installed licence (opschain admin licence)	Work-stop parks all new dispatch until a renewed licence is applied.
Concurrency pools	Scheduler pool limits (set programmatically; admission checked at dispatch, never at enqueue)	A full pool defers dispatch without affecting satisfiable.

No config.yaml key tunes the dispatch plane today; the scheduler's shard count, poll cadence, lease TTL (3 minutes) and liveness window (3 minutes) are built-in defaults.

Terminating entries, and the orphan sweep

Termination acts through the owning item's own cancel lane, never by deleting the queue row out from under the scheduler: a change or workflow run is cancelled (the same lane as opschain runs cancel / ChangeService.CancelChange), a node background task is aborted (the task-cancel lane, DELETE /api/…/node_background_tasks/{id} parity), and the scheduler then clears the entries itself — a parked entry is discarded, a leased runner is directed to abort on its next renewal. Placement stays resolved once at enqueue: terminate and recreate is the remedy for a mis-placed entry; there is no re-placement of a live entry.

A parked entry whose owner is already finalised is an orphan (the crash window between an owner finalising and its entry being discarded); no cancel lane can clear it, so the orphan sweep discards exactly those rows — leased entries, live owners and unresolved owners are left alone, and no owner is touched.

Both verbs are admin-only (the /admin updatable determination) and serve on API v2 as FleetQueueService.TerminateQueueEntry / SweepQueueOrphans, beside the owner-enriched FleetQueueService.ListQueueEntries read; the CLI face is opschain admin queue list|terminate|sweep. Every termination and every swept run writes a customer-visible audit event (api:admin_queue:terminate / api:admin_queue:sweep) naming the administrator, the queue entries and the owning item, joined to the owner's correlation chain where one exists.

Events vocabulary

Queue-relevant lifecycle events, all visible in opschain events and the Activity feed:

• info:runner:registered, info:runner:goodbye, info:runner:forgotten
• info:runner:drain_changed, info:runner:group_drain_changed, info:runner:group_deleted
• info:runner:fleet_quiesced, info:runner:fleet_resumed
• warn:runner:fleet_drained
• warn:runner:aged_out — a registered runner went silent past the liveness window without a goodbye; emitted once per offline episode and re-armed when the runner returns.
• api:admin_queue:terminate, api:admin_queue:sweep — an administrator terminated a queue entry's owner / swept orphaned entries; the event names the administrator, the entries and the owning item.

Failure modes

• Unsatisfiable placement is never silent: the entry parks with satisfiable: false and a human-readable reason. Fix the fleet (bring a matching runner online, undrain, register the group/attributes) or cancel the run; the requirement itself is frozen at enqueue.
• A lost runner is handled by whether the job ever reached it — two cases, never a silent redrive:
  • The lease expires before the job was delivered (a claim the dispatcher held but no runner took, or an offer a runner never acked): the lease is reclaimed and the entry is requeued automatically at its original queue position and attempt — nothing external saw it. The same give-back applies when a drain, halt or quiesce lands while the runner's poll is parked with a delivered-but-unseen lease (the abandon verb on opschain docs dispatch).
  • The runner held the job and went silent: no automatic retry. When the lease TTL (opschain server start --lease-ttl, default 3 m, counted from the runner's last renewal) lapses, the attempt fails fast — the step reads system_error with the cause runner lost: lease expired without a result (api:steps:system_error), its log is sealed as it stands, and the owning change rolls up to error (api:changes:error). The fleet watcher announces the runner itself with warn:runner:aged_out once it is silent past the 3-minute liveness window. The work is retried by a human — opschain runs retry <id> re-enqueues a new attempt that any eligible runner may take. A run pinned to the lost runner (run-scoped execution) fails the same way and is never re-spread.
• A crashed replica's claimed-but-undelivered entries are reclaimed by its peers via heartbeat staleness — at their original attempt when no runner saw the offer, as a new attempt when one did.
• A store (database) outage is not a lost runner. While the store is unreachable every replica keeps serving (/healthz 200) but reports not ready (/readyz 503, the store check failing), so a load balancer drains it; a runner that reaches a replica directly hears unavailable (store unavailable: …) on its renews, chunk uploads and result submission — never credential invalid, never a fence — and retries where it is, keeping the job running and its log chunks spooled. The job completes once the store is back; only an outage longer than the lease TTL (counted from the runner's last recorded renewal) ends in runner lost as above.
• A restart changes no posture: the fleet quiesce/drain verbs and the halt are durable rows; only the shutdown-time per-replica hold dies with the process (deliberately — it IS the restart posture).

Related pages

• opschain docs cli/runs-create — how work enters the plane, including the placement a creation payload may demand.
• opschain docs dispatch — the lease lifecycle (claim, deliver, renew, release, abandon, expire) behind the failure modes above.
• opschain docs cli/runs-retry — the human retry of an attempt that failed runner_lost.