The dispatch plane

Included from Basic tier

The dispatch plane is the server-side scheduler that carries every ready step from the orchestrator to a runner: a store-backed lease queue over the shared database. Ready steps live in queue_ready, sharded by run; live leases live in scheduler_job_leases; both tables are shared-store facts, so every replica sees the same queue and a restart loses nothing.

Its neighbours: the orchestrator enqueues a step when it becomes ready; a runner's parked AcquireJob long poll is the consumer; the fleet queue page (opschain docs fleet-queue) is the per-entry read model with placement verdicts, and the runner registry (opschain docs runner-registry) supplies the fleet facts placement matches against.

The two loops

Each replica runs two loops:

• The coordination loop (every 5 seconds): writes this replica's heartbeat row, computes rendezvous shard ownership over the live membership and holds the matching shard leases, contends for the cron/maintenance leader lease (cron-leader), reclaims a crashed peer's un-renewed claims, and expires lost-runner leases. A replica silent past 15 seconds is treated as dead.
• The dispatch loop: claims batches (up to 64 entries) of due ready steps from this replica's owned shards, orders them by priority lane and stride fair-share, checks concurrency-pool admission and placement, and hands leases to parked acquire callers. Polling is adaptive — 250 ms when busy, backing off to 2 seconds idle — with same-process kicks from enqueue/acquire/complete for sub-tick latency. A claim buffered longer than the 2-second hold window goes back to the queue, so one replica never sits on work another could dispatch.

The replica identity defaults to hostname plus a per-boot random suffix: a restarted process is a new replica, and its stale rows are reclaimed by the peers exactly as a crash would be. The membership row itself is reaped by the leader once its heartbeat ages past the replica retention window (24 hours by default, opschain server start --replica-retention): recent corpses stay visible in /admin/status for crash forensics, a row still holding a live shard or leader lease is never deleted, and the table stops growing one corpse per restart or deploy.

Shard ownership and replica failover

The shard space is fixed at 64 shards. A run hashes to one shard (FNV(run_id) % 64), so every step of a run lands on the same shard and one dispatcher owns a run's dispatch. Shards are assigned to replicas by rendezvous (highest-random-weight) hashing over the live replica set, so a membership change — a replica joining, crashing, or restarting — moves only the departed replica's shards; the fixed shard count means stored shard values never need rewriting. Ownership is enforced with named shard leases in the coordination table, claimed and released on each heartbeat tick.

When a replica dies mid-dispatch, its peers reclaim its claims within the 15-second staleness window instead of waiting out the full lease TTL: claims that were exposed to a runner come back as a new attempt (a late holder is fenced); claims nobody saw keep their attempt and queue position.

The aged any-shard lane

A shard's owner offers its claims only to the runners parked on it, and a runner's long poll parks on whichever replica the load balancer picked. On a small fleet — or a wide one with most runners drained — a step whose owner has no eligible runner parked would otherwise wait until some eligible runner's poll happened to land there: tens of seconds per poll window, with a one-in-N chance per window. So a ready entry that has sat for the shard fallback window (15 seconds) becomes claimable by any replica that has a runner parked, whichever shard it lives on. By then its owner has provably found no matching runner (an owner with one parked delivers within a dispatch tick plus the hold window), and the window matches the staleness window after which a dead owner's shards have moved anyway. First-dispatch latency on such a fleet is therefore bounded by the window plus one dispatch tick — inside a single runner poll window — instead of by replicas × poll windows.

Fresh entries keep their owner-only isolation: this aged lane and the site lanes below are the only cross-shard claims. Placement matching is unchanged — a replica that claims an aged entry no parked runner of its own can take returns it to the queue at its original position, where it keeps aging for the next replica.

Site-affine claiming (multi-site estates)

On an estate whose replicas carry a site identity (the site config key — the same vocabulary as the reserved location group attribute, e.g. dc1; see opschain docs config), claiming becomes site-preferring, layered under the placement machinery without changing its guarantees:

• A step demanding exactly one location selector value is stamped with that value on its queue row at enqueue. Steps with no location demand (or an in-set demand spanning several) carry no stamp and behave exactly as before.
• A replica claims its own site's stamped steps from any shard — the site's runners park on the site's replicas, and shard assignment is site-blind — alongside unstamped steps from its own shards.
• Another site's stamped step never strands: it becomes claimable anywhere the moment its site has no live replica (quiesced, held and stopped nodes all leave the membership table — so during site maintenance the fallback is immediate), or once it has waited out the 15-second fallback window with its site live but idle.

The stamp is a claim preference, never a placement check: wherever the claim lands, delivery still requires a runner whose group satisfies the step's selectors. During site maintenance the site's runners fail over on their ordered server list, park on the surviving site's replicas, and the fallback hands them their own site's work there. Replicas without a site key claim everything exactly as before — set the key on every replica of a multi-site estate, or on none. Each replica's site appears on opschain admin status (its own, and per membership row).

Lanes and fair-share

Dispatch order within a claimed batch is (priority lane, stride fair-share, ready_at) — never plain FIFO.

• Priority lanes: system (priority ≥ 100), interactive (≥ 50), bulk (below). Higher lanes always dispatch first. The lane names appear in the admin queue view and metric labels.
• Stride fair-share within a lane: each fair key accumulates a pass value of 2²⁰ / weight per dispatched step, and the next pick is the lowest pass. The guarantee: a project flooding the queue advances its own pass quickly and interleaves with everyone else instead of starving them, with throughput proportional to weight. A key first seen joins at the current minimum pass, so a new project competes immediately without draining its backlog ahead of established keys. Passes advance on delivery only, and the state is per-dispatcher and in-memory — fairness is a dispatch-ordering policy, not durable state.
• FIFO (claim order, which preserves ready_at order) within a key.

A step's priority, fair key, weight, pools and placement requirement are resolved once, at enqueue, by the server's classifier. The classifier wired today resolves only the placement requirement (explicit request → node settings → node-path bindings — the resolution ladder on opschain docs fleet-queue); it assigns no priority, fair key or pools, so all current work rides the bulk lane under one shared fair key. The lane and fair-share machinery is live in the dispatch path and takes effect for any classification that sets those fields.

Enqueue idempotency

Enqueue is idempotent per (run, step, attempt), checked in one transaction: an attempt already recorded done (scheduler_done), a step already holding a lease, or a step already parked ready (ON CONFLICT (step_id) DO NOTHING) is left alone. A retried step re-enters as a new attempt. The scheduler_done bookkeeping this relies on is aged out by the leader-run retention job (retention.scheduler_done_days, see opschain docs config); the cutoff must stay far beyond the lease TTL, since a pruned row makes a repeat enqueue of that attempt re-queue the step.

The lease lifecycle

1. Claim: the dispatcher moves a due ready row into scheduler_job_leases, held by this replica, TTL 3 minutes by default (opschain server start --lease-ttl).
2. Deliver: the lease is marked delivered and handed to a parked acquire caller; the runner protocol then offers it with an ack deadline.
3. Renew: the runner's lease renewals extend the expiry and record that the runner is executing. A renewal, completion or release presenting a stale token gets lease expired or revoked — the holder is fenced and must abandon the work. That verdict is the only one: a renewal, ack or completion the server cannot record because its store is unreachable answers unavailable (store unavailable: … (this server's readiness check 'store' is failing; retry)) and changes nothing — the lease, the offer and the job binding stand as they were, the runner retries on its cadence, and the job survives any outage shorter than the lease TTL. The same answer comes back from a replica that cannot read the job's binding (runner_jobs) or the runner's registration while the store is down; credential invalid is reserved for a binding or registration the store says is gone.
4. Complete: idempotent by (step, attempt); the entry leaves the plane.
5. Release: an unfinished step returns to the queue as a new attempt (un-acked offer re-delivery, graceful holder shutdown). An acquire call cancelled after delivery hands the claim back at its original position and attempt — nothing external ever saw the lease.
6. Abandon: a delivered lease the server may not hand on after all — a drain, halt or quiesce landed while the runner's poll was parked — is handed back at its original attempt and queue position, exactly like the cancelled acquire: nothing external saw it, so there is no new attempt to track. Attempt numbers in runs show and the fleet queue therefore count real holds only.
7. Expire: a lease past its TTL whose runner went silent is fenced and reported through the expiry sink — the attempt fails fast as runner_lost, no auto-retry, and its log is sealed. The step's status is system_error (an infrastructure failure, not the action's verdict) with the cause runner lost: lease expired without a result, recorded as api:steps:system_error; the run rolls up to error (api:changes:error). The TTL counts from the LAST renewal (delivery, ack or a 60 s renew), so the failure lands between 2 and 3 minutes after the runner died, on the next coordination tick.

During an active runner-plane emergency halt the expiry sweep is held entirely: lease rows stay in place, nothing ages into runner_lost, and a lease whose TTL lapsed during the halt renews by token match alone (its un-reclaimed row proves the holder was never superseded). The first sweep after resume reconciles any lease that genuinely died during the halt. See opschain docs platform-halt.

Concurrency pools

A named pool caps how many leased steps may hold it concurrently. Admission is checked at dispatch, never at enqueue: a full pool defers dispatch (the claim is retried next tick) without affecting the entry's satisfiable verdict. A step demanding several pools acquires every hold or none, with pool rows locked in name order; pools without a limit row are unlimited. Limits are set programmatically (SetPoolLimit) — there is no admin verb or config key for them today.

Placement admission

Each parked acquire call carries its runner's placement posture (group, labels, executors, group attributes, exclusive flag). The dispatcher hands a claim only to a waiter that satisfies its requirement — group equality or set membership, every selector resolving to an in-set group attribute, every demanded label and executor present, and an exclusive group's runner only taking work targeted at it. Delivery per tick is restricted to the waiters parked at the tick's start, so a fast acquire loop cannot drain a batch in claim order and defeat fair-share.

A claim no currently-parked waiter can take is returned to the queue rather than buffered: the fleet queue view reads lease rows as leased, so holding it would hide unsatisfiable placement from the queued-with-reason surface.

Drained runners are never matched. Each dispatch tick that has claims to hand out reads the fleet's drain state once — every runner and group under an operator drain, one set-based read however large the fleet — and skips the parked polls of drained runners and of runners in drained groups. A drain requested on any replica therefore takes effect on every replica within a tick, without the offer first being handed to a drained runner and taken back. A drained runner's poll does not park on the queue at all: the server holds it for the poll window and answers it empty, so an idle drained runner costs one liveness write per window instead of a request and a write every quarter second; a lifted drain is seen on the runner's next poll, at most one poll window later. Placement itself — the requirement shape, the enqueue-time resolution ladder, write-time validation and the verdict ladder — is documented on opschain docs fleet-queue.

Run→runner affinity (run-scoped execution)

A step of a run-scoped run (actions.rb changes and any declared change whose tree carries a Ruby-lane step (actions.rb# bridge or mintpress# graft — the census; the Ruby lane is only applicable to existing MintPress customers and those using the MintPress SDK or the packs:mintpress feature) by default, or runner.execution_scope: run, see opschain docs settings) carries a run-scope mark in its class, resolved at enqueue beside the placement requirement. The first such claim delivered pins the run to that runner — a durable scheduler_run_pins row, written before the delivery so a parallel sibling claimed in the same tick routes identically, and surviving a server restart the way job leases do. Every later claim of the run is offered only to the pinned runner's polls, and at most runner.max_parallel_steps of the run's claims are out with it at once (default 1 — strictly serial, today's behaviour): past that cap the claim returns to queue_ready, queued behind the runner rather than spreading — that is what keeps one workspace's filesystem state visible to every step (opschain docs runner-executors).

With a cap above 1, a capable runner (protocol 4+) holds and executes up to that many steps of the one pinned run concurrently in the shared run workspace — the parallel branches of the run's tree, which the engine's readiness ordering alone makes eligible together. The runner's extra polls are run-restricted (only_run_id on the acquire call), so a concurrent poll can never pull another run's work: cross-run isolation stays one run per runner at a time. The cap is frozen into each step's class at enqueue; an older runner never receives the cap and simply stays serial (a system log line on the step says so), and the server-side admission caps delivery regardless of what the runner asks for. The runner may also clamp the cap to its own declared capacity (opschain runner --max-parallel-steps): effective concurrency is min(the cap, the declaration), the clamped runner stops polling for extras it would not run, and the run's step log carries one notice naming both numbers whenever the clamp binds. See opschain docs settings for the runner.max_parallel_steps setting and opschain docs runner-executors for how concurrent steps share the run workspace.

The pin is removed when the run finalises, and it is never re-pointed: a pinned runner whose lease expires mid-run fails the attempt — and with it the run — with a cause naming the pinned runner (plus an error:changes:pinned_runner_lost event on the run), exactly as upstream's change fails when its pod dies. No silent re-spread, no re-pin. The fleet queue view exposes the pin per entry (pinned_runner) and reads a pinned entry whose runner is gone as unsatisfiable with a reason naming it.

Inspection

• GET /api/fleet/queue (admin token) — every entry, ready and leased, with its placement verdict: the queued-with-reason surface (opschain docs fleet-queue).
• GET /api/admin/queue (opschain admin queue) — the aggregate view: ready depth per queue and lane, the oldest waiting age, and every live lease with its holder replica, delivered/runner-renewed flags, acquire time and expiry, enriched with the owning run's action and target path.
• Both are set-based reads — a handful of statements regardless of queue size.

What changes dispatch behaviour, and who owns it

Lever	Authority	Effect
--lease-ttl	opschain server start flag	The lost-runner fencing window (default 3m): how long a silent runner holds a step before it fails as runner_lost.
--poll-window	opschain server start flag	How long a runner's acquire call parks before answering empty (default 50s) — the offer latency/connection-count trade.
--replica-retention	opschain server start flag	How long a dead replica's membership row stays visible before the leader reaps it (default 24h) — forensics visibility vs table growth.
retention.scheduler_done_days	config.yaml (deploy-authoritative, opschain docs config)	Ages out the enqueue-idempotency bookkeeping; keep far beyond the lease TTL.
Fleet quiesce / drain / resume	Fleet admin API (opschain admin server quiesce|resume, POST /api/fleet/drain)	Holds offers fleet-wide through one durable posture row every replica reads; survives restarts, lifted by the resume verb.
Platform halt, system quiesce, licence work-stop	opschain admin platform halt|quiesce, the installed licence	Stop all dispatch estate-wide; see opschain docs platform-halt and the verdict ladder on opschain docs fleet-queue.
Pool limits	Programmatic (SetPoolLimit)	Caps concurrently leased holders per named pool.

The shard count (64), claim batch (64), poll cadence (250ms–2s), heartbeat (5s), staleness window (15s), shard fallback window (15s) and hold window (2s) are built-in defaults with no config key.

Failure modes

• A lost runner never loses the work silently: a lease that expires undelivered is reclaimed at its original attempt; a delivered lease that expires fails the attempt fast as runner_lost (no auto-retry), and a human retry (opschain runs retry) re-enqueues a new attempt. The two cases are spelled out on opschain docs fleet-queue.
• A crashed replica's claimed-but-undelivered entries come back within the 15-second staleness window — exposed offers as a new attempt, unexposed claims at their original position.
• A full pool or unsatisfiable placement parks work without losing it; only the fleet queue's verdict (not the admin queue aggregate) says why.
• Fairness state is in-memory: a dispatcher restart resets stride passes, which can momentarily reorder a mixed backlog but never loses or duplicates entries.

Related pages

• opschain docs fleet-queue — the entry dictionary, requirement resolution and the verdict ladder.
• opschain docs runner-registry — the fleet facts placement matches against.
• opschain docs platform-halt — the postures that stop dispatch.
• opschain docs cli/admin-queue — the aggregate CLI view.