6.1 How a runner works, and what happens when one dies

Included from Basic tier

After this you can: explain the offer and lease between server and runner, and say what the server does when a runner dies mid-step.

Before you start — tick each one:

• Lesson 3.1 — Your first change (opschain learn 3.1)

In the GUI: the Runners panel in the Operations area shows every member with its derived state, last-seen age and current job, and the queue backlog beneath it with each entry's placement verdict.

Every server-side step you have run since lesson 3.1 executed on a runner: a process that dials out to the server, takes one job at a time, runs it, and streams the logs back. This lesson is the mechanism — how a runner joins, how work reaches it, and the two honest outcomes when a runner disappears with work in hand. Nothing here needs a fleet of your own yet; lesson 6.2 starts one.

1. Outbound only, identity by keypair

• Outbound only. A runner connects to the server (opschain runner --server …); it binds no listener and needs no inbound firewall holes. Anywhere that can reach the server's URL can host a runner — the datacenter next to the database, a locked-down subnet, your laptop.
• Enrolment. The first start redeems a registration token (minted by an administrator), generates a keypair, and persists a durable identity in its state directory. Identity is the keypair, not the name: restarts re-attach the same fleet member, and names are display labels.
• Executors. A runner advertises what it can execute: hostexec (processes on its own host) and, when a container runtime is reachable, containerexec (steps declaring image:). Placement only offers a step to runners advertising the executor it demands.

2. The offer and the lease

When a run's step becomes ready, the server enqueues it with its placement requirement — the group it must land on, the executors it needs, any selectors — resolved once, at enqueue. Runners never poll the queue themselves: each parks an acquire call on the server, and the dispatcher hands a parked runner a lease for the next entry it is eligible for. The lease is the contract:

• the runner renews it while the step runs, and uploads log chunks and the result under it;
• the lease lives for a fixed TTL (opschain server start --lease-ttl, default 3 minutes) counted from the runner's last renewal;
• the runner is considered online while it keeps polling; silent for 3 minutes, it is offline.

One runner, one job at a time. Scale is more members, not bigger ones (lesson 6.3).

3. When nobody can take the work

Work nobody can take never hangs on a phantom lease: the entry stays queued with a reason. Following a change into an empty fleet shows it once:

opschain runs create change acme/environments/dev/assets/web --action deploy --follow

created change 019a0d7e-63fe-70e7-8d24-daca5d8db348 (deploy on acme/environments/dev/assets/web) — pending
run 019a0d7e-63fe-70e7-8d24-daca5d8db348 queued — no online runner currently serves this run

The change's show payload carries the same fact as waiting_on_runner_capacity (true once a step has queued past ten seconds with no online, non-draining runner able to take it; cleared the moment one connects), and the GUI's change page shows the matching banner. The administrator's view names the verdict per entry:

opschain admin queue list

RUN       OWNER   STATUS  TARGET                                   CHAIN     STATE   REASON
019a0d7e  change  queued  /projects/acme/environments/dev/assets/web  019a0d7e  parked  no runners are registered

The reasons form a ladder — a halt, a system quiesce, an expired licence, a fleet quiesce, then placement (no runners are registered, no registered runner satisfies the placement requirement {…}, every eligible runner is draining, every matching runner is offline …). Offline and draining are separate truths: an eligible-but-silent fleet reads as offline, never as satisfiable. The full ladder is opschain docs fleet-queue.

4. When a runner dies mid-step

A runner that goes silent is handled by whether the job ever reached it — two cases, never a silent re-run:

• The job was never delivered. The server held a lease for it, or offered it, and no runner acknowledged it. The lease is reclaimed and the entry is requeued automatically at its original queue position and attempt; nothing outside the server saw it.

• The runner held the job and went silent. No automatic retry. When the lease lapses the attempt fails fast: the step reads system_error with the cause runner lost: lease expired without a result, its log is sealed as it stands, and the owning change rolls up to error. Whether the work runs again is your call — opschain runs retry <run-id> creates a new run that any eligible runner may take:

  opschain runs retry 019a0d7e-63fe-70e7-8d24-daca5d8db348

  created change 019a0d8a-1c20-7b3e-9f41-2a6c0d5e7f10 (deploy on acme/environments/dev/assets/web) — pending

  A run pinned to the lost runner (run-scoped execution, lesson 6.3) fails the same way and is never re-spread.

Read it back on the chain: opschain chains triage <correlation-id> attributes the failure to the runner (lesson 5.1).

5. Liveness, from the record and from the status read

The fleet names the runner itself: once it is silent past the liveness window the estate records warn:runner:aged_out (once per offline episode, re-armed when the runner returns); a clean shutdown records info:runner:goodbye instead, and the fleet view reads it stopped immediately rather than three minutes later.

opschain events --type-search runner --since 1h

opschain admin status is the one-page health read — the server's version, the answering replica, the store's ping, and the queue summary that tells you whether work is parked or leased:

opschain admin status

status:    ok
version:   v5.7.0
replica:   replica-a (leader: true, shards: 64, draining: false)
store:     ok
queue:     1 ready, 0 leased

1 ready, 0 leased with a runner you believe is running is the signal to look at the Runners panel's last-seen age — the runner has stopped polling, or cannot reach the server's URL.

Check yourself

• What does --follow print when no runner can take a run, and what cause does the step record when the runner holding it is lost?
• Which of the two lost-runner cases requeues automatically, and which waits for opschain runs retry?

What you learned

• A runner dials out, enrols once by keypair, and advertises its executors; placement only offers it work it can execute.
• Work reaches a runner as a lease the runner renews; the lease TTL and the liveness window are both three minutes.
• Work nobody can take is queued with a named reason, never hung on a phantom lease.
• An undelivered lease requeues itself; a delivered one fails fast as runner lost, and a human retries.
• warn:runner:aged_out and opschain admin status are where liveness shows.

Next: lesson 6.2 — Your first runner (opschain learn 6.2), or back to your track (opschain learn).