The runner

Included from Basic tier

The runner is the execution client: a single opschain runner process that registers with a server, long-polls it for leased work, executes one step at a time through the executor contract (opschain docs runner-executors), and streams logs and results back. Everything is outbound ConnectRPC from the runner to the server — runners live in their own network and dial in; the server never connects to a runner. The server-side bookkeeping (groups, drain, forget, liveness) is the peer page opschain docs runner-registry; how queued work is matched to runners is opschain docs fleet-queue.

It holds exactly one job at a time (the one-job-at-a-time lease model), so capacity scales by running more runner processes, not by tuning one. The single exception is a pinned run-scoped run whose runner.max_parallel_steps setting exceeds 1 (opschain docs dispatch): its pinned runner holds and executes up to that many steps of that one run concurrently, polling for the extras with run-restricted acquire calls — it still never takes another run's work while doing so. The runner can bound that concurrency to its own resources with a declared capacity (--max-parallel-steps / OPSCHAIN_RUNNER_MAX_PARALLEL_STEPS): effective concurrency is min(the offer's cap, the declared capacity), and it also stops polling for extras it could not run. Undeclared (the default) follows the server's cap unchanged. The clamp is honest, never a silent downgrade: when it binds, the run's step log carries one system notice naming both numbers (max_parallel_steps clamped to this runner's declared capacity: the server's offer allows 50 parallel steps but this runner declares 8 …), so an operator sees why a cap-50 run executes at 8. Size the declaration as N × per-step peak RSS ≤ the host/container memory (each concurrent MintPress step is its own Ruby SDK process — only applicable to existing MintPress customers and those using the MintPress SDK or the packs:mintpress feature).

Registration and persisted identity

On first run the runner redeems a registration token (--token, minted with opschain admin runner-token create) against --server, enrolling an Ed25519 keypair. The resulting identity is persisted in the runner's state directory (--dir, default <opschain home>/runner):

File	Contents
runner.json	server_url, runner_id, group, name (0600)
runner.key	The Ed25519 private key, base64 (its own 0600 file)

The runner stores nothing else — no database credentials and no long-lived bearer tokens. Each process start runs a sign-the-challenge handshake with the saved key to open a short-lived session token, re-run automatically 30 s before expiry. Pointing a registered state directory at a different server is refused: runner is registered with <url>; remove <dir> to re-register. Starting unregistered without a token fails with runner is not registered and no registration token was given: set OPSCHAIN_RUNNER_TOKEN (or pass --token).

If the server stops recognising the runner (for example after opschain admin runner forget), the runner re-registers automatically — reusing its durable keypair, provided the process was started with a still-valid registration token in OPSCHAIN_RUNNER_TOKEN (or --token); without one it exits with runner session is no longer recognised and no registration token (OPSCHAIN_RUNNER_TOKEN or --token) is available to re-register. A rejected re-registration ends the process (re-registration rejected by the server).

The work loop

1. Acquire — a long poll against the server's park window. The client deadline is 70 s by default, deliberately above the server's 50 s park. An empty poll loops immediately (with a sub-250 ms jitter); errors back off exponentially with full jitter, 1 s doubling to a 60 s cap.
2. Ack — a received offer is acked, binding the lease to this runner. An offer lost before ack is simply dropped and the loop re-polls.
3. Execute — repo content and action packs are fetched, the executor is built for the offer's executor kind, and the step runs under the same executor contract the server's local dispatcher drives.
4. Renew — while the job runs, the lease is renewed every 60 s. Each renewal can carry back a directive:
   • CANCEL — graceful executor stop; the outcome submits as cancelled.
   • ABANDON — the lease was fenced (for example reclaimed after expiry); stop and submit nothing.
   • DRAIN — finish the in-flight job, then stop acquiring.
   • HALT — suspend the step in place (see Suspend below) while the runner keeps renewing.
   • CONTINUE — lift a suspension; every non-HALT directive also resumes first, so a frozen step is never signalled stopped. Renewal transport errors leave the job running: in-flight jobs ride out disconnects up to the server's lease TTL.
5. Submit — the result (exit code, cancelled/timed-out flags, step_result.json, or an execution error) is submitted with up to 5 retried attempts. A fenced lease stops retrying (result not applied (lease fenced)); after 5 failures the runner prints result submission gave up after 5 attempts and moves on — the server redrives the attempt when the lease expires.

Log spooling and streaming

Executors stream masked log lines into a per-attempt disk spool under <dir>/spool/<run>/<step>-<attempt>/; an uploader goroutine ships the backlog to the server as compressed frames, removing each chunk only after the server acks it (at-least-once). The spool is bounded — 64 MiB per step attempt by default (--spool-max-mb) — and over budget the oldest chunks are dropped and replaced with a gap marker, so loss is honest and visible in the log rather than silent. The runner also notes it on stderr: log spool over budget: dropped the oldest N chunk(s) under a gap marker. When the step finishes, the runner drains the remaining backlog for up to 2 minutes before submitting; the server performs the authoritative seal at result submission. Server-rate-limited uploads honour the Retry-After the server sends.

Git content fetch

A job that pins repo content (a git remote + sha, or action packs) fetches each tree from the server's mirror over the runner protocol — the runner never runs git and needs no git credentials. Each archive (decompressed cap 512 MiB) is extracted into its own directory under the state dir and handed to the executor as a pinned, read-only tree; the directories are removed when the job ends. Asking the materialised tree for any other ref fails (content tree is pinned to "<sha>"; cannot materialise "<ref>").

Host metadata

The runner self-reports a static host inventory — hostname, up to 8 non-loopback interface addresses with MACs, CPU count, Go version, the runner binary's version and commit, pid, start time, state directory and the declared step capacity (--max-parallel-steps; 0 = undeclared) — at registration, on every session handshake (so every process start refreshes it), and piggybacked on a poll when the facts drift, re-checked at most every 10 minutes. It is inventory, not telemetry: no memory, load or disk metrics. The server renders it on the runner detail (opschain admin runner show, where an undeclared capacity reads unlimited (follows the server cap)) and surfaces the capacity as max_parallel_steps beside the executors on the admin runners list (GET /api/runners).

Drain, stop and suspend

• Local signal: SIGTERM or SIGINT drains — draining: finishing the in-flight job, no new work (signal again to stop hard). A second signal stops hard (the in-flight job's lease then expires server-side and the attempt fails fast as runner_lost, for a human to retry). An idle long-poll is cancelled immediately on drain so the exit lands within a container stop grace.
• Server-side drain: opschain admin runner drain (or a group/fleet drain) reaches the runner on its next poll or renewal; it finishes the in-flight job, then stops acquiring and exits: drained: no longer acquiring jobs. While drained, each of its polls is held for the poll window and answered empty (one liveness write per window), and the resume reaches it on the poll after that.
• Goodbye: a drained exit announces itself (bounded to 2 s, best-effort) so the fleet view reads stopped immediately instead of aging out — goodbye: the server marked this runner stopped, or on failure goodbye not delivered (the server will age this runner out). Hard stops and rejections send no goodbye.
• Suspend (platform halt): a HALT directive freezes the running step in place — hostexec SIGSTOPs the step's process group, containerexec pauses the container — while the runner itself keeps polling renewals; a later CONTINUE resumes the step exactly where it stopped. A halt arriving before the step process exists is retried every renewal. Executors that cannot suspend run to completion instead, reported back on the renewal and printed locally: halt: this step cannot be suspended in place and keeps running to completion. See opschain docs platform-halt.

There is no separate runner control binary: the control surfaces are the server-side fleet verbs (opschain admin runner drain|resume|forget, opschain admin runner-group drain|resume) plus local process signals.

State vocabulary (as the server reports it)

The API (GET /api/runners) exposes booleans and timestamps, not an enum: online, draining, group_draining, last_seen_at, stopped_at, drain_requested_at, current_jobs. A runner is online when it has not said goodbye (stopped_at null) and has polled within the 3-minute liveness window. The CLI (opschain admin runners list) derives one STATE label from them, checked in this order:

STATE	Meaning
stopped	The runner said goodbye (stopped_at set).
draining	Drain requested (runner or group) and still holding a job. A (group) or (runner+group) suffix names where the drain came from.
quiesced	Drain requested and idle — safe to stop the process. Same suffixes.
ready	Online and undrained.
-	Offline: silent past the 3-minute liveness window without a goodbye.

The fleet queue's placement verdicts use the same truths: offline means stopped, or silent past the liveness window, and offline and draining are reported separately, never conflated.

What changes the runner's behaviour, and who owns it

Lever	Authority	Effect
--server, --token, --name, --labels, --dir	The operator starting the process	Registration identity and state location. Flags are summarised here only — see opschain docs cli/runner.
--executors auto|<list>	The operator starting the process	Which executor kinds the runner advertises (opschain docs runner-executors).
--env-passthrough / OPSCHAIN_RUNNER_ENV_PASSTHROUGH	The operator starting the process	Extra host environment variable names handed from the runner's own env to every hostexec step. OPSCHAIN_VAULT_* names are dropped from it: the runner's vault access is its config file's secrets_providers section, never its process environment.
<runner dir>/config.yaml → secrets_providers:	The operator owning the runner host	One entry per secrets provider code the runner resolves at Prepare (kind, address, mount, auth method and material, TLS); the codes are the capability the runner declares for placement. A host still carrying OPSCHAIN_VAULT_* with no section refuses to start until opschain runner migrate-vault-env has written it (opschain docs runner-executors).
--spool-max-mb	The operator starting the process	Per-step log spool budget (default 64 MiB); over it, oldest chunks drop under a gap marker.
--max-parallel-steps / OPSCHAIN_RUNNER_MAX_PARALLEL_STEPS	The operator starting the process	Declared step capacity: clamps a pinned run's runner.max_parallel_steps to min(offered, declared), with one per-run notice in the step log when the clamp binds. Unset = follow the server's cap.
OPSCHAIN_RUNNER_RENEW_INTERVAL	Environment of the runner process	Lease renewal cadence (Go duration, default 60 s) — also how quickly cancel/drain/halt directives reach an executing job.
OPSCHAIN_RUNNER_POLL_TIMEOUT	Environment of the runner process	Acquisition poll deadline (default 70 s, above the server's 50 s park).
HTTP_PROXY / HTTPS_PROXY / NO_PROXY	Environment of the runner process	The runner's HTTP client honours the standard proxy environment.
Drain / resume / forget	Fleet admin API (opschain admin runner drain, runner-group drain, POST /api/fleet/drain)	Stop taking new work; forget removes a stale registration (refused while online or holding a job).
Platform halt / resume	opschain admin platform halt|resume	HALT/CONTINUE directives: suspend the in-flight step in place.

Events vocabulary

Runner lifecycle events, visible in opschain events and the Activity feed (recorded by the server, not the runner):

• info:runner:registered, info:runner:goodbye, info:runner:forgotten
• info:runner:drain_changed, info:runner:group_drain_changed, info:runner:group_deleted
• info:runner:fleet_quiesced, info:runner:fleet_resumed, warn:runner:fleet_drained
• warn:runner:aged_out — silent past the liveness window without a goodbye; once per offline episode
• warn:runner:log_spool_overflow — a step's log upload backlog dropped chunks under a gap marker

Failure modes

• Server unreachable — an idle runner retries its poll with jittered backoff (1 s → 60 s); an executing runner keeps the step running through renewal failures up to the lease TTL, spooling logs on disk meanwhile.
• Lease fenced mid-job (ABANDON, or an unauthenticated renewal) — the step is stopped and nothing is submitted; the server has already settled the attempt without this runner (failed fast as runner_lost after the lease TTL, or — for an offer this runner never acked — re-delivered as a new attempt).
• Killed without drain — no goodbye, so the registry ages the runner out after 3 minutes (warn:runner:aged_out) and the in-flight lease expires: the attempt fails fast as runner_lost (step system_error, change error, log sealed) for a human to retry with opschain runs retry. Nothing is lost silently; the two cases are on opschain docs fleet-queue.
• Log backlog over budget — oldest chunks are dropped under an explicit gap marker in the step log; the result itself is unaffected.
• Server refuses the runner outright (failed precondition, e.g. a revoked identity) — the process exits with server refused this runner rather than retrying forever.

Related pages

• opschain docs cli/runner — the full opschain runner flag reference.
• opschain docs runner-registry — the server-side registry: groups, tokens, liveness, drain and forget.
• opschain docs runner-executors — what actually runs the step.
• opschain docs fleet-queue — how queued work is matched to runners.