Skip to main content
Version: v5.7.0

6.3 Fleets, groups and placement

Included from Basic tier

After this you can: start a fleet, use the example runner image, group runners by attributes, place work on them, read the fleet view and make everyday fleet moves.

Before you start — tick each one:

  • Lesson 6.2 — Your first runner (opschain learn 6.2)

In the GUI: the Runners panel in the Operations area shows the groups and their members with derived state, the drain/resume/forget verbs, and the queue backlog with each entry's placement verdict; the Overview's group cards carry per-group demand. New group… and each row's Edit group… open the group dialog — description, the Exclusive switch, the attributes as key/value rows offered from the registered vocabulary, the bindings one path per line — and the Placement attributes panel below the queue registers keys and their allowed values; both render the server's refusals verbatim.

Lesson 6.2 enrolled one runner. This lesson turns that into a fleet: several members, grouped and attributed so work lands where it should, read through the fleet view, and moved with the everyday verbs. Every fleet verb needs administrator rights (lesson 6.2).

1. Starting a fleet by hand​

Three moves put a runner on the fleet: mint a registration token, start the runner with a state directory of its own, and watch it appear in the fleet view. Minting needs admin rights, and the token value prints exactly once — capture it:

opschain login --server https://opschain.example.com
TOKEN=$(opschain admin runner-token create \
    --server https://opschain.example.com --group iac)

Your own login session is the credential (automation uses a service-account API key in OPSCHAIN_ADMIN_TOKEN; the server's static admin token works only on local-lane compositions — a full-auth server refuses it).

Minting for a new group name creates the group (subject to the licence's group entitlement), so there is no separate "create a group" step. Now start a member — in the foreground, one terminal per member:

opschain runner --server https://opschain.example.com --token "$TOKEN" \
    --dir ~/opschain-runners/iac-1 --name iac-runner-1
  • Every member gets its own --dir state directory, never shared — it holds the durable identity lesson 6.2 described. --token is needed on the first start only; restart with the same command minus --token and the saved identity re-attaches the same fleet member.
  • One token enrols any number of members within its TTL (default 1h; cap redemptions with --max-uses) — repeat the start with a fresh --dir and --name for each.
  • Stopping is graceful: Ctrl-C (SIGTERM) drains — the runner finishes its in-flight job, says goodbye (the fleet view reads it stopped immediately) and exits. A second signal stops hard.

The containerised equivalent runs the runner image with the same ingredients as environment variables and the state directory bound to /var/opschain/runner:

docker run -d --name iac-runner-2 --restart unless-stopped \
    -v "$HOME/opschain-runners/iac-2:/var/opschain/runner" \
    -e OPSCHAIN_SERVER=https://opschain.example.com \
    -e OPSCHAIN_RUNNER_TOKEN="$TOKEN" \
    -e OPSCHAIN_RUNNER_NAME=iac-runner-2 \
    limepoint/opschain-runner:local

docker stop delivers the same SIGTERM goodbye, and docker start re-attaches the kept identity. One container gotcha: from inside a container, localhost is the container itself — point OPSCHAIN_SERVER at a URL the container can reach (host.docker.internal for a server on the docker host).

2. The example runner image​

examples/opschain-runner-image builds limepoint/opschain-runner:local — a containerised runner carrying a pinned, checksum-verified IaC toolchain (tofu, terraform, ansible-playbook, kubectl, aws, az, gcloud, gh, python3, git — every tool a build-arg layer defaulting ON; trim with --build-arg WITH_<TOOL>=0) plus the opschain binary, paired with the common action packs (lesson 2.4):

cd examples/opschain-runner-image
OPSCHAIN_PACKAGE=/path/to/opschain_v5.7.0_linux_arm64.tar.gz ./build.sh

Start members from the built image with the docker run form in section 1.

Three principles the image demonstrates, worth copying into your own:

  • Tools come from the image, never from run time. A step needing a missing binary fails loud rather than installing things mid-run; tool version bumps are build args — change one place, rebuild.
  • Credentials never bake into layers. Secrets reach steps at run time via vault-backed properties, not the image.
  • The runner binary comes from an official released package. Customer image builds feed build.sh the released opschain_<version>_linux_<arch>.tar.gz — never a source build (DEV_BUILD=1 exists for LimePoint development only).

Vault credentials ride the runner, not the settings​

When templates or actions call the secret vault (SecretVault.get_password), the provider configuration comes from the secrets_providers section of the runner's own config.yaml — one entry per provider the estate registered with resolution: runner — projected into every hostexec step. Write it before you start the fleet (opschain docs runner-executors; a host still carrying the retired OPSCHAIN_VAULT_* variables migrates them once with opschain runner migrate-vault-env). They are deliberately NOT settings: a vault credential stored in settings would need the vault to decrypt itself. The runner's own token stays with the runner: each step receives a short-lived child token minted for it at dispatch and revoked when the step finishes, so action code never holds the fleet credential — and if your vault refuses child minting, steps fall back to the runner's token with a named notice at the top of the step log. Which fleet a node's work runs on — and therefore which vault serves it — is the runner.group setting, so "this project uses that vault" is a routing decision, not a credential one. The MintPress guide (opschain learn pack-mintpress) covers the SDK fleet's vault environment and routing end to end.

3. Groups and placement​

Runners belong to groups; work lands on groups. Three placement mechanisms, resolved once per step at enqueue:

  • Name the group. A change can target one directly:

    opschain runs create change acme/environments/dev/assets/web \
        --action deploy --metadata runner.group=iac

    Node settings carry the same runner block to make a subtree's default (leaf-most node wins), so "everything under prod runs on the prod runners" is configuration, not per-change discipline.

  • Select by attribute. Admins register attribute keys (location, network ship reserved) and tag groups with values; work then demands selectors ({"runner": {"selectors": {"location": "dc1"}}} in metadata or node settings) and any matching group's runners qualify. Unregistered keys or values are refused at write time, not silently never-matched.

  • Bind a subtree. --binding /projects/acme on a group restricts that subtree's work to the bound groups' runners — a hard zone, and bound work never escapes it. --exclusive is the converse: the group's runners take only work that names or selects them.

opschain admin runner-attributes set location --values dc1,dc2
opschain admin runner-group set iac --exclusive --attribute location=dc1
opschain admin runner-group list
NAME     EXCLUSIVE  ATTRIBUTES    BINDINGS  DRAINING
default  false      -             -         false
iac      true       location=dc1  -         false

opschain admin runner-group delete iac is refused while runners or redeemable tokens still reference the group — forget the runners and let the tokens expire first.

The same two writes sit on the Runners page: the Placement attributes panel registers location with its allowed values, and the group dialog (New group…, or Edit group… on the iac row) sets the exclusive flag, the location=dc1 attribute and any bindings. Saving the dialog replaces the whole definition, exactly as runner-group set does — re-state everything the group should keep.

One more routing rule rides beside placement: an actions.rb change holds one runner for its whole run (run-scoped execution; actions.rb is only applicable to existing MintPress customers and those using the MintPress SDK or the packs:mintpress feature). Its first step pins the run to whichever eligible runner takes it; every later step queues behind that same runner and shares its workspace, so a file one step stages on disk is there for the next — upstream's single change pod, preserved. Declarative/YAML runs spread per step as usual, and {"runner": {"execution_scope": "run" | "step"}} in metadata or node settings overrides either default. If the pinned runner dies mid-run the run fails with an event naming it (lesson 6.1's lost-runner cases) — steps are never silently re-spread. By default the pinned runner executes the run one step at a time; give the same runner block a max_parallel_steps (a positive integer, e.g. {"runner": {"max_parallel_steps": 4}}) and it executes up to that many of the run's parallel steps at once in the one shared workspace — upstream's in-pod parallelism, preserved — while still never taking other runs' work mid-run (opschain docs dispatch). A runner can cap that to its own resources: start it with --max-parallel-steps N (or OPSCHAIN_RUNNER_MAX_PARALLEL_STEPS) and it executes at most min(the run's setting, N) steps at once, logging one honest notice on the run naming both numbers whenever the clamp binds; the declared capacity shows beside the executors in admin runners list/runner show.

Work nobody can take stays truthfully queued with a reason (no registered runner satisfies the placement requirement …) rather than hanging on a phantom lease, and opschain admin runner-group demand <name> reports {queued, running} per group — the autoscaler signal: queued counts work the group could satisfy, even with zero members online:

opschain admin runner-group demand iac
{
  "group": "iac",
  "queued": 3,
  "running": 1
}

opschain admin queue list shows every queued and leased entry with its owning work item — a change, a workflow run, or a node background task (a Regenerate Actions job, which the activities feed does not show) — plus the target node path and correlation chain. A stuck entry terminates with opschain admin queue terminate <run-id> (cancelling its owner through the owner's own lane, with an audit event naming you), and opschain admin queue sweep clears leftover entries whose owner already finished.

4. The fleet view​

opschain admin runners list
GROUP    NAME                   ID                                    ATTRIBUTES  ONLINE  LAST SEEN             JOB  STATE
default  default-runner-1       ocr_m9X6WFUT7QWg5VfO616Qk4XNdbEG6qX2  -           ✓       2026-10-12T09:51:12Z  -    ready
default  default-runner-2       ocr_7cg70dCQOW6r7kYhfibG8nmxPY0MDVaQ  -           ✓       2026-10-12T09:50:53Z  -    ready
iac      opschain-runner-iac-1  ocr_o5jMGidhzIdZA7W1GTk1hyDr_hw9yfPn  -           ✓       2026-10-12T09:51:16Z  -    ready
iac      opschain-runner-iac-2  ocr_VrT1gtRQMEbYH7rMBThTtd4LkMrVEM4m  -           ✓       2026-10-12T09:51:06Z  -    ready

STATE is the lifecycle: ready, draining (hold in force, finishing in-flight work), quiesced (hold in force, idle), stopped (clean goodbye — immediate, versus the three-minute age-out for a crashed runner), - (offline, nothing held). Group-scoped holds read quiesced (group) / draining (runner+group).

opschain admin runner show <runner-id> is the detail surface — worth a look at least once (trimmed here to the highlights):

ID                ocr_m9X6WFUT7QWg5VfO616Qk4XNdbEG6qX2
NAME              default-runner-1
GROUP             default
STATE             ready
…
EXECUTORS         hostexec,containerexec
OS/ARCH           darwin/arm64
…
HOSTNAME          runner-host-1.example
CPUS              10
…
RUNNER VERSION    v5.7.0
…
STATE DIR         ~/.opschain/runners/default-1
OBSERVED REMOTE   [::1]:51354
KEY FINGERPRINT   SHA256:o6U8mWokVl9xVhaBMKvkTiF4UvRCNotZOwtEgNNSBvI

Identity, lifecycle, current jobs, the host inventory it self-reported at registration, the server-observed peer address, and the key's fingerprint (the raw public key is never rendered).

5. Everyday fleet moves​

opschain admin runner drain <runner-id>        # finish in-flight, take nothing new
opschain admin runner resume <runner-id>
opschain admin runner-group drain iac          # the same, group-wide
opschain admin runner-group resume iac
opschain admin runner forget <runner-id>       # remove a stale registration row

Each verb answers with the flag it set, and records who set it (info:runner:drain_changed, info:runner:forgotten):

opschain admin runner drain ocr_o5jMGidhzIdZA7W1GTk1hyDr_hw9yfPn
{
  "draining": true,
  "id": "ocr_o5jMGidhzIdZA7W1GTk1hyDr_hw9yfPn"
}

The step boundary is the drain boundary: draining never cancels in-flight work, it just stops offering new work — which makes drain-replace-resume the safe patching loop for runner hosts. forget is cleanup, not revocation (the identity may re-register, and an online or job-holding runner is refused — drain and stop it first).

6. Provisioning runners​

Hand-started workstation fleets are the dev loop; for real environments there are two provisioning lanes, each documented end to end (customer installs and image builds always start from official released opschain packages, never a source build):

  • A hostexec runner on a host you own — the production hardening checklist: a dedicated user and a 0700 state directory, installing the binary from the released package, minting a registration token (opschain admin runner-token create), first-start enrolment, a systemd unit (and macOS launchd note), fleet verification, and the lifecycle (drain, upgrade in place, decommission).
  • The opschain-runner image (examples/opschain-runner-image, default image name everywhere: limepoint/opschain-runner) — the containerised lane of section 2, carrying the full pinned IaC toolchain (OpenTofu, Terraform, Ansible, kubectl, AWS/Azure/GCP CLIs, gh) as build-arg layers defaulting ON; trim per fleet with --build-arg WITH_<TOOL>=0.

Check yourself​

  • How does a change choose a runner group, and what does opschain admin runner-group demand report for a group whose work is queued with no member online?

What you learned​

  • One token enrols any number of members; each keeps its own state directory, and Ctrl-C drains gracefully.
  • The example runner image bakes tools in and credentials out; vault credentials ride the runner's environment, never settings.
  • Work lands on groups by name, by registered attribute selectors, or by subtree binding; unsatisfiable work queues with a reason.
  • admin runners list and admin runner show are the fleet view; STATE is the lifecycle vocabulary.
  • Drain, resume and forget are the everyday moves, and the step boundary is the drain boundary.

Next: lesson 6.4 — Hosts without a runner: brokered access (opschain learn 6.4), or back to your track (opschain learn).