12.4 Releases, stages and the rollover — hire to retire

packs:agentspacks:agent-governance

After this you can: walk Felix from hire to retire: sign a release, climb the stage ladder, run instances on lanes, roll a release over without stopping, then retire.

Before you start — tick each one:

• Lesson 12.3 — Evidence: activity, trust review, transcripts (opschain learn 12.3)

In the GUI: Governance → Agents shows the release catalog and the stage ladder, and each agent's detail names the release and lane its instance runs; a held promotion or rollover waits in Reviews & approvals.

Felix was hired in lesson 12.2 and has been patching prod under supervision. Now a new build is ready. There is exactly one way an agent changes on this platform: a signed release, promoted one rung at a time up the estate's stage ladder, tried beside production on a lane, and rolled into the serving lane by a platform-instructed rollover — then the old release is retired. This lesson walks that whole arc with the output of every step and the events it lands. The release catalog and the rollover need packs:agent-governance; the model behind every step is opschain docs agent-releases.

The starting state: Felix's stable lane runs two instances of felix@2.0.0, promoted to prod.

1. One agent, many instances, one release each

An agent is the logical identity — the code its service account, owner, grants and recommendations attach to. Its instances are its running processes: every process started from a state directory with the same code enrols another instance, with its own keypair, bound at enrolment to the release it runs and labelled with a lane. Work addresses the agent; the platform picks one instance — the lane the request names (default stable), then the least-loaded connected instance of that lane whose release stage passes admission.

opschain admin agent instances felix

RELEASE      LANE    INSTANCE                              MEMBER        STAGE  ONLINE  LAST SEEN             STATE
felix@2.0.0  stable  01a1aecc-175c-7572-b3c4-49d2dd1dec76  ocr_FeodwmG…  prod   ✓       2026-11-06T13:58:04Z  ready
felix@2.0.0  stable  01a1aecc-1795-7c21-9a18-0101dabdc665  ocr_JWhTz8U…  prod   ✓       2026-11-06T13:58:05Z  ready

STAGE is the release's current rung, cached on the instance and rewritten on every live instance when the release moves. A new build therefore runs beside the one serving, never instead of it.

2. The ladder is the estate's

A release's stage says where it is trusted to act. Stages form an ordered ladder the estate configures; the default is dev → test → prod with prod the one governed rung:

opschain admin agent-release stages

RANK  STAGE  GOVERNED
1     dev    no
2     test   no
3     prod   yes

An estate with its own lifecycle replaces the ladder (--set dev,uat:governed,preprod:governed,prod:governed) or renames a rung in place (opschain admin agent-release rename-stage uat test); a change that would remove or reorder a rung a release sits on, or strand a floor or grant condition, is refused by name. Either write lands info:agent_release:ladder_changed. Past the top rung sits the implicit retired state — never a rung.

3. Sign the release

Signing is local, verification is server-side. Register a signing key once — the private half is written to a file and never reaches the server:

opschain admin agent-release keygen --name release-pipeline --out ./felix-signing.key

registered signing key 019a0d7e-… (release-pipeline)
fingerprint: SHA256:…
private key written to ./felix-signing.key — keep it offline; it never reaches the server

The event is info:agent_release:key_registered; opschain admin agent-release keys lists the registry by fingerprint. Now register the release — a <name>@<version>, the frozen manifest document, the content-addressed artifact (a git sha or image digest) and an Ed25519 signature over all of it. Sign in the pipeline and pass the signature file, or let create sign inline from the key file:

opschain admin agent-release sign --name felix --version 2.1.0 \
  --artifact sha256:9f2c… --manifest felix.json \
  --key-file ./felix-signing.key --out felix-2.1.0.sig
opschain admin agent-release create --name felix --version 2.1.0 \
  --artifact sha256:9f2c… --manifest felix.json --type service \
  --key-id 019a0d7e-… --signature-file felix-2.1.0.sig

created agent release felix@2.1.0 (id 019a0d8a-…)
artifact: sha256:9f2c…
signed by: SHA256:…
stage: dev

The server verified the signature against the active key before the row landed (info:agent_release:created); an unsigned, tampered or non-active-key submission refuses by name and lands warn:agent_release:refused. --type service froze the type into the signed content, so an instance enrolling from this release cannot be misdeclared, and a disclosures object in the manifest becomes authoritative for every instance that enrols from it. Releases are immutable: a fix is a new version, never an edit. opschain admin agent-release list --name felix now shows both versions with their STAGE and STATUS.

4. Promote one rung

Promotion moves a release exactly one rung up, re-verifying the signature first — never skipping, never demoting:

opschain admin agent-release promote felix@2.1.0

promoted felix@2.1.0 to test

info:agent_release:promoted records it, and the move is a durable promotion attestation on the release — who requested, who decided, when — that opschain admin agent-release show felix@2.1.0 renders.

5. Try it in the shadow lane

Mint another one-use token (lesson 12.2) and start a third instance of Felix from the new release, on the shadow lane:

opschain admin agent-token create --project limebank --code felix --ttl 1h --max-uses 1
opschain agent --server https://opschain.example.com --token <token> --code felix \
  --dir ./felix-shadow --release felix@2.1.0 --lane shadow
opschain admin agent instances felix

RELEASE      LANE    INSTANCE                              MEMBER        STAGE  ONLINE  LAST SEEN             STATE
felix@2.0.0  stable  01a1aecc-175c-7572-b3c4-49d2dd1dec76  ocr_FeodwmG…  prod   ✓       2026-11-06T13:58:04Z  ready
felix@2.0.0  stable  01a1aecc-1795-7c21-9a18-0101dabdc665  ocr_JWhTz8U…  prod   ✓       2026-11-06T13:58:05Z  ready
felix@2.1.0  shadow  01a1aecc-17d0-7a1c-8c2e-5b1f0c6d9e01  ocr_Q2mRv7k…  test   ✓       2026-11-06T13:58:05Z  ready

Enrolment verified the release — unknown, unsigned, tampered, wrong type, below the estate floor, retired or revoked each refuse by name with warn:agent_release:enrolment_refused. The shadow lane receives the same dispatch as stable and can never write: every change it requests is forced to the advise tier, and its recommendations carry lane: shadow, linked to the stable lane's proposal for the same incident as a duplicate of one card — never a second change:

opschain recommendations list limebank --agent felix --filter 'lane=shadow'

That is "test 2.1.0 beside 2.0.0 while production serves", without a second project.

6. Stages as policy on scope

The stage is a principal attribute of every running instance, evaluated on every decision — not only an enrolment check. Three controls build on it, from the outside in:

• The estate floor — agents.release_stage in the server configuration: a declared release below it cannot enrol at all.

• The environment floor — release_stage_floor on an environment. Every action, run or governed read whose target sits under it is refused when the agent's release stage is below the floor (or when the agent enrolled without a release), before any grant is consulted:

  opschain nodes update limebank/environments/prod --release-stage-floor prod

  The write lands info:agent_release:floor_changed; a refusal is a 422 naming the environment, the floor, the release and its stage, and lands warn:agent:release_stage_refused on Felix's path.

• The grant condition — opschain admin trust grant … --release-stage prod admits only instances whose release sits at or above that rung, so a grant over a production scope can demand a production-stage build explicitly.

Floors and conditions compare by ladder position, so a custom ladder works unchanged. The outcome: a dev build can never act on production nodes by policy, not by convention.

7. Promote into the governed rung

The next promotion is into prod. On an estate carrying the approval fabric it does not complete directly:

opschain admin agent-release promote felix@2.1.0

promotion of felix@2.1.0 to prod (governed) is held for approval (promotion 019a0d9c-…) — approve or reject it through the approvals inbox

info:agent_release:promotion_held records the hold, with a content fingerprint binding every approval to the release identity, artifact, stages and key the approver saw. Approvers resolve from --approver, else agents.approvers; a governed promotion is never approver-less. Decide it from the one inbox:

opschain approvals list
opschain approvals approve <promotion-id>
opschain admin agent-release show felix@2.1.0

Quorum completion moves the stage (info:agent_release:promoted); rejection or an expired quorum refuses it (info:agent_release:promotion_rejected) and the stage never moves. show now lists both promotions with who requested and decided each. The shadow instance's STAGE reads prod at once — the cache refreshes on every connected instance of the release without re-enrolling.

8. Roll the stable lane

The rollover is the platform verb that changes a running agent. It refuses unless the target release's stage satisfies every floor the agent's scopes require — the estate's, each reached environment's release_stage_floor, each grant's release_stage condition — and unless the release is still serving. Roll one canary first:

opschain admin agent rollover felix --to felix@2.1.0 --strategy canary=1

Because prod carries a governed floor that Felix's grant reaches, the rollover lands held in the same funnel as the promotion (the shadow lane, which never writes, is never held):

rollover 01a1aecc-… of limebank/felix lane stable to felix@2.1.0 is held for approval — approve or reject it through the approvals inbox

Approve it from the inbox (opschain approvals approve <rollover-id>) and info:agent:rollover_started records the start; rejection or an expired quorum refuses it. Without the fabric — or on a lane with no governed floor in reach — the verb starts the rollover directly and prints it:

rollover 01a1aecc-… of limebank/felix lane stable: felix@2.0.0 → felix@2.1.0 (canary=1), 2 instance(s) targeted, status rolling

Per instance the platform drains it (in-flight work finishes, nothing new is dispatched — info:agent:instance_drained), instructs the restart over the instance's own channel, and resumes dispatch once the instance re-enrols from the new release, verified at enrolment (info:agent:instance_restarted). The instance itself logs platform: restart from release felix@2.1.0 instructed (exec mode); draining, then restarting: on a host it fetches the artifact by its content-addressed reference from --artifact-store, verifies the digest and re-executes itself; in a container it exits with code 75 for the platform's restart policy to bring it back on the image the release names. State directory and keypair are kept; the restarted process re-enrols with the token it was started with. With --drain-only the platform drains and reports, you restart the instances yourself, and the rollover completes as they re-enrol.

The canary is now the one instance on 2.1.0 and the rollover holds (info:agent:canary_held):

opschain admin agent instances felix
opschain admin agent rollover continue felix
opschain admin agent rollover finish felix

rollover 01a1aecc-… of limebank/felix lane stable: finished (felix@2.0.0 → felix@2.1.0)
the source release can now be retired: opschain admin agent-release retire felix@2.0.0

continue rolls the rest (info:agent:rollover_continued); abort would have returned the canary; rollback returns every instance after a continue; finish closes the rollover and offers the retirement — printed, never run (info:agent:rollover_finished carries the retire_offer). Every step is an attestation on the release (agent-release show) and an info:agent:rollover_* event on Felix's path, all under ONE correlation chain; a refusal at any step is warn:agent:rollover_refused with a typed reason.

--strategy blue-green restarts nothing: a second set of instances comes up from the new release in a green lane (--lane stable-green; the licence's agents headroom for the overlap is checked first), rollover continue flips dispatch to them in one write (info:agent:rollover_flipped), rollback flips back and finish retires the old lane — opschain docs cli/admin-agent-rollover.

9. Retire the old release

Beyond the top rung sits retired: no new enrolment and no rollover to the release, while any instance still running it is untouched:

opschain admin agent-release retire felix@2.0.0

retired felix@2.0.0 (stage prod): it accepts no new enrolment or rollover; running instances are untouched

info:agent_release:retired records it on the platform path. The other end of the ladder is revoke — opschain admin agent-release revoke <name@version> --reason … halts every instance running the release with the typed release-revoked reason — which is a kill switch, so it is lesson 12.5's. Both are one-way, both attested on the release, and neither touches the signing keys: rotation stays opschain admin agent-release key-status.

Felix has been hired, trusted, changed under policy and its old build retired, and nothing stopped serving. Retiring Felix itself — drain, halt, revoke, with its history intact — is the next lesson.

Check yourself

• What does the release stage floor refuse, and what happens to a running instance when you roll a release over?
• In order, which events does one canary rollover land on Felix's path — started, drained, restarted, held, continued, finished — and which single id joins them all so that opschain chains show <correlation-id> renders the rollover as one chain?
• Does opschain nodes show report the release stage floor you set on the environment — the info:agent_release:floor_changed event the Academy's release-stage-floor exercise reads?
• Does opschain admin agent instances felix show every instance on the new release, and opschain admin agent-release show the rollover's attestations under one correlation id — info:agent_release:created, info:agent:rollover_started, info:agent:rollover_finished, the sequence the rollover exercise reads? Remember: re-enrolling an instance from a release needs a fresh enrolment token, and a release below an environment's floor cannot roll into a scope that reaches it.
• Verified version: opschain academy runs the release-stage-floor and rollover exercises against this lesson.

What you learned

• An agent is one identity with many instances, each bound to a signed release and a lane; the new build runs beside the old one in the shadow lane, forced to advise.
• A release is signed locally, verified server-side, registered at the ladder's first rung and promoted one rung at a time; the governed rung is held in the approvals inbox.
• Stages are policy on scope: the estate floor, the environment's release_stage_floor and the grant's release_stage condition keep a dev build off production by policy.
• The rollover is the one way a running agent changes — drain, platform-instructed restart, re-enrol, resume — canary first, attested on the release under one correlation chain.
• Retire closes the old release without touching its running instances; revoke is the kill switch.

Next: lesson 12.5 — Kill switches and offboarding (opschain learn 12.5), or back to your track (opschain learn).