Skip to main content
Version: v5.7.0

The agent plane

packs:agents

The agent plane runs AI (and other automation) agents against OpsChain under explicit, auditable governance. An agent is outbound-only: it registers once, holds an agent-initiated channel open to the server, and serves its capabilities through that tunnel — it binds no listener and needs exactly one egress and one credential. Everything the agent is allowed to do is decided server-side from trust grants; the agent's payload can never name a tier, waive a gate or supply a self-computed identity.

Two separable control surfaces: opschain admin agent manages whether an agent runs (drain/halt/revoke/forget); opschain admin trust manages what it may do (grants, assignments, approvals). See opschain docs cli/admin-agents, cli/admin-agent and cli/admin-trust.

Naming note: GET /api/trust_store is the TLS certificate trust store (public CA material, the certificates table) — a different subsystem that merely shares the word "trust".

Registration and identity​

An agent enrols by redeeming an agent registration token — minted by opschain admin agent-token create (cli/admin-agent-token-create), scoped at mint time to kind agent, a project, a fleet group, optionally a fixed agent code, with a TTL and max-uses. Possession of the token cannot enrol outside its scope. admin agent-token list lists live tokens.

Enrolment registers the agent's Ed25519 keypair (the runner credential ladder — no long-lived bearer tokens): the identity persists in the agent's state dir as agent.json plus the private key in its own 0600 agent.key. Sessions are short-lived tokens obtained by signing a server challenge with that key, renewed automatically before expiry. The server signals an unrecognised agent with a re-register code, and the agent re-registers with the same durable keypair when it still holds a registration token. An identity registered with one server refuses to be presented to another (a named error pointing at --dir).

Agents are rows of the unified fleet registry (kind agent, alongside runners), identified by member id, with a per-project-unique code and a fleet group. The code names the agent — the logical identity; each enrolled process is one of its instances (a fleet member with its own keypair, bound to the release it runs, labelled with its lane), and a second state directory with the same code enrols a second instance — see opschain docs agent-releases for instances, lanes and the release rollover. Registration and session establishment are gated on the packs:agents entitlement (or the bare agents feature it supersedes; see opschain docs entitlements), with the licence's agents limit bounding concurrently connected instances (existing connections stay live on lapse or a shrunk limit; a new registration or reconnect refuses loudly with a named error plus a warn:licence:feature_refused event). The manifest may declare a type (typevocabulary: service or ai): an ai-type enrolment additionally hard-depends on packs:agent-governance

  • packs:ai-agents, refused naming every missing pack (opschain agent dev --type ai and the SDKs' type options declare it). Registration is also refused while the agent plane is halted or the system is quiesced.

The service account​

Every registration mints (find-or-create) a durable service-account principal: a users row with the distinct auth_provider = 'agent' identity kind, the deterministic username svc-agent-<project>-<code>, no password and no interactive login — its only credential is the channel's keypair/session handshake, and it can never start a session or mint an API key. It survives re-registration and key rotation, and is addressable by policy assignments like any principal. On the identity surfaces the row badges agent beside the human kinds, carries the fleet cross-link (agent: {project, code, member_id} on the user listings, with service_account_id/_username on the fleet detail pointing back), rides the open-set provider=agent filter, and is dormant once its agent is revoked or forgotten — out of pickers by default, visible on request, exactly like a dead-provider shadow. It is never managed through the human user lifecycle, nor through the service-account verbs that manage 's human-minted automation principals (auth_provider = 'service_account', opschain docs cli/admin-service-accounts) — fleet verbs are its lifecycle.

The owning human or team​

Every agent names its owner as first-class data: owned_by on the manifest — declared at registration (opschain agent dev --owned-by, Go SDK OwnedBy, Python owned_by=) and validated against the identity catalog as a known user or group; an unknown name refuses registration. Admins override it with opschain admin agent owner <agent-id> --owned-by <name> (audited as info:agent:owner_changed); a manifest that declares no owner preserves the stored one, so the override survives re-registration. The owner shows on the fleet views and the trust review, and rides the agent's warn events (warn:agent:halted, warn:agent:request_refused) so escalations route to the owning team. Ownership is accountability only — it never derives a privilege ceiling or floor, and offboarding a user with opschain admin users reassign-automations moves the agents they own alongside their triggers and schedules.

Trust presumes an owner: a trust grant (or raise — every raise is a grant write) naming supervised or autonomous for an unowned agent refuses by name at grant time, with the one-verb fix in the refusal body — assign the owner, then retry the grant. advise carries no such requirement. Existing grants are untouched: an agent that lost or never had an owner keeps working and is instead flagged as unowned on the fleet list, the agent sheet and the project agents view, each of which also filters to unowned agents. A composition without an identity backend cannot hold a validated owner, so the requirement does not apply there. The GUI assigns and clears owners in place (the agent ops sheet and the project agents view, under the same role that may drain or halt), riding the owner override — on v2, AgentService/SetAgentOwner, the POST /api/agents/{id}/owner twin.

Signed releases — enrolling from an AgentRelease​

An agent ships as a signed AgentRelease: an immutable <name>@<version> row holding the frozen manifest document, the content-addressed artifact identity (git sha or image digest) and an Ed25519 signature over both, verified against the registered signing-key registry (opschain docs cli/admin-agent-release). Releases promote one rung at a time up the estate's stage ladder (default dev → test → prod); the signature re-verifies at every promotion, and on a fabric-carrying estate a promotion into a governed rung is held in the same approval funnel governed change enters. The stage then travels with the instance as a principal attribute: an environment's release-stage floor and a grant's release_stage condition decide where that build may act (opschain docs agent-releases, "Where a stage may act").

A manifest that names a release (release_name/release_version; Go SDK ReleaseName/ReleaseVersion, Python release="name@version") is verified at registration and refused by name when the release is unknown, unsigned, tampered (the stored content no longer matches its signature), of another declared type (a signed release cannot be misdeclared), or below the estate's agents.release_stage floor. A verified enrolment binds the release to the instance: the fleet detail and the trust review answer "which release is this instance running, at which stage", and two instances of one agent may run two releases — the shadow lane beside stable, the canaries of a rollover. An agent that names no release enrols exactly as before; a retired or revoked release refuses enrolment by name. Changing the release an instance runs is opschain admin agent rollover (opschain docs agent-releases).

Manifest disclosures and the type attestation​

The manifest discloses what the agent is and does. Every type may declare a purpose — the "what is this for" label. The ai type must additionally disclose, at registration:

  • model identification — model_provider, model_name (and an optional model_version);
  • adapter/SDK identification — adapter (optional adapter_version), e.g. opschain-claude-adapter;
  • tool surface — tool_surface, the estate tools the agent wields (e.g. request_lane, docs, asker_reads), stored sorted and deduplicated;
  • custody mode — custody_mode, one of the two modes: agent-host or vault-to-host. Any other value refuses at the manifest edge; the credential itself never appears in the manifest.

Declare them with opschain agent dev --purpose/--model/--adapter/ --tool-surface/--custody, the Go SDK Disclosures option, or the Python disclosures= dict; the Claude adapter attaches its own disclosures automatically when the agent declares none and ships without a release. Disclosures render on the fleet detail (disclosures, with disclosures_from_release when frozen — see below), the trust review and opschain admin agent show (PURPOSE/MODEL/ADAPTER/TOOL SURFACE/CUSTODY/ DISCLOSED BY).

The type attestation binds the declared type to the enrolment: an ai-type manifest missing any required AI disclosure refuses by name, and a manifest of any other type declaring them refuses as a type contradiction — both land the audited warn:agent:type_attestation_refused event. The same contradiction is detected at the adapter seam: adapter-transit transcripts arriving from an agent not declared ai refuse and land warn:agent:adapter_transit_refused. Where the agent enrols from a signed release whose frozen manifest document carries a disclosures object, the frozen set is authoritative and immutable: the live views re-read it from the release row, and a registration declaring a differing set refuses by name (change what an agent disclosed by shipping a new release). What stays honestly out of technical reach: the platform cannot classify arbitrary customer code that never transits its seams — a self-authored LLM agent misdeclared as service that avoids the platform adapter is an audit and licence compliance matter (posture), not a technical enforcement.

The manifest and capabilities​

At registration the agent declares its manifest: spec version (current 1.0; the server bounds accepted versions between a floor and a target), agent name/version, and its capabilities as name → JSON config. The runtime supplies the mandatory five built-ins itself — healthz, readyz, info, metrics (Prometheus), logs (a bounded 500-line ring) — so a customer agent only ever writes its real endpoints. Optional capabilities (chat, config, x-* namespaces) are served by an in-process handler or reverse-proxied to a supervised local process (--target, loopback only; opschain agent can also supervise and restart the child itself, handing it its port via OPSCHAIN_AGENT_PORT/PORT). chat defaults to {"streaming":true,"timeout_seconds":120}.

The channel and the tunnel​

The channel is agent-initiated: a Connect bidi stream (h2c against a plain-HTTP server), falling back automatically to long-poll (--transport auto|stream|poll) — the long-poll carriage is fully supported, not degraded. Reconnects use exponential backoff with full jitter (1 s → 60 s); the heartbeat (the stream's 60 s keepalive, or each long-poll) refreshes the agent's durable channel-ownership record, so online status is estate-wide: every server replica answers GET /api/agents, the detail view, the project-plane rows and the governance dashboard's fleet counters from the same record — a connected agent reads online (with its carriage) whichever replica serves the read, a record not refreshed within its two-minute freshness bound (twice the slowest heartbeat cadence) reads offline honestly, and a clean shutdown's goodbye marks the agent stopped everywhere at once. A silent agent ages out.

Inbound, the server materialises every registered agent's declared surface under /api/projects/{project}/agents/{selector}/{capability} (GET and POST) — the ordinary API, ordinary caller authorisation: the decision walks the agent's governance path /{project}/agents/{code}, with read sufficing for the five minimum surfaces and execute required for every interactive capability (chat, config, x-*). Deny renders 404, so an unauthorised caller cannot distinguish a denied agent from an absent one, and an agent cannot be probed for undeclared surfaces. Selector resolution order: exact member id (one instance, addressed directly), then per-project code — the agent, routed to ONE of its instances: the lane the request names in X-Opschain-Agent-Lane (default stable), then the least-loaded connected, readyz-green instance of that lane whose cached release stage passes admission; a shadow-lane instance beside it receives the same dispatch with its answer discarded — then group name (routed to one healthy member — channel-connected, readyz-green, least-in-flight). The answering member is named in the X-Opschain-Agent-Id response header, the exchange in X-Opschain-Exchange-Id. Proxied exchanges are bounded: a 30 s default per-exchange deadline (a capability config may raise it, never past the server's 10-minute hard ceiling), at most 16 concurrent exchanges per agent, 8 MB request bodies (413 above), and a chunk-credit window that caps per-exchange server memory. An edge refusal is audited as warn:agent:tunnel_denied. Chat exchanges are captured as transcripts (log segments keyed to the agent, one sealed segment per exchange) and read back at …/chat_transcripts/{exchange}. GET …/chat_transcripts (no exchange id) lists the caller's own sealed exchanges with the agent, newest first — exchange id, started/sealed timestamps and a redacted first-line preview per row (?limit= default 50, capped at 200; ?cursor= keyset paging) — filtered server-side by the asker principal recorded on the exchange, never anyone else's; authorisation is the same read decision as the per-exchange transcript read, and an aged-out exchange leaves the list because the evidence itself is gone. AI-type agents additionally ship adapter-transit transcripts — every model interaction crossing the Claude adapter — over AgentTranscriptService/AppendAdapterTranscript; the server redacts each line (secret-shaped values, credentials, email addresses) before persistence, seals the exchange on the final batch (audited info:agent:adapter_transcript), and serves it back at …/adapter_transcripts/{exchange} (see opschain docs ai-agents).

The request lane: agents calling the platform API​

Call on the agent RPC is the agent's one way to call the platform API: GET and POST only, /api/... paths only, bodies bounded. The server dispatches the call in-process through the composed HTTP surface under the agent's service-account principal — so authorisation, request logging and tier enforcement apply exactly as for any caller — and writes the R16 audit event itself (info:agent:request, or warn:agent:request_refused), never the agent. The audited payload carries ids and codes only, never usernames or emails.

Run-as (acting on behalf of a user)​

A call may name a run-as mode — never an identity the agent computed. The server resolves it against an explicit act_as-class trust grant covering the call's governance target ("owner" resolves the target asset's designated owner from the properties key named by the agents.owner_property_key config, default metadata.asset_owner; a username grant resolves statically). The acted-as principal's own authority is the ANDed ceiling, resolution failures are named refusals (never a silent fallback), and the acted-as identity plus how it was resolved land in the audit event and the run's metadata. The governance target is derived server-side from the request itself (POST /api/changes, POST /api/workflow_runs and their node-path variants).

A third mode serves chat answers: asker:<exchange-id> resolves to the authenticated user whose live chat exchange the agent is currently serving — the server stamps the exchange id onto the proxied chat request (X-Opschain-Exchange-Id, any caller-supplied copy stripped) and keeps the exchange → caller record only while the exchange is open. On a multi-replica estate the record lives with the exchange: a delegated read served by another replica resolves the asker through the exchange-holding replica, so the mode needs no sticky topology. The mode is read-only (GET), requires an explicit act_as:asker trust grant covering the read path (--act-as asker), and runs the read under the asking user's own principal, so the answer can never widen the asker's view; the audit event records the acted-as principal and the chat-exchange asker via grant … @ exchange … source.

Governance: tiers, grants, assignments​

The governance fabric sits behind the packs:agent-governance entitlement — engine in base, pack gates creation and surfaces. Gated (each a named 403 refusal): trust-grant create, approve and reject, the grants listing, the recommendations lifecycle, and the per-agent activity and trust-review reads (REST and the v2 twins alike). Base and never gated: tier enforcement on the request lane (existing grants keep governing on lapse), assignments — the ADVISE floor's carrier — grant revoke (shedding authority is never licence-gated), the halt scalpels and the audit trail.

The funnel's reads are not admin-only: a path-RBAC rule granting read on /governance/agents admits a governance role to the agent trust surface read-only — the fleet and agent detail (disclosures and owner included), the release catalog, the grants listing with approval state, assignments, activity and the trust review — without admin status, pack gates unchanged. The GUI renders this as the Governance → Agents view, read-only for that role; its Trust tab lists the grants, assignments and ceilings (type and node-classification) estate-wide. For a caller the /admin updatable determination admits the same view carries the trust verbs — write a grant (from the tab or from an agent's sheet, pre-addressed to that agent), approve or reject a pending proposal, revoke, assign and unassign, set and remove ceilings — exactly as opschain admin trust does, step-up floor included. The fleet verbs (drain, halt, forget, revoke, owner) stay on the Operations area's Agents panel.

A trust grant is deliberately the authz-rule shape — (subject, node-path scope, action patterns) — with the decision widened from allow/deny to a tier:

TierMeaning
adviseMay only produce recommendations.
supervisedExecutes behind a server-injected approval gate.
autonomousExecutes ungated.

No matching grant means the request is refused — ADVISE is the lowest grant, not the default. Subjects: one agent's service-account principal (service_account), every agent in a named group (agent_group), or every agent whose group carries a registered attribute value (attribute — trust policy that survives fleet churn). Scopes are node-path globs covering their whole subtree, so an asset created tomorrow is covered by today's grant. Grants may carry their own approver list, an act_as class, an expiry, and a lifecycle status (active, pending, revoked). Grants compile into a version-keyed in-memory snapshot (tier resolution is a trie walk, no database on the decide path), invalidated wholesale on every write.

Resolution for one (path, action): grants whose scope covers the path and whose action pattern matches, ranked most-specific-scope first, least-permissive tier among ties. The grant outcome then composes with the agent type's tier ceiling: a manifest type with a ceiling row never acts above it, whatever the grants say — effective = min(grant tier, type tier_ceiling). The ai type ships ceilinged at supervised; opschain admin trust ceilings lists the rows and opschain admin trust ceiling <type> --ceiling <tier> writes one (step-up floored, pack-gated, audited as info:trust:ceiling_set).

Node-classification ceilings (over metadata) compose the same way from the other axis: a rule keys on metadata predicates (e.g. core.criticality=critical), and at every target node whose effective classification matches, the agent's tier is additionally capped — effective = min(grant tier, type ceiling, every matching node ceiling), multiple matching rules resolving min-wins. Matching evaluates the compiled effective-metadata snapshot at the frozen target on each request, so a reclassification moves the clamp on the next request — "autonomous in Dev, supervised wherever critical" is classification policy, not per-subtree grant bookkeeping. opschain admin trust node-ceilings lists the rules and opschain admin trust node-ceiling set|remove writes them (the same step-up floor trust.ceiling_write, pack gate and audit discipline, as info:trust:node_ceiling_set/info:trust:node_ceiling_removed with the rule before and after).

A clamped decision stamps grant_tier, tier_ceiling (the type conjunct) and node_ceiling/node_ceiling_ids (the matching rules) beside applied_tier in the run metadata and the request event, so the audit explanation is always spelled out.

Grants also carry the cap fields — rate (governed work requests per minute), budget (invocations per UTC day) and tokens (model tokens per UTC day spent through the model gateway), 0 meaning capless — rate and budget enforced as O(1) counter reads at the same chokepoint as tier resolution, the token budget enforced pre-flight at the model gateway from usage figures read off the provider's own responses. An over-cap request refuses loudly naming the grant, the cap and the window reset, and lands the audited warn:trust:cap_refused event.

On a mutating agent call that creates work, the resolved tier is enforced server-side:

  • autonomous — passes clean.
  • supervised — the un-bypassable root approval gate is injected. Approvers resolve in order: the winning grant's approvers → the target's designated owner (the owner-metadata convention) → the agents.approvers config fallback. With none of the three: Agent-created changes hold for approval, but no approver is configured: set an approver on the trust grant, an owner in the asset's owner metadata, or agents.approvers in the server configuration.
  • advise — the mutating call is refused: Trust tier ADVISE on <path>: this agent may only recommend here — file a recommendation via POST /api<project>/recommendations instead. An instance serving the shadow lane is forced to this outcome whatever its grants say (the refusal names the lane): its recommendations carry lane: shadow and link to the stable lane's proposal for the same incident — a second proposal for an open (agent, subject node path, correlation root) key lands as a duplicate of the open card, which shows both provenances; a duplicate is never accepted on its own. opschain recommendations list|show <project> renders the cards with their lane, correlation id and linked duplicates (opschain docs cli/recommendations).
  • no grant — No trust grant covers <path> for action "<action>": this agent is not trusted to request work here (grant trust via the trust-grant admin API).

The agent attribution (agent_id — the fleet member id, agent_instance_id — the acting instance's id, agent_code, agent_project, requested_by_service_account, applied_tier, trust_grant_ids, and on delegation on_behalf_of/on_behalf_of_id/resolved_via) is stamped into the run's stored metadata — durable data, not log archaeology. The run's own initiator (initiated_by_type: agent, initiated_by_id) names the instance that raised it: the instance id is the stable identity across re-enrolment and release restarts, where the member id is the keypair in hand.

Assets under management​

A trust assignment records an agent's inventory — the scope it is the SME for — and is the natural scope to hang higher-tier grants on. With no matching grant, an assignment confers the configured assignment floor: advise (the default) or deny (agents.assignment_floor in config.yaml — see opschain docs config).

Raising trust is an approval-fabric event​

Managing grants is itself a governed action: an agent proposing a grant change stages it as pending, and only active grants compile. A proposed revoke leaves the grant active — and compiled — until a human applies or rejects it. The admin verbs: opschain admin trust grant|list|revoke|approve|reject|assign|assignments|unassign. The human half — list, approve, reject and the assignment verbs — rides the API v2 TrustService; grant and revoke stay on the REST routes (POST /api/trust/grants, POST /api/trust/grants/{id}/revoke) that the agent-credential lane shares, since the v2 plane authenticates users, never agent keypairs.

On a composition carrying the approval fabric (the adopted PostgreSQL schema), every grant write naming supervised or autonomous is a trust raise and enters the same pending_approval funnel human-created governed change uses — never a direct edit. The write lands the grant pending and opens a tier-sourced strict checklist (quorum 1, self-approval off, the grant id as the subject): approvers resolve from the grant's own approvers, else the agents.approvers config — with neither the write refuses by name, since a governed raise is never approver-less. The raise appears in the one approvals inbox (opschain approvals list); opschain approvals approve <grant-id> activates the grant, approvals reject revokes it, and an unanswered quorum TTL lapses it to revoked — an unapproved raise never takes effect. While the ceremony is open the direct admin trust approve|reject verbs refuse by name — the typed failed_precondition pointing at the approvals inbox. Events: info:trust:raise_held at creation, info:trust:grant_approved or info:trust:grant_rejected at the decision. Without the fabric composed, grant writes keep the base-posture behaviour above.

The trust review​

GET /api/agents/{id}/trust-review (opschain admin agent trust-review) is the decision-support view before promoting or demoting an agent: requests by applied tier, requests refused, recommendation outcomes by status, the active grants applying to its governance subject, its assignments, the server's assignment floor, and — when set — the agent's owned_by and its type's tier_ceiling. GET /api/agents/{id}/activity is the raw audit timeline, GET /api/agents/{id}/assignments its inventory.

Lifecycle control​

Admin-gated fleet verbs (GET /api/agents, GET /api/agents/{id}, and the actions below; the detail view fingerprints the public key, never renders it raw):

VerbEffect
drain / resume (`POST /api/agents/{id}/drainresume`)
halt / resume (POST /api/agents/{id}/halt, …/halt/resume)The per-agent emergency scalpel: hard stop — the durable halt flag flips, channel and sessions are severed immediately, every credential refused until an explicit resume. Reversible by design, no tombstone. A reason is required: a reason is required: the halt verb is break-glass audited. Group form: POST /api/agent_groups/{name}/halt (runner members of the group are unaffected). cli/admin-agent-halt, cli/admin-agent-halt-resume.
revoke (POST /api/agents/{id}/revoke)Tombstones the agent's key and severs its channel and sessions; the identity stays refused until re-enrolled with a fresh keypair and token. A revoked id reads as 404 — the tombstone stays un-exposed. cli/admin-agent-revoke.
forget (POST /api/agents/{id}/forget)Removes a stale registration so the identity may re-register later (not revocation). An online agent is refused: agent is online: stop or drain it before forgetting it (409). cli/admin-agent-forget.
rollover (v2 AgentService.RolloverAgent / AdvanceAgentRollover)Moves one lane of the agent to another release: drain each instance, instruct its restart over its channel, resume on re-enrolment; replace or canary=<n> with continue / abort / rollback / finish. blue-green restarts nothing: a green lane comes up on the release beside the serving lane (licence headroom for the overlap checked first), continue flips dispatch to it in one relabel, rollback flips back, finish drains, halts and retires the old lane. cli/admin-agent-rollover, cli/admin-agent-rollover-show, opschain docs agent-releases.

Plane-wide, opschain admin platform halt --plane agents severs every channel and suspends every service-account credential (the direct-HTTPS gate answers agent plane halted (posture <id>): <reason>) until a platform resume — see opschain docs platform-halt.

What the agent sees — platform-driven state changes​

Every state the platform imposes on an agent is something the agent itself observes, logs and reports — an operator reading the agent's own log file, its logs surface (GET …/agents/{code}/logs, the GUI's capability panel) or its readyz/info surfaces sees what happened to it and why, without consulting the server.

Typed reasons on the wire. Every refusal the agent plane answers for a platform-driven state carries a machine-readable reason beside its human text: an ErrorInfo detail (domain opschain.agent.v1, with metadata) on the agent-protocol error, mirrored in the Opschain-Agent-Reason response header; the capability proxy's 503 body is JSON carrying the same reason, message and metadata (Content-Type: application/json, the reason also in the header); and the model gateway's refusal JSON carries error.reason. Additive — clients that ignore them see the same Connect codes and texts as before.

ReasonWhenMetadataThe agent…
AGENT_PLANE_HALTEDthe agent plane's emergency posture is activeposture_id, reasonwaits
AGENT_HALTEDthis agent, or its group, is halted by the scalpelreason, scope (agent / group)waits
PACK_ENTITLEMENT_REQUIREDthe licence no longer grants a pack the agent needs (packs:agents; for the ai type packs:agent-governance + packs:ai-agents)feature, tierwaits
LICENCE_LIMIT_EXCEEDEDthe connected-agents limit is fulllimit, count, tierwaits
SYSTEM_QUIESCEDthe maintenance quiesce barrier is closedreasonwaits
AGENT_UNAUTHENTICATEDthe presented session is not live (expired, severed)—re-handshakes silently
AGENT_REREGISTER_REQUIREDthe server no longer knows the registration—re-registers with its durable key
AGENT_REVOKEDthe identity is tombstoned—exits, naming the reason
AGENT_RELEASE_REVOKEDthe release this instance runs was revoked; the instance is halted through the per-agent haltrelease, reasonwaits, until rolled to a current release and resumed
AGENT_INSTANCE_LIVEa fresh identity key presented the id of an instance that is still live under another key (the one already_exists left)instance_idexits — enrol without the instance id to start a second instance
AGENT_UNREACHABLE / AGENT_BUSY / AGENT_DRAININGthe proxy's own 503 classes (no live channel; concurrent-exchange bound; only the minimum spec while draining)last_seen_at for unreachable— (caller-side)
MODEL_CALL_REFUSEDthe model gateway's own refusal (unknown, disabled or unreferenced provider, missing credential, token cap)—— (the model SDK surfaces it)

A platform instruction reaches the instance at once on the replica that holds its channel. The instruction (a rollover's restart_from_release) travels down the channel as an exchange the moment it is issued: it is pushed onto an open stream, and a parked long-poll is completed with it rather than waiting out its window. The runtime acknowledges it and arms its drain, then acts on the drain at its next carriage turn — and the server supplies that turn the moment it has the acknowledgement: the long-poll the runtime re-parked in the meantime is completed (empty, "poll again"), an open stream is ended cleanly, so a runtime that reads its armed state only between polls — every long-poll runtime, the Python SDK included — restarts within milliseconds rather than up to one poll window (50 s by default) later. On an estate of several replicas this is replica-local by design: the replica holding the channel issues the instruction and gives the turn; a replica that does not hold the channel issues nothing and signals nothing across replicas, so a rollover started on another replica waits for the holding replica's next sweep (seconds) and then proceeds at the same pace. Halts, revocations and the plane halt reach a connected agent the same way, below.

The channel closes with the reason. When an agent is halted, its group or the plane is halted, its identity is revoked, or a licence change withdraws a pack it needs, the server does not just drop the connection: the bidi stream ends with the typed refusal and a parked long-poll answers it at once, so a connected agent learns the state the moment it is imposed, not at its next handshake. A licence change is reviewed on every server replica (packs:agents withdrawn → every connected agent; the ai dependency withdrawn → every connected ai-type agent); a lowered connected-agents limit is not swept — existing connections stay live and the limit bites at the next handshake.

Wait, never die. Every platform-driven state above except revocation is a wait: the agent keeps its identity, logs the transition once, and retries the channel under its bounded backoff (1 s → 60 s, full jitter) until the platform resumes or re-entitles — then logs the restoration. This includes a re-registration refused for one of these reasons: the identity directory is kept and the registration retried. Only AGENT_REVOKED (and an unrecoverable registration refusal, such as a bad token) ends the process, and the last log line names why. A fresh identity registering an already-registered code is simply a second instance of the agent; already_exists survives only for a fresh keypair presenting the id of an instance that is still live — stop that process, or enrol without the instance id. A platform-instructed restart from a release (restart_from_release, the rollover verb) is not a refusal: the runtime acknowledges it, drains, says goodbye and restarts itself (exec mode) or exits with code 75 for its process manager (exit mode), then re-enrols the same instance from the new release.

One line per transition. The Go runtime (opschain agent) and the Python SDK log the same lines, exactly once on entering a state — never once per backoff tick — to the process log and the logs surface:

channel open (stream) ← first connection (or "long-poll")
platform: agent plane halted — <reason> (posture <id>); channel closed, retrying every ≤60 s until resumed
platform: agent halted — <reason> (operator); channel closed, retrying every ≤60 s until resumed
platform: agent group halted — <reason> (operator); channel closed, retrying every ≤60 s until resumed
platform: entitlement <pack> withdrawn — <text>; channel closed, retrying every ≤60 s until re-entitled
platform: licence limit reached — <text>; retrying every ≤60 s
platform: system quiesced for maintenance — <reason>; retrying every ≤60 s until resumed
platform: server unreachable — <error>; retrying every ≤60 s
platform: channel lost — <error>; retrying every ≤60 s
platform: session refused repeatedly — <text>; re-handshaking retrying every ≤60 s
platform: channel refused — <text>; retrying every ≤60 s ← an unclassified refusal
platform: channel restored after <duration> (stream|long-poll) ← recovery from any of the above
platform: agent release revoked — <text>; channel closed, halted until an operator rolls this instance to a current release and resumes it, retrying every ≤60 s
platform: restart from release <name@version> instructed (<mode> mode); draining, then restarting ← a rollover
platform: agent revoked — <text>; exiting ← the only fatal line
platform: re-registration refused — <text>; exiting ← an unrecoverable registration refusal

A halt whose reason changes (a second halt with a different reason) is a new transition and logs again; a severed session's single re-handshake is silent (the handshake answers the real reason), and only a repeatedly refused handshake becomes a logged state.

On the surfaces. readyz carries checks.platform (the state name) and a platform object — state (connected, halted:plane, halted:agent, entitlement_withdrawn, licence_limit, quiesced, unauthenticated, unreachable, refused, revoked, or starting), detail (the posture id, the pack, the halt scope and reason), since and, when connected, carriage; info carries the same platform object. The Python SDK exposes it as agent.platform_state().

What changes behaviour, and who owns it​

LeverAuthorityEffect
Trust grants / assignmentsAdmin API (opschain admin trust …)The tier, approvers and act_as class per (subject, scope, action).
agents.approversconfig.yaml (opschain docs config)Fallback approvers for SUPERVISED outcomes.
agents.assignment_floorconfig.yamladvise (default) or deny with no matching grant on assigned scope.
agents.owner_property_keyconfig.yamlThe properties key naming an asset's designated owner (default metadata.asset_owner).
Registration tokensopschain admin agent-token createWho may enrol, into which project/group/code, for how long.
Drain / halt / revoke / forgetAdmin API (opschain admin agent …)Whether the agent runs at all.
The packs:agents entitlement + agents limitThe installed licence (opschain docs entitlements)Whether agents can register and how many may be connected at once; packs:agent-governance opens the grant/recommendation/review surfaces, and with packs:ai-agents admits AI-type enrolments.

Events vocabulary​

All searchable via /api/events, carrying the agent's governance path /{project}/agents/{code}, emitted server-side at the enforcement seam — no agent code path can omit its own trail:

  • Lifecycle: info:agent:registered (with the instance_id, lane and release stage), info:agent:goodbye, info:agent:forgotten, info:agent:revoked, info:agent:drain_changed, warn:agent:halted, info:agent:halt_resumed, warn:agent:group_halted, info:agent:group_halt_resumed
  • Rollover: info:agent:rollover_started, info:agent:instance_drained, info:agent:green_ready, info:agent:rollover_flipped, info:agent:instance_restarted, info:agent:canary_held, info:agent:rollover_continued / rollover_aborted / rollover_rolled_back / rollover_finished, warn:agent:rollover_refused — one correlation chain per rollover
  • Channel/tunnel: info:agent:channel_opened, info:agent:channel_closed, warn:agent:tunnel_denied, info:agent:chat
  • Request lane: info:agent:request (service account, method, path, status, applied tier, acted-as principal), warn:agent:request_refused
  • Governance: info:trust:granted, info:trust:grant_proposed, info:trust:raise_held, info:trust:grant_approved, info:trust:grant_rejected, info:trust:revoked, info:trust:revoke_proposed, info:trust:revoke_rejected, info:trust:assigned, info:trust:unassigned, info:trust:ceiling_set, info:agent:owner_changed, warn:trust:cap_refused
  • Transcripts: info:agent:chat, info:agent:adapter_transcript
  • Recommendations (the ADVISE output): info:recommendation:filed, info:recommendation:accepted, info:recommendation:dismissed, info:recommendation:expired, info:recommendation:superseded, info:recommendation:duplicate
  • Licence: warn:licence:feature_refused (a registration or reconnect refused over the packs:agents feature/limit, or an AI-type enrolment missing its pack dependencies — throttled per subject)
  • Releases (platform path — releases are estate-global): info:agent_release:key_registered, info:agent_release:key_status_changed, info:agent_release:created, info:agent_release:promoted, info:agent_release:promotion_held, info:agent_release:promotion_rejected, info:agent_release:retired, info:agent_release:revoked, warn:agent_release:refused; on the agent path: warn:agent_release:enrolment_refused

Failure modes​

  • An unreachable agent is routed around under group addressing; with members but none healthy the caller sees 503 "unreachable", not "absent". A silent agent ages out of the online view; its identity and grants persist.
  • A multi-replica server estate needs no agent stickiness: an agent's channel terminates on one replica, its owner is recorded durably, and a capability request (or a long-poll response) landing on another replica is forwarded to the owner over the replica-to-replica lane — the caller sees the same answer whichever replica the load balancer picked. An act_as: asker delegated read dispatched through another replica resolves the asking user the same way — the exchange-holding replica answers the lookup, so the delegation works without sticky topology and still dies with the exchange. Fleet accountability needs no stickiness either: online status reads the shared ownership record, so every replica's list, detail and dashboard counters agree. Prerequisites: each replica's peer_url set and the estate-shared token secret (see opschain docs config); without peer_url the non-owning replica answers the honest 503 with the agent's last-seen time.
  • A misbehaving agent has a graduated response: drain (graceful) → halt (immediate, reversible, audited with a reason) → revoke (credential tombstone). Grants can be revoked independently, effective immediately (snapshot invalidation on every write). The agent itself logs each of these as it lands (see "What the agent sees"): a halt or a withdrawn entitlement is a wait it rides out with its identity kept, a revocation is the one state that ends the process.
  • A dead agent cannot pin the server: exchange deadlines, concurrency caps and chunk credits bound every proxied call.
  • Governance cannot be bypassed from the agent side: tiers resolve from the compiled snapshot server-side, the approval gate is injected server-side, run-as identities are resolved server-side, and the audit events are written server-side.
  • opschain docs agent-posture — the inventory and posture of every agent enrolled in this estate: one row per agent with its tier, owner (and whether the owner departed), scope, release, last action, caps and spend, transcripts and health.
  • opschain docs cli/admin-agents, cli/admin-agent, cli/admin-trust, cli/admin-agent-token-create, cli/agent, cli/agent-dev — the command surface.
  • opschain docs entitlements — the agents feature and count limit.
  • opschain docs platform-halt — the plane-wide emergency posture.
  • opschain docs runner-registry — the shared fleet registry and groups.
  • opschain docs events — the audit stream the R16 events land in.