8.6 Upgrade and roll

Included from Basic tier

After this you can: upgrade a single server, roll a fleet of servers node by node without downtime, patch on a cadence, downgrade or roll back when you must, and choose a lane for a PostgreSQL major upgrade.

Before you start — tick each one:

• Lesson 8.4 — The posture ladder (opschain learn 8.4)

In the GUI: the Server panel in the Operations area shows the build, the replicas with their live/stale split and the leader, so you watch a rolled node leave and rejoin; the readiness checks tell you what the load balancer sees.

An upgrade is a binary swap. On one server that is a stop, a swap and a start; on a multi-replica estate you can do better — roll the fleet node by node, with runs in flight the whole time, and let peers carry each node's work while it is being replaced. This lesson is both: the single-server upgrade first, then the roll, then the ways back.

1. A single server: stop, swap, start

Take a backup before the swap (lesson 8.1). Migrations are additive-only, so the upgrade itself needs nothing else — but that backup is the lever for a true downgrade (section 7), and it is cheap insurance taken while the old binary still runs. Then stop the server, replace the binary, start it: pending schema migrations apply automatically on boot, and the migration ledger verifies that what the database records is what this binary carries, refusing loudly on a mismatch rather than running against a schema it cannot trust (lesson 8.5).

To stage the migrations ahead of the boot, with the server stopped:

opschain server migrate status
opschain server migrate up

migration ledger OK: the postgres database's migration history matches this server's migrations
applied 119 (add_agent_release_stage_names)

Patch releases apply exactly the same way — backup, swap, boot. Keep to a regular cadence and treat the pre-upgrade backup as part of the patch, not an extra. The documentation you are reading ships inside the binary, so after any swap the served lessons match the release you run (v5.6.1). On one replica this is a stop-the-world restart: runs still resume, clients see the outage. Everything below removes the outage.

2. What a roll relies on

The HA foundations of lesson 8.7 — a PostgreSQL store, a shared S3-compatible blobstore, identical keys.env across replicas, a load balancer probing /readyz — plus:

• A stable server name. Each node carries server_name (config.yaml; OPSCHAIN_SERVER_NAME wins; default: the hostname) — the durable key its maintenance posture is held under. Set it explicitly wherever hostnames are not stable.
• Node quiesce (opschain admin server node quiesce <name>, lesson 8.4): one durable posture row. Within a couple of seconds the named node leaves readiness (the load balancer routes around it), stops offering jobs, hands its scheduler shards, leadership and membership back to its peers, and detaches its in-flight run drives — peers adopt the runs within the staleness window, and runners ride through untouched. The node is then safe to kill, kill -9 included: every run is resumable from the store.
• Held-at-boot. The posture survives the kill, so the rolled binary boots held: healthy, logs and API inspectable, migrations applied — but taking no work, and /readyz answering 503.
• Human-gated resume (opschain admin server node resume <name>): the only thing that lifts the hold. Each quiesce and resume is audited (info:server:node_quiesced / info:server:node_resumed).
• Graceful handback on every planned stop, quiesced or not: shutdown releases the node's scheduler work to its peers immediately, so a polite restart leaves no gap.

3. Pre-roll checklist

1. Read the release notes' migration phase. An expand-phase release (additive schema only) rolls back binary-only; a contract-phase release needs a backup taken first and a system quiesce on the way forward. Inspect the phases any time with opschain admin database migrations list. Every migration declares its phase in its file and a build gate enforces it, so a release cannot ship one the previous release's schema gate cannot judge. Expand covers the older replicas' live database connections as well as their code: a column type widened by the first rolled replica cannot fail the others' cached query plans — the server plans queries so that it cannot, and if it ever did it retries once and records one warn:database:plan_invalidated event per replica. That guard sits in the serving binary, so it protects rolls from v5.6.0 onward; nodes still on v5.5.0 during the v5.6.0 roll refuse one git-remote query per pooled connection after the first node applies 00125, then recover on their own — the rolling-upgrade runbook's pre-roll section says how to plan around that window.
2. Trust the schema gate. A binary booting against a schema newer than it knows starts only when every unknown migration is additive; otherwise it refuses, naming the versions and the remedy (run the matching or newer build, or restore the backup taken at the old schema level). The refusal is the protection working, never a fault to bypass.
3. Fleet health: opschain admin status — exactly one leader, all 64 shards held, every replica live, no quiesce, halt or held node already in force (opschain admin server node list shows nothing held).
4. Version skew windows: keep servers at most one release apart during the roll; runners tolerate two minor releases of skew (stragglers are flagged by warn:runner:version_skew events); keep CLIs on the same minor — one minor apart warns.
5. Browser traffic: two binaries embed two fingerprinted GUI asset sets. Scope load-balancer cookie affinity for browser traffic to the roll window if you can; without it the fallback is built in — a stale-asset request answers an uncacheable 404, so one reload heals the browser.

4. The VM / bare-metal lane (verb-driven)

Per node, in order — rehearse it until it is boring:

opschain admin server node quiesce node-a

server node node-a quiesced: it drains, boots held after a restart, and takes no work until 'opschain admin server node resume node-a'

opschain admin server node list          # watch it park; the LB takes it out
# stop the process (SIGTERM is polite; kill -9 is safe here)
# replace the binary; start it — it migrates, then boots HELD
opschain admin server node resume node-a # after you have inspected it

{
  "name": "node-a",
  "resumed": true
}

Between the quiesce and the stop, confirm the park: the node's own /readyz answers 503, and a peer's log shows it adopting the node's runs. After the resume, the node heartbeats, re-contends for shards and rejoins the load balancer within seconds. Verify (section 6), then move to the next node.

systemd note: keep TimeoutStopSec above shutdown.timeout_seconds (default 60 seconds) so a polite stop is never escalated mid-drain — though with the node quiesced, escalation is harmless.

5. The Kubernetes lane (probe-driven)

Routine rolls need no verb at all — that is what the probes buy. The shipped deploy/k8s/server-deployment.yaml probes readiness on /readyz, liveness on /healthz, sleeps in preStop so the load balancer observes the drain, and sets a termination grace above the shutdown budget. A kubectl rollout is then the VM choreography automated: readiness drains each pod, shutdown detaches and hands back, and the new pod gates the rollout on /readyz.

For the belt-and-braces variant — the first upgraded pod inspected by a human before it takes work — give each replica a stable OPSCHAIN_SERVER_NAME (a StatefulSet's ordinal hostnames work as-is), quiesce the node about to roll, and let the rollout replace it: the new pod boots held and stays NotReady, pausing a maxUnavailable: 1 rollout by construction. Inspect, then resume to let the rollout proceed.

Keep shutdown.timeout_seconds plus the preStop sleep below the termination grace if you tune any of them.

6. Verification signals, and rollback

Per node, and at the end:

• opschain admin status: shards re-converge to 64 held across the fleet, exactly one leader, no node held once the roll completes.
• Runners logged a re-poll or a failover, never a job failure; no spike of runner_lost steps.
• Spot-check one run that spanned the roll: its steps continued on a peer and it finalised exactly once.
• Zero failed-request burn at the load balancer from drained nodes.

Rollback is the roll again with the previous binary. An additive (expand-phase) schema tolerates it by construction — the server boots in rollback mode (section 7). A contract-phase release rolls back binary plus restore point.

7. Downgrading, and rollback mode

Because migrations are additive-only, an older binary runs correctly against a database a newer build migrated. That state is rollback mode, and the server tells you it is in it: on every boot where the database schema is ahead of the binary, the boot log carries

running against a database schema 1 migration(s) newer than this binary (rollback mode)

naming both schema levels, and a matching warn:database:schema_ahead audit event is recorded — visible in Audit history under Governance, so the state is surfaced to operators rather than hidden in a boot log. It is a notice, never a refusal: the server starts and serves.

Rollback mode is for running an older binary temporarily — backing a release out while you investigate, or verifying a restore. The schema itself is not downgraded: the newer migrations stay applied, and the newer binary picks them straight back up. The true downgrade is restoring the backup taken at the older binary's schema level (lesson 8.1).

Do not roll the schema back by hand. The migration files carry down sections, but they are destructive — a down drops the tables and columns its migration added, and the data in them. No supported verb runs them; rollback mode and restore-from-backup exist so you never need to.

8. PostgreSQL major upgrades

Minor version updates need nothing from OpsChain — a supervised switchover is invisible to a healthy estate. Majors have two lanes.

Default — the bounded window. Self-managed PostgreSQL: system quiesce (opschain admin platform quiesce --reason …, lesson 8.4) → final backup (lesson 8.1) → pg_upgrade → validate → resume. Managed cloud PostgreSQL: the provider's maintenance lane, still inside a system quiesce.

Advanced — zero-downtime logical blue/green, for estates whose availability target will not accept the window. Run it at a regular cadence: every rehearsed cutover doubles as a site-failover drill (lesson 8.7). Prerequisites, all in place before the first cutover:

1. Replica-identity audit. Logical replication cannot replicate updates or deletes on a table without a replica identity. The shipped schema passes in full; audit a live estate (any extensions or manual tables included) with:

   SELECT c.relname
   FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
   WHERE n.nspname = 'public' AND c.relkind = 'r'
     AND c.relreplident NOT IN ('f')
     AND NOT EXISTS (SELECT 1 FROM pg_index i
                     WHERE i.indrelid = c.oid AND i.indisprimary);

   Zero rows means ready. A row names a table without a key or identity — fix it before subscribing.

2. DDL freeze. Logical replication does not carry schema changes: no OpsChain upgrade carrying a migration, and no manual DDL, from the moment the new cluster subscribes until failback is complete. The release notes' migration labels make "does this release migrate?" answerable; the freeze itself is a change-calendar commitment.

3. The cutover (a write fence of seconds to a minute — near-zero, stated honestly): quiesce writes (the system quiesce closes the write barrier) → wait for the subscription to drain to zero lag → synchronise sequences (logical replication does not carry them — script the setval set from the old primary and apply it on the new) → verify row counts on the hot tables → fence the old primary. The fence is mandatory: two writable primaries is the one silently wrong state this procedure must make impossible.

4. Repoint the servers by updating the store address in configuration and rolling the fleet node by node with sections 4–5 — each node comes back on the new primary, and there is no flag day.

5. Failback is the same procedure in reverse after upgrading the old primary — which is exactly what makes the cadence a disaster-recovery rehearsal.

Check yourself

• What holds a rolled node at boot, and who resumes it?
• What does the server print and record when it boots against a newer schema, and what is the true downgrade?

What you learned

• An upgrade is backup, swap, boot; the ledger verifies the schema and refuses a mismatch.
• A roll is node quiesce → kill → swap → boot held → inspect → resume, per node, with runs in flight throughout.
• Kubernetes rolls on the probes alone; a stable server name adds the human gate.
• Rollback mode is a loud notice, not a refusal; the true downgrade is a restore.
• PostgreSQL majors take the bounded window or the logical blue/green lane whose cutover fences the old primary.

Next: lesson 8.7 — Replicas and sites (opschain learn 8.7), or back to your track (opschain learn).