8.7 Replicas and sites

Included from Basic tier

After this you can: run several server replicas and two sites — say what survives which failure, keep work site-local, take a site down for maintenance without a failover, promote the other site when you must, and drill it.

Before you start — tick each one:

• Lesson 8.6 — Upgrade and roll (opschain learn 8.6)

In the GUI: the Server panel in the Operations area lists the replicas with their live/stale split and the leader, and its readiness checks tell you what the load balancer sees of each one; the Sites panel on the Runners page shows which sites are held, quiesces a site behind a type-to-confirm that names the failover, and resumes it.

One OpsChain server is a complete estate. This lesson is the step up, in two halves. First, several replicas against one database, so that losing a replica — a crash, a node failure, a deliberate kill during an upgrade — costs seconds of latency instead of an outage. Then two sites: each data centre running its own replicas and runner fleet, work staying local to its site, site-scoped maintenance that never strands a site's workload, and the loss of a whole site handled by a drilled, human-decided promotion.

The one rule everything else serves: there is exactly one writable PostgreSQL primary, at one site, at any moment. Either site can hold that role; nothing in this lesson ever produces two writable primaries, and the fence step in the promotion procedure exists to make that impossible.

1. The shape of an HA estate

An HA estate is N identical server binaries sharing three things:

• One PostgreSQL database (store: postgres://…). Every durable fact — the estate, runs, events, leases, postures — lives here. On the embedded SQLite store every mechanism in this lesson degenerates to single-process behaviour; HA requires PostgreSQL.
• One shared blobstore (blobstore: kind: s3 in config.yaml). Step logs, artifacts, pack bundles and retention archives are content-addressed blobs; with the default filesystem blobstore a second replica cannot read the first one's bytes, so multi-replica estates point every replica at one S3-compatible bucket. See opschain docs storage for the configuration and for migrating an existing estate's blobs into a bucket.
• Identical key material. The keys.env secrets — above all the shared token secret — must be byte-identical on every replica. This is what lets runner and agent sessions authenticate on whichever replica a load balancer picks.

In front of the replicas sits your load balancer, probing readiness on GET /readyz and liveness on GET /healthz. No session affinity is required for API or runner traffic: every replica honours every credential, and job-scoped runner calls (renewals, log uploads, result submission) apply correctly wherever they land (section 3). The one narrow exception is browser traffic during a binary roll (lesson 8.6).

2. What survives what

A replica dies mid-run. Every run is driven by exactly one replica under a short-lived drive lease. When the driving replica dies — power loss and kill -9 included — a peer claims its runs within about 15 seconds, resumes them from the store exactly where they were, and the runners' next renewals and submissions land and apply on the survivor. The run finalises exactly once; no step executes twice. A deposed driver is fenced, so its writes can never interleave with the new owner's.

A replica is stopped gracefully. A planned stop drains first: readiness flips off so the load balancer routes around it, in-flight requests get a grace period, parked runners re-poll and land on a peer, and the replica hands its scheduler work back before exiting — peers take over immediately, without waiting out any staleness window. During a graceful restart under load you should see zero failed requests and zero failed jobs.

A server becomes unreachable from a runner. Start runners with an ordered server list — opschain runner --server https://primary.example.com,https://standby.example.com — and a runner that cannot reach its current server fails over to the next entry after three consecutive connection failures, keeping its in-flight job alive, and fails back to the primary on its next natural session re-handshake after the primary recovers (lesson 6.2; opschain docs cli/runner).

A replica crashes while holding a step's log. The log lines a runner had uploaded and had acknowledged survive: on its next boot the replica seals the orphaned in-flight log spools into the blobstore, marking the seam with an honest gap marker in the step log; anything the runner never had acknowledged is re-uploaded by its at-least-once protocol. If the step is still running, the next replica to receive one of its chunks takes the attempt over, marks the hand-over in the log — *** log gap: log ingest moved to another server replica at sequence N … — and the run completes.

Leadership moves. Background work that must run once per estate — retention sweeps, database maintenance, licence verification rounds, directory reconciliation — runs on an elected leader, with leases anchored to the database clock (replica clock skew is never a correctness input). A leader that loses contact with the database demotes itself immediately and a peer takes over; interrupted sweeps stop cleanly rather than running twice.

3. Live behaviour across replicas

• Configuration converges in about a second. Disabling a secrets provider, changing authorisation rules, applying a licence — a write on one replica invalidates its peers' caches within roughly one second, with bounded time-to-live backstops so a missed signal can only delay convergence, never prevent it.
• One step attempt's log lives in one place. The first replica that receives a chunk of a step attempt becomes the attempt's log owner; every other replica forwards the attempt's chunks to the owner over the replica-to-replica lane (peer_url) and acknowledges the runner only once the owner has stored them, so a round-robin load balancer never splits a step's log. When the attempt ends, the owner seals its log into the blobstore and releases the row. If the owner cannot be reached while it is still a live member, the upload is refused with a retryable error and the runner re-uploads (nothing is dropped); the estate records warn:logs:ingest_forward_failed (once per attempt per minute) so a missing or wrong peer_url is visible. In a multi-replica estate peer_url is therefore required on every replica. A chunk that is given up anyway — refused by verdict, unreadable in the runner's spool, still unsent when the runner's end-of-step flush times out, or arriving after the attempt ended and its log sealed — is never silent: the log carries a *** log gap: <n> chunk(s) <first..last> not delivered — <reason> *** marker where the lines would have been (a sealed marker also names the path the verdict took and the replica that refused) and the run records one warn:logs:chunks_dropped event per attempt and reason, so a log that stops short always says why. A chunk of a step that is still running is never refused as sealed: if no replica holds the attempt's ownership row — a replica on an older release spooling it during a rolling upgrade, or a row released under a live owner — the replica that receives the chunk adopts the attempt, records warn:logs:attempt_adopted, and the line lands. The same holds for the seal itself: a seal re-delivered to the owner after a timed-out acknowledgement (a replica restarting mid-roll) is already stored and simply succeeds, a seal that fails on the owner records warn:logs:seal_failed and is retried (the spool stays until it lands, at the latest at that replica's next boot), and a sealed segment that cannot be read back records warn:logs:segment_unreadable and shows as an unreadable segment marker in the live tail — nothing about a step's log goes quiet.
• Live log tails work from any replica when each replica declares the address its peers can reach it at (peer_url — the pod or host address, never the load balancer). A tail landing on a replica that is not the attempt's log owner streams from the owner, mid-step.
• Event-driven surfaces trail real time by a few seconds. Event streams, trigger firings and notifications read behind a short settle window (about five seconds plus the poll interval) so that transactions committing out of order are never skipped. This is a bounded delay, by design.
• Security limits hold estate-wide. Login backoff, agent trust rate and budget caps and related security counters are shared through the database, so N replicas never multiply an attacker's budget. Performance rate limits (api.rate_limit.*) are deliberately per-replica — size them against your replica count.

4. What you configure

Concern	Key	Guidance
Database pools	database.pool.*	Each replica opens three capped pools (defaults total 25 connections). The sum across replicas must fit PostgreSQL's max_connections with headroom — the defaults fit four replicas on the default 100.
Blobstore	blobstore:	kind: s3 for any multi-replica estate; see opschain docs storage.
Peer address	peer_url	Per replica: the direct address peers reach this replica at. Enables cross-replica live log tails and agent routing.
Stable node name	server_name	Per replica: the durable name the per-node maintenance verbs address (lesson 8.4). Set it explicitly wherever hostnames are not stable.
Site identity	site	Multi-site estates only — section 6.
Shutdown budget	shutdown.*	Keep your process supervisor's stop timeout (systemd TimeoutStopSec, Kubernetes termination grace) above shutdown.timeout_seconds.

All keys are specified in opschain docs config. The shipped Kubernetes manifests under deploy/k8s/ wire the probes, preStop sleep and termination grace consistently.

5. Reading replica status

opschain admin status is the fleet's one-page health read:

opschain admin status

status:    ok
version:   v5.7.0
replica:   replica-a (leader: true, shards: 22, draining: false)
store:     ok
queue:     3 ready, 2 leased
REPLICA    HEARTBEAT AGE  LIVE
replica-a  0.4s           true
replica-b  1.2s           true
replica-c  171.6s         false

On a healthy HA estate you should see exactly one leader and 64 scheduler shards held across the fleet (work claiming is partitioned; a gap re-converges within seconds), a live membership row per replica, each heartbeating, carrying its site where set, and no posture in force you did not put there. replica-c above died without a goodbye: its row stays listed with LIVE false until the leader's reaper retires it, while its shards, leadership and run drives moved to its peers at the 15-second staleness boundary.

GET /readyz per replica tells you what the load balancer sees, and names the reason when a replica is out of rotation (draining, held, blobstore unreachable). opschain admin server node list shows which nodes are held by a maintenance posture; the queue view (opschain docs fleet-queue) explains queued work's state whenever dispatch looks idle.

6. Two sites: the symmetric topology

Each site runs:

• N server replicas, every one carrying its site identity (site: dc1 in config.yaml, or OPSCHAIN_SITE) and a peer_url. Set site on every replica or on none — a site-less replica claims work for any site.
• The site's runner fleet, in runner groups attributed location=<site> (the same vocabulary as the site key — lesson 6.3), each runner started with both sites' addresses in its ordered server list: opschain runner --server https://dc1.example.com,https://dc2.example.com.
• A PostgreSQL presence and an S3-compatible blob presence (the data tiers below).

DNS or a global load balancer fronts the active site for interactive API traffic, with low-TTL records prepared in advance. The passive site's server binaries are cold standbys: installed, configured against the passive database address, not started — the server requires a writable primary, deliberately; there is no read-only boot mode. The keys.env secrets, TLS material and vault key material are byte-identical across sites.

The database tier is vendor-neutral — managed cloud PostgreSQL services are first-class, and a self-managed estate picks its own replication stack. The tiers are symmetric, so either site can be promoted and the roles swap:

• Default — asynchronous physical replication across sites: the active site's primary streams to a continuously applying standby cluster at the other site (plus an in-site synchronous replica for zero data loss on single-replica failure). This is the shape the recovery numbers in section 10 describe.
• Advanced — synchronous replication across sites: buys zero data loss across sites; every write pays the inter-site round trip, so this is metro-distance guidance. Still one writable primary.
• Advanced — logical blue/green: zero-downtime database major upgrades via logical replication, run at a regular cadence so every major upgrade doubles as a rehearsed failover (lesson 8.6).

The blob tier follows the database. Step logs, artifacts and pack content live in the blobstore and are referenced by rows in the database, so the two tiers must fail over together: cloud estates point every site at one shared bucket and let the provider's geo-redundancy do the replication; on-premises estates run a per-site S3-compatible store with the store's own cross-site replication, and monitor the blob replication watermark against the database replication lag wherever any asynchronous replication exists.

7. Work stays local: site-affine dispatch

Pin work to a site with the standard placement vocabulary: location-attributed runner groups, or a location runner selector in the node's settings (lesson 6.3). On an estate whose replicas carry site identities, claiming is also site-preferring: a step demanding one location is claimed by that site's replicas first, so DC-local work is dispatched by DC-local servers to DC-local runners. Work never strands: another site's step becomes claimable anywhere the moment its site has no live replica, or after a short fallback window with the site live but idle — and placement rules still decide which runners may execute it, wherever the claim lands. The mechanics are in opschain docs dispatch.

Interactive traffic: both sites' replicas may serve API traffic against the one primary. Full interactive quality holds at an inter-site round trip of about 20 ms or less (same metro, adjacent regions); beyond that, home interactive traffic on the primary site and run the remote site as runners plus GUI nodes — the runner-plane calls tolerate seconds-scale latency, so site affinity works at any sane distance.

8. Planned site maintenance (no promotion)

A whole site's servers can be patched, rolled or taken down without touching the estate and without a database failover, as long as the database primary is not at the affected site (if it is, either accept a bounded window or promote first):

opschain admin server site quiesce dc2

site dc2 quiesced: its server nodes drain, boot held after a restart, and take no work until 'opschain admin server site resume dc2'

1. Every node of the site drains, hands its scheduler work back and leaves readiness. Any replica serves the verb.
2. Kill, patch or roll the site's servers at will; a rebooted binary boots held and takes no work.
3. The site's runners fail over on their ordered server list, and the surviving site claims the quiesced site's work on its behalf.
4. opschain admin server site resume dc2 — the nodes rejoin, and runners fail back on their next natural re-handshake.

The Sites panel on the Runners page is the same operation from the browser: type the site identity, Quiesce site… arms on the identity typed back and names what drains and where the work fails over to, and each held row carries Resume; the panel also shows the answering node's own site and whether it is held.

Each posture change is audited (info:server:site_quiesced / info:server:site_resumed; lesson 8.4).

9. What a site failover loses — and does not

• Not lost: all committed state. Leases, postures, cursors and idempotency keys are ordinary replicated data.
• Lost, self-healing: every in-memory cache and session. Caches rebuild; runner and agent sessions re-handshake; run drives re-attach when the promoted site's servers boot.
• Lost, bounded — this is the recovery point: the last seconds of commits inside the asynchronous replication window. Runners re-deliver unacknowledged results and log chunks; a completion lost inside the window ages into a runner_lost step — operator-visible and retriable (lesson 6.1), never silent.
• Lost, named loudly: in-flight steps' live log tails at the failed site (the log carries a gap marker), and blob objects written inside the blob replication lag — reads name a missing blob rather than pretending it never existed.
• The failed site's runners are gone with the site; the surviving site's (and any customer-hosted remote) runners reconnect on their server list and resume.
• Manual, always: the promotion decision, and failback.

10. Recovery expectations

What the architecture achieves, verified by drilling:

Measure	Expectation	Notes
Cross-site recovery point	≤ 60 seconds of commits, steady-state	Alert when asynchronous lag exceeds 15 seconds. Under a WAN partition the lag — and therefore the worst case — is unbounded until the partition heals; that is intrinsic to asynchronous replication, and the synchronous tier is the zero-loss option.
Cross-site recovery time	≤ 30 minutes for the full manual procedure, declare → serving	≤ 10 minutes is achievable once drilled. Automated cross-site promotion is deliberately not built: unsupervised promotion is where split-brain comes from.
Intra-site	Replica loss: dispatch resumes in under 15 seconds; zero committed-state loss with an in-site synchronous replica	The single-site HA behaviour of section 2, unchanged.
Step-log tails	Last moments may be lost, marked with an explicit gap	Never silent.

Promotion is a human decision, always.

11. Promotion (site failover) — the procedure

1. Declare. A human decision against monitoring evidence: the active site is unreachable or will not recover within your recovery-time objective. Record the time.
2. Fence the old active — not skippable. Make the old primary unable to accept writes when it comes back: demote or hibernate the old database cluster (your replication tooling's demotion, or the managed provider's), and remove the old site from DNS/load balancing so stray API traffic dies. This step is what makes two writable primaries impossible.
3. Promote the passive site's database cluster to primary. Verify the blob tier has drained to its watermark, and record both watermarks — the data loss actually incurred — in the incident log.
4. Start the standby server replicas against the promoted database. Boot does the rest: migrations no-op under the cluster-wide migration lock, boot resume and the adoption sweep take over every run, lease expiry ages the dead site's in-flight attempts into runner_lost, and the scheduler re-forms. (If the failed site had also been site-quiesced earlier, resume that posture — a held site posture would park the promoted site's nodes carrying the same identity.)
5. Cut over DNS/load balancing to the new site. CLIs follow the name; runners follow their ordered server list.
6. Verify: /readyz green on every replica; opschain admin status shows one leader, all 64 shards held, live membership with the right site; a canary change executes end to end; the runner_lost set matches the window's expected in-flight work and is triaged (retry or acknowledge). Record declared → serving — the recovery time actually incurred.
7. Failback is a planned, quiesced operation, never automatic: rebuild replication toward the recovered site, then a system quiesce (lesson 8.4), the reverse promotion, DNS/load balancing back, resume.

12. Drills

Run the full procedure on a staging estate at least quarterly: kill the active site for real, promote, and record declared → serving and the watermark delta as evidence against the expectations in section 10. An estate on the logical blue/green cadence additionally rehearses promotion at every database major upgrade — which is what keeps the real one boring. The procedure is only proven while the drill stays boring.

Check yourself

• What does a run in flight do when the replica driving it dies, and within how long?
• Which verb takes a whole site down for patching without a failover, and what does a site failover lose?

What you learned

• An HA estate is identical binaries sharing one PostgreSQL primary, one blobstore and one keys.env, behind a /readyz-probing load balancer.
• A dead replica's runs are adopted within about 15 seconds and finalise once; a planned stop hands back immediately.
• Logs, caches and security limits converge across replicas; peer_url is required.
• Sites are the site key on replicas and location on runner groups; claiming prefers the local site and never strands work.
• Site maintenance is a quiesce; failover is a fenced, human-decided promotion whose numbers only a drill proves.

Next: this closes the module Operate the server — back to your track (opschain learn).