8.1 Backup, restore and prove it

Included from Basic tier

After this you can: back up the server with a verified manifest, restore it, and prove the restore from the record.

Before you start — tick each one:

• Lesson 0.1 — Install and sign in (opschain learn 0.1)

In the GUI: the Overview's emergency controls in the Operations area show the system-quiesce posture the backup window needs (and lift it); the Database panel shows the quiesce state beside the table statistics.

A backup you have never restored is a hope. This lesson is one ceremony, run end to end: take the backup inside a verified window, restore it onto an empty store, and prove — from the manifest and from the event record — that what came back is what you captured. You administer the server itself for this one (an admin login, lesson 0.1).

1. What a backup is

An OpsChain backup is the database plus the blobstore, captured together while the system is quiesced. The database holds the estate, runs, events and settings; artifacts and log segments live in the blobstore, not in the database — a dump without the blobstore copy is half a backup, and the verification in section 3 will say so. With the default filesystem blobstore the copy is the blob_root directory tree; an estate on the S3-compatible blobstore (blobstore: kind: s3 in config.yaml) keeps its bytes in the bucket, so the blob half of the backup is the bucket's own durability story — enable versioning or replication there, and capture the bucket (or a sync of it) inside the same window.

Keys are deliberately not included, on either backend. Back up the server's keys.env (and the secret-store master key) separately and securely — encrypted data is unrecoverable without them, and a backup that contained them would be a secret itself.

2. PostgreSQL: quiesce → manifest → dump → resume

OpsChain makes the window safe and verifiable; your database tooling moves the bytes. First take the estate to a verified standstill — the command drains running work, then closes the write barrier (lesson 8.4 has the whole posture ladder):

opschain admin platform quiesce --reason "nightly backup"

draining: 2 attempt(s) still running
draining: 1 attempt(s) still running
system quiesced: the write barrier is closed — resume with 'opschain admin platform resume'
{"posture":{"id":"ocsq_…","reason":"nightly backup",…},"phase":"quiesced","remaining_attempts":0,"attempts":[]}

Now fetch the manifest. It refuses while the system is not quiesced — it describes a database that must not be changing:

opschain admin database backup-info --out manifest.json

wrote the backup manifest to manifest.json (quiesced: nightly backup)
run: pg_dump --format=custom --no-owner --file=opschain.dump "$DSN"

The manifest records the build identity, the schema level, exact per-table row counts of the durable tables, the blobstore root and every referenced blob digest, the quiesce posture, and a content digest over its own body, so any later alteration of the file is detectable. The volatile tables — replica membership and heartbeats, leases, runner liveness — are named under tables_excluded rather than counted: a restored server joins with its own membership row, so their counts legitimately differ and verify-restore never compares them. Run the recipe it printed, copy the blobstore root it names alongside the dump, and keep the manifest with both:

pg_dump --format=custom --no-owner --file=opschain.dump "$DSN"
# filesystem blobstore: archive the blob root the manifest names
# S3-compatible blobstore: snapshot or sync the bucket in the same window

Then resume — the system stays quiesced until you do; there is no auto-resume:

opschain admin platform resume

{"id":"ocsq_…","reason":"nightly backup","quiesced_by":"admin","resumed_by":"admin","resumed_at":"2026-10-15T02:14:09Z"}

Both ends of the window are on the record — api:platform:quiesced and api:platform:resumed, carrying the posture id, the reason and the actor.

3. Restore, and prove it

Restore the dump into an empty database, put the blobstore copy back at its recorded root, and start the server against it — pending migrations replay on boot if the binary is newer than the backup:

pg_restore --no-owner --dbname "$DSN" opschain.dump

Then verify. The command replays the manifest's checks against the running store and reports pass or fail per check:

opschain admin database verify-restore manifest.json

CHECK            RESULT  DETAIL
manifest_digest  PASS    content digest matches
schema_level     PASS    migration 118 (manifest 118)
table_counts     PASS    all 41 durable table counts match the manifest; 9 volatile table(s) not compared (replica membership, leases and runner liveness differ legitimately after a restore): leases, log_attempt_owners, run_inbox, runner_challenges, runner_jobs, scheduler_job_leases, scheduler_pool_holds, scheduler_replicas, step_api_tokens
blobstore        PASS    1,204 blob digest(s) present under /var/opschain/blobs
restore verified — resume the system with 'opschain admin platform resume' when ready

What verify-restore compares: the manifest's own digest, the schema level, every durable table's row count, and every referenced blob. The volatile tables it lists as "not compared" are the ones whose rows describe live processes, never estate data — a difference there is the restored server being alive, not the restore being wrong.

Read the schema-level check carefully. Matching passes. Newer passes with a note — this server replayed the newer migrations on boot, as expected after restoring an older backup under a newer binary. Older fails: the running binary predates the backup, so verify (and run) with a binary at least as new as the one that took it. Any failed check exits non-zero. Do not resume a failing restore.

A restored database comes back still quiesced — the backup was taken that way — so resume once verification passes:

opschain admin platform resume

That resume is the proof from the record. The restored estate's event trail ends with the api:platform:quiesced row your backup window wrote, carrying the same posture id the manifest recorded; your api:platform:resumed closes it. Read the pair back:

opschain events --type-search platform --since 24h

Same posture id in the manifest, in the restored database's last quiesce event, and on your resume: the database you reopened is the one you captured. On boot the server also reconciles in-flight state honestly — steps whose runner leases died with the backup fail fast as runner_lost (retriable, lesson 6.1), and queued steps become dispatchable again.

4. The embedded store

On the embedded SQLite backend the built-in verbs are the right tool, and no quiesce is needed — the snapshot is transactional against a running server, and the archive holds the manifest, the database and the blobs together:

opschain admin database backup --out nightly.zip

backed up opschain.db and 1204 blob(s) to nightly.zip

Restore targets a fresh home — an existing store file or a non-empty blob root is refused by name, never overwritten, and a tampered archive refuses verification:

OPSCHAIN_HOME=/fresh/home opschain admin database restore nightly.zip

restored opschain.db (1204 blob(s), backup taken 2026-10-15T02:10:44Z)
start the server to reconcile: leases that died with the backup fail fast as runner_lost (retriable); queued steps dispatch again

opschain server backup <dir> and opschain server restore <dir> (--force to restore over an existing store) are the offline twins for a stopped server — the manifest is verified before anything is touched, so a truncated backup never restores. Either verb refuses on a PostgreSQL store with guidance: that lane is section 2, by design.

5. Habits

• After every backup: the manifest sits beside the dump and the blob copy, and names the digest, counts and blobstore root you expect.
• Quarterly, and before any release whose notes name a contract-phase migration (lesson 8.6): restore into a scratch environment and run verify-restore. A backup is only proven by a verified restore.
• Keys travel through your secrets process, never inside a backup.

Check yourself

• What does the backup manifest carry, and what does verify-restore report when the restored schema is newer than the manifest's?
• Which event pair, sharing which id, proves the database you resumed is the one you captured?

What you learned

• A backup is the database and the blobstore captured together inside a quiesce; keys travel separately.
• backup-info seals the window in a manifest and prints the dump recipe; verify-restore replays the manifest's checks.
• A restored database comes back quiesced; resume only after every check passes.
• The posture id ties the manifest, the restored record and your resume together.
• The embedded store backs up and restores with one verb each; the offline twins serve a stopped server.

Next: lesson 8.2 — Definitions under change control (opschain learn 8.2), or back to your track (opschain learn).