Retention

Included from Basic tierpacks:security-governance

Retention is the leader-run, age-based purge of operational data: each configured data class is swept on an interval, expired rows are exported to a filesystem archive and verified before anything is deleted, and blob bytes the deleted rows referenced are released from the blobstore once nothing else references them. It is the successor to the upstream DataCleanupDefinition with 09 tamper evidence on the archives.

Its neighbours: the scheduler supplies cron leadership and prunes its own scheduler_done bookkeeping through the job's PruneDone callback; the blobstore (opschain docs storage) holds the segment and artefact bytes the sweep exports and releases; and database maintenance (opschain docs database-maintenance) runs an ANALYZE of the churn tables right after any sweep that removed something.

The data classes

A class is swept only when its *_days key is set and greater than zero — an absent or zero value means never touched (two exceptions: chat_transcripts_days defaults to 365 and evidence_ledger_days to 730 when the keys are absent — exempting either class takes an explicit 0). The cutoff is now - <days> (UTC, whole days).

Class	What it ages	Age column	Removal	Export before removal
step_logs	Step log segment rows (log_segments) and the blobstore blobs holding their bytes. Segments of a run cited as run evidence on a legal-held ledger entry are exempt from every batch until the hold is lifted; the exemption set is re-read every batch, so a hold placed mid-sweep still bites	last_at	Row delete	Yes — rows plus the referenced blob bytes
chat_transcripts	Chat and adapter transcript segment rows (transcript_segmentsboth transcript kinds under the one horizon) and their blob bytes. Exchanges cited as transcript evidence on a legal-held ledger entry are exempt from every batch until the hold is lifted; the exemption set is re-read every batch, so a hold placed mid-sweep still bites	last_at	Row delete	Yes — rows plus the referenced blob bytes
events	Event rows (events)	created_at	Row delete	Yes — rows
step_transitions	Step state-transition audit rows (step_transitions)	created_at	Row delete	Yes — rows
artefacts	Artefact records (artefacts) and their blob bytes — DSL-stored and step-collected artifacts alike (opschain docs artifacts)	created_at	Tombstone, not delete: purged_at is set (UPDATE artefacts SET purged_at = COALESCE(purged_at, ?) …) and the blob bytes are released; the record row survives	Yes — rows plus the referenced blob bytes
scheduler_done	The scheduler's completion bookkeeping, pruned through the scheduler's own dialect	(scheduler-owned)	Row delete	No export
recommendations	Agent recommendations still open after the configured days	created_at (open rows only)	Status flip to expired, never a delete — the row is the audit artifact, so no export applies; each flip emits its lifecycle events	No
evidence_ledger	Governance evidence-ledger rows past the hot window — a chain operation, never a bare purge (see the sealed-segments section below). Rows under legal hold cap the segment: a held row, and every row after it, stays until the hold lifts	created_at	Row delete, permitted by the database only for rows a verified sealed segment covers	Yes — a verified, chain-preserving segment in the blobstore (not archive_dir), content-addressed and recorded in the chain itself

Blob release is reference-counted, not blind: blobs are content-addressed and shared, so a digest still referenced by any remaining log_segments or transcript_segments row or any un-purged artefact survives the release pass. On PostgreSQL the delete also removes each row's paired go_port_shims bridge row, which would otherwise be the new leak.

The sweep loop

• Leader-only. IsLeader gates every batch, and the serve composition wires it as the scheduler's cron-leader check and the system-quiesce posture: leaderUnlessQuiesced returns false from drain entry, so retention stops churning rows the moment a system quiesce (opschain docs platform-halt) begins and stays stopped until the posture is lifted. A failover never runs two sweeps at once because leadership is re-checked between batches — the sweep stops mid-class the moment this replica loses the lease.
• Interval. One sweep per retention.interval_minutes on the leader (default 60 minutes). A startup sweep fires as soon as a freshly booted replica first holds leadership, so a new leader catches up on backlog promptly rather than waiting a full interval.
• Batching. Each class is swept to completion in batches of retention.batch_size rows (default 500), with a 200 ms pause between consecutive delete batches so vacuum pressure never spikes the hot path.
• Order per batch: select expired → withhold rows whose blobs are missing from the blobstore (reported as warn:retention:blob_missing, never deleted) → export + verify → delete → release unreferenced blobs.

Export-before-delete archives

Every batch of a row-deleting class (and the artefact tombstone class) is archived before anything is removed:

• Location: the blobstore. Each batch is sealed as ONE tar object, content-addressed like every other blob, so archives are reachable from every server replica instead of living on whichever one led the sweep. The retention:sweep event's archives list carries each reference as <class> <UTC timestamp> -> sha256:<hex>. (retention.archive_dir is retired: accepted and ignored.)
• Contents (inside the tar): manifest.json first, then rows.ndjson.zst (the expiring rows as zstd-compressed NDJSON, every exported column included) and a blobs/ directory holding the referenced blob bytes for the blob-carrying classes (file name = digest with : mapped to -).
• Manifest (format opschain-retention-archive/v1): the class, cutoff, creation time, row count, and every file's SHA-256 and size. A missing_blobs list records digests whose bytes were already absent from the blobstore at export time — recorded honestly rather than failing the sweep. Rows with missing blobs are withheld before export, so the list is normally empty; it can only name a blob that vanished between that check and the export.
• Verification gate: after staging, the job verifies every file against its manifest hash and size, seals the tar into the blobstore, then reads the sealed object BACK from the store and verifies it again; rows are deleted only after the read-back verifies. Fetch an archive by its digest and unpack it with tar xf for the offline check — the manifest inside re-verifies the contents.

The evidence-ledger class — sealed segments

The governance evidence ledger (opschain docs governance) is hash-chained and append-only, so it can never ride the row purge above — deleting rows would break the chain. Its class is a chain operation:

• Boundary. The archivable segment is the aged contiguous prefix of the ledger: rows past the last sealed segment, strictly older than the evidence_ledger_days cutoff, stopping before the first legal-held row (a held entry is never archived-and-deleted out from under its hold) and capped at 50000 rows per segment — a deep backlog drains across successive sweeps.
• Verify first. The candidate segment's chain is re-verified from the previous sealed boundary before anything is exported. An unverifiable prefix refuses archival outright and emits the loud warn:governance:ledger_archive_blocked event — restore the database or re-anchor the chain first; a tamper or data-loss incident is never laundered into an archive.
• Export + verify again. The segment is written to the blobstore as zstd-compressed canonical NDJSON (opschain-evidence-export/v1: every row carries its stored prev_hash/row_hash verbatim), then read back by its content digest, decompressed, decoded and re-verified — row count, digest, per-row hashes and chain linkage must all agree before anything is deleted.
• Seal, then delete. A segment_seal ledger row records the segment's range, closing hash, row count and blob digest inside the chain, a registry row lands beside it, and only then are the sealed rows deleted — all in one transaction. The database guard refuses any ledger delete a verified sealed segment does not cover, so the posture holds against every code path.
• Verification crosses the boundary. opschain governance evidence verify seeds at the sealed boundary's recorded closing hash: the first stored row must chain off it, and the verdict reports the archived boundary (archived_through_seq) and segment count. Each archived segment also re-verifies offline from its own rows.
• Archived segments are never deleted by the sweep — they are kept in the blobstore to the operator's regulatory horizon, and the sealing rows citing them stay in the chain forever.

What changes retention's behaviour, and who owns it

All levers live in config.yaml's retention: block — deploy-authoritative, never editable through the API. See opschain docs config for the keys, types and defaults (retention.enabled, interval_minutes, batch_size, and the per-class step_logs_days, chat_transcripts_days, events_days, step_transitions_days, scheduler_done_days, artifacts_days, recommendations_open_days, evidence_ledger_days). Two shape rules worth knowing:

• An absent retention: block disables retention entirely; a present block is enabled unless enabled: false.
• A class whose days key is absent or 0 is never touched, so a minimal block retains everything except the classes you name — chat_transcripts_days (absent means 365) and evidence_ledger_days (absent means 730) excepted: exempting either takes an explicit 0.

There is no API or stored-settings override and no on-demand sweep verb — changing the policy means editing config.yaml and restarting the server.

Events and metrics

• One retention:sweep event is appended per sweep that deleted, expired or released anything (nothing is emitted for an idle sweep). Its payload is the sweep summary: deleted (per-class counts), expired (per-class counts, when any), blobs_released, and archives (the archive directories written). Visible in opschain events.
• One warn:retention:blob_missing event is appended per batch that found expiring rows whose blobs are absent from the blobstore (deduplicated per sweep): its payload names the class and each withheld row's id and missing digest. The rows survive; see Failure modes.
• Operator notices (the server log): retention sweep: deleted <class=n …>; released <n> blob(s) (plus ; expired <class=n …> when any), retention <class>: <n> row(s) reference blobs missing from the blobstore — skipped, not deleted (warn:retention:blob_missing) when rows are withheld, and retention sweep failed: <error> on a failed sweep. A sweep whose summary or blob-missing event cannot be recorded notices retention: … event not recorded: <error>.
• Prometheus: opschain_retention_rows_deleted_total{class}, opschain_retention_rows_expired_total{class}, opschain_retention_sweep_failures_total.

Interaction with database maintenance and quiesce

• After a sweep that removed anything, the job's AfterSweep hook runs the database-maintenance post-retention ANALYZE of the churn set — wired only when database.maintenance is enabled. See opschain docs database-maintenance.
• While a system quiesce posture is in force, the leader check returns false, so no sweep starts and an in-progress sweep stops at its next batch boundary. Nothing is lost: the backlog is swept when the posture lifts and leadership returns.

Failure modes

• A failed class fails the sweep but keeps its progress: the summary reports what earlier classes (and earlier batches) already deleted, the error is wrapped as retention <class>: <cause>, the failure counter is bumped and the notice retention sweep failed: … is emitted. The next interval retries from the top — deletes are idempotent by age, so nothing is double-counted.
• A failed archive verification refuses the delete: the error reads archive <dir> failed verification: <cause> and no row in that batch is removed. The partially written archive directory remains on disk for inspection.
• A row whose blob is missing is never purged: deleting it would be permanent, silent evidence loss (a diverged or lost blob root must never erase the archive trail — archives are never silently deleted). The sweep withholds such rows from export and delete, emits a warn:retention:blob_missing event naming each row and its absent digest, and keeps purging the rows whose evidence is intact. The withheld rows are retried — and re-reported — every sweep until the blob reappears or the rows are dealt with by hand. Missing blobs never fail the sweep.
• Losing leadership mid-sweep simply stops the sweep between batches; the new leader's startup sweep picks the backlog up.
• Nothing configured is silent by design: with the block absent (or every days key zero) the job is never started, and no event or notice marks that.

Related pages

• opschain docs config — the retention.* keys, types and defaults.
• opschain docs database-maintenance — the post-sweep ANALYZE hook.
• opschain docs storage — the blobstore the sweep exports from and releases into.
• opschain docs events — where retention:sweep lands.