Skip to main content
Version: 2026-11-04

Backup & restore

This guide goes into detail about how to backup and restore your OpsChain resources, how OpsChain is designed to facilitate backups and how you can use a backup tool to backup and restore your OpsChain resources.

Introduction​

OpsChain is designed to be backup and restore friendly, ensuring you can easily export and import your OpsChain instance in different environments. Most of the settings and configurations used by OpsChain reside encrypted within its database, making it the most important resource to back up on a regular basis.

There are two things to back up: your deployment configuration (the values.yaml file) and the OpsChain database. OpsChain can back the database up for you on a schedule. A database backup comes in one of two forms — a physical backup (a complete copy of the database files) or a logical backup (a portable export of its contents) — each suited to a different situation, as explained below. To restore an instance you need both the values.yaml file and a database backup.

Backup​

Configuration file​

Make sure you have a backup of the values.yaml file used to deploy OpsChain, ideally one by deployment date, such that you can restore to a specific version of OpsChain. This file contains the exact configuration for OpsChain resources and services, such as its database, vault, LDAP server, etc., as well as the encryption secrets that were used to encrypt all your sensitive data. Without this file, restoring your OpsChain instance will not be possible.

Keys to keep with your backups​

A database backup can only be restored with the encryption keys and passwords the installation used. Keep these values from your values.yaml with your backups, stored separately from them.

You also need the backup private key (opschain-backup-key) to decrypt a backup. Keep it offline, apart from the cluster and the backups. Its public key is the db.backup.encryption.publicKey value in your values.yaml.

Database​

OpsChain can back up its database automatically — see automated backups below to enable and configure scheduled and pre-deploy backups. A few things to keep in mind:

  • The backup must always be performed on the primary database cluster - if you're not running in a high availability setup, this is the only cluster you have. Backing up from replica clusters will create unusable backups.
  • The backup should ideally be performed when no changes, workflows or agents are running. If any of these are running, it's possible that they will be in an inconsistent state when restoring from the backup.
  • When in a high availability setup, replica database clusters will need to be recreated after the backup is restored.
Limited backup

Backing up the database is limited to the data that is stored in the database. While this includes change and step states, it will not include the actual long-running agents, change deployments or their respective resources.

Automated backups​

OpsChain can run scheduled database backups for you, and can also take a backup automatically before each upgrade. These backups run inside the cluster using a backup utility built into the OpsChain database image, so your data never leaves your local network.

note

Automated backups only run on the primary database cluster. In a high availability setup, backup resources are not created on replica clusters, and the backup utility refuses to run if it detects the target database is a replica.

Configure automated backups​

Automated backups are controlled by the db.backup section of your values.yaml. Scheduled backups are enabled by default, and need a backup encryption key:

db:
  backup:
    encryption:
      publicKey: ssh-ed25519 AAAA... opschain-backup
    periodic:
      enabled: true
      schedule: '0 2 * * *'

Backups are written to a dedicated opschain-db-backup persistent volume claim. Two backup methods are available:

  • physical — a pg_basebackup of the database, including the write-ahead log (WAL) written while it runs. This is the recommended method for scheduled backups and is restored using the database recovery steps.
  • logical — a pg_dump (custom format) export. This is portable across database versions and is recommended for the pre-deploy backup, as it can be restored even after an upgrade changes the database schema or major version.

A physical backup is limited to the transfer rate set in db.backup.maxRate (100M by default).

Physical backups on a busy database

A physical backup fails if the database writes more WAL during the backup than the server keeps (wal_keep_size). The job then reports an error and does not save the backup. A lower db.backup.maxRate makes the backup slower, so this is more likely on a busy database.

Which method should I use?

Use physical backups for your regular scheduled backups and for disaster recovery — they are a complete, fast-to-restore copy of the database. Use a logical backup around upgrades — it can be restored into a newer PostgreSQL major version if needed. Keeping both a recent physical backup and a pre-upgrade logical backup covers the common cases.

Copy your backups off the host

With the default local storage, the backup volume lives on a single node and shares a failure domain with that node. Copy your backups to a safe location off the host — for example a file server on your local network — regularly. The backup volume should be treated as a staging area, not as your only copy. See the copying backups off the host section for available options.

Backup encryption​

Backups are compressed with zstd and encrypted with age. OpsChain holds only the public key, so it can write backups but cannot read them. While scheduled or pre-deploy backups are enabled, you must set db.backup.encryption.publicKey to install or upgrade OpsChain.

Generate the key pair with ssh-keygen on a machine outside the cluster:

ssh-keygen -t ed25519 -N "" -C opschain-backup -f opschain-backup-key

This creates the private key opschain-backup-key and the public key opschain-backup-key.pub. The key has no passphrase so the restore steps can read it without a prompt. Keep the file protected. Set publicKey to the contents of opschain-backup-key.pub in your values.yaml:

db:
  backup:
    encryption:
      publicKey: ssh-ed25519 AAAA... opschain-backup
Keep the private key safe

opschain-backup-key is the private key. Keep it offline and separate from your backups. Do not copy it into the cluster. You need it to restore any backup.

In a high availability setup, set the same public key on every cluster. Any cluster can become the primary, and the primary takes the backups.

To change the key, generate a new key pair and set the new public key on every cluster. New backups use the new key. Keep the old private key for as long as you keep backups encrypted with it.

Backup reliability​

Each backup is written to a temporary location and checked while it is written. It is only saved — and counted towards retention — once it has completed and passed that check, so an interrupted or corrupt backup is never mistaken for a good one and never displaces a valid backup. A failed check fails the job.

Each completed backup directory contains:

  • a physical backup: base.tar.zst.age, holding the base backup, its WAL and its backup_manifest
  • a logical backup: globals.sql.age and opschain.dump.age
  • either method: SHA256SUMS and the opschain-backup.json marker. Directories without the marker are considered incomplete.

Retention (db.backup.retention.count and days) is applied to completed backups only, and to each trigger independently. Size the volume (db.backup.storage.size) for roughly your largest expected database multiplied by the retention count, with headroom — with local-path/hostPath storage the volume cannot be resized in place, so err on the generous side. This doesn't apply if you're using Longhorn instead: its storage class supports volume expansion, so an undersized volume can be grown later rather than requiring a fresh one.

Backups and high availability

The opschain-db-backup volume is preserved when a cluster changes role (primary to replica, or back) during failover or failback, and is intentionally retained when OpsChain is uninstalled — see persistent data. Only the primary cluster runs backups, but the volume and its history stay in place so they remain available after a failover.

Copying backups off the host​

In the host filesystem, create a directory to store the backups. Because the K3s data directory /limepoint is root-owned, create this directory once as root and assign it to the opschain group created during the installation process, so the unprivileged installation user can write backups into it without sudo (the setgid bit keeps new files in the opschain group):

mkdir -p /limepoint/backups
chown root:opschain /limepoint/backups
chmod 2775 /limepoint/backups

To extract the backups from the PVC, you can either copy it directly from the volume or use the recovery helper pod. Which one applies depends on your storage class — see the note after each.

  1. Copying directly from the volume: only works with K3s's default local-path storage, where the PV is a plain directory on the host. For those, this is the most efficient option and is easy to schedule (for example a cron job that copies to a file server on your local network).

    Privileged access required

    You must run the commands below as sudo or root user. The volume's directory and the backup files in it are readable only by root and the database's own user.

    The last command gives the opschain group read access to the copies, so the installation user can use them later, for example to copy a backup into the recovery helper pod:

    VOL=$(kubectl -n ${KUBERNETES_NAMESPACE} get pvc opschain-db-backup -o jsonpath='{.spec.volumeName}')
    DIR=$(kubectl get pv "$VOL" -o jsonpath='{.spec.local.path}')
    cp -r "${DIR:?The backup volume is not a local directory, use the recovery helper pod instead}"/. /limepoint/backups/
    chmod -R g+rX /limepoint/backups
    Does not work under Longhorn, or any other non-host-path storage class

    A Longhorn-backed PV has no .spec.local.path at all, so DIR comes back empty and the command silently becomes cp -r "/"/* /limepoint/backups/ — it copies the node's root filesystem rather than failing outright. If you're on Longhorn (or any storage class that doesn't expose the PV as a host directory), use the recovery helper pod below instead.

  2. Using the recovery helper pod, which mounts the backup volume read-only. This works with any storage class, including Longhorn. Start the pod, copy the backups out of it, then stop it again:

    kubectl scale deployment opschain-db-recovery --replicas=1 -n ${KUBERNETES_NAMESPACE}
    kubectl rollout status deployment opschain-db-recovery -n ${KUBERNETES_NAMESPACE}
    POD=$(kubectl -n ${KUBERNETES_NAMESPACE} get pod -l app=opschain-db-recovery -o jsonpath='{.items[0].metadata.name}')
    kubectl cp ${KUBERNETES_NAMESPACE}/$POD:/opt/backup /limepoint/backups/
    kubectl scale deployment opschain-db-recovery --replicas=0 -n ${KUBERNETES_NAMESPACE}

To check that a copy is intact, run this command in its backup directory. You do not need the private key:

sha256sum -c SHA256SUMS
Scheduled backups​

When db.backup.periodic.enabled is true, a CronJob named opschain-db-backup runs on the configured schedule (a standard cron expression). Older backups are pruned according to db.backup.retention.

To pause the schedule without redeploying:

kubectl patch cronjob opschain-db-backup -p '{"spec":{"suspend":true}}' -n ${KUBERNETES_NAMESPACE}

To resume it:

kubectl patch cronjob opschain-db-backup -p '{"spec":{"suspend":false}}' -n ${KUBERNETES_NAMESPACE}

The next deploy sets the schedule back to what db.backup.periodic.enabled says.

You can trigger a backup at any time without waiting for the schedule, including while it is paused. The $(date +%Y%m%d-%H%M%S) suffix gives each run a unique name so repeated runs never clash:

kubectl create job --from=cronjob/opschain-db-backup opschain-db-backup-manual-$(date +%Y%m%d-%H%M%S) -n ${KUBERNETES_NAMESPACE}

A manual job counts towards the CronJob's job history, so later runs remove it like any other finished job. To remove it sooner, delete it by name:

kubectl delete job opschain-db-backup-manual-<suffix> -n ${KUBERNETES_NAMESPACE}
In-flight changes

Scheduled backups run while OpsChain is online and do not stop the API or workers. The database backup itself is consistent, but any changes, workflows or agents that are mid-execution may be captured in an inconsistent state and could resume inconsistently after a restore. To avoid that possibility, scale the API and workers down first, take the backup, then scale them back up:

kubectl scale deployment opschain-api opschain-api-worker --replicas=0 -n ${KUBERNETES_NAMESPACE}
kubectl scale deployment opschain-api opschain-api-worker --replicas=1 -n ${KUBERNETES_NAMESPACE}

If you set api.replicas or apiWorker.replicas in your values.yaml, scale each back to that number instead.

Pre-deploy backups​

Set db.backup.preDeploy.enabled to true to take a backup automatically before every helm upgrade. It is skipped while the database is stopped. This runs as a Helm pre-upgrade hook: if the backup fails, the upgrade is aborted, so you always have a fresh backup before changes are applied. A logical backup is used by default so that it remains restorable across database version upgrades.

note

The pre-deploy backup writes to the same opschain-db-backup volume, which must already exist. Enable scheduled backups (or deploy once with backups enabled) before turning on the pre-deploy backup.

Kubernetes resources (optional)​

Besides the database, you can also opt to use a Kubernetes backup tool to backup and restore OpsChain's Kubernetes resources, such as its deployments, services, etc. Kubernetes backup options include Velero, Kasten K10 and Portworx. These tools are outside the scope of this guide and you should refer to their documentation for more information on how to use them.

Creating a snapshot of your OpsChain resources​

Prior to backing up your resources, we recommend stopping OpsChain. See the stopping and starting guide for more information on how to stop OpsChain.

It is recommended that you backup the entire opschain namespace so that in an unlikely event of a failure, you can get OpsChain up and running after the recovery and restore process. Using the backup tool of your choice, make a snapshot of the opschain resources. In addition to the resources in the opschain namespace, the OpsChain persistent volumes that fulfil the persistent volume claims (in the opschain namespace) need to be backed up as well (e.g. database, Git repos, LDAP, step data). Once the snapshot has been created, you can restart OpsChain.

Recovery​

Restoring an instance has two parts: redeploy with the backed-up values.yaml file, then restore the database from a backup. Follow the path that matches the backup you hold — a physical restore (scheduled backups use the physical method by default) or a logical restore (pre-deploy backups use the logical method by default). The opschain-backup.json marker in each backup directory records the backup method. In a high availability setup, also review the extra requirements for high availability clusters.

Configuration file​

Recover the values.yaml file for the version of OpsChain you're recovering from and use it to redeploy OpsChain. Copy it to the OpsChain's host filesystem and keep a note of its location to use it in the next steps.

Database recovery​

This section restores a physical backup (a pg_basebackup) by replacing the database's data files. For a logical backup (a pg_dump, such as a pre-deploy backup), follow restoring a logical backup instead.

To restore OpsChain from a database backup, first uninstall it as described in the uninstall guide. Uninstalling keeps the opschain-db-backup volume, so its backups stay available to the restore. If you also delete the namespace, copy your backups off the host first.

Turn off pre-deploy backups before recovering

Set db.backup.preDeploy.enabled to false in your values.yaml before starting a recovery.

High availability clusters

If you're restoring from a backup in a high availability setup, check the extra requirements for high availability clusters when restoring from a backup.

Then, do a regular installation of OpsChain as described in the installation guide, but with the additional --set stopped=true and --set stopDatabase=true flags in the Helm command, as described in the deploy OpsChain in stopped mode and hibernating the database sections.

Leave out --wait while OpsChain is stopped

Remove the --wait flag from the Helm command for every deploy in this guide that uses --set stopped=true. While OpsChain is stopped, the volumes of its stopped components are never attached and the secret vault cannot become ready, so Helm waits for the full --timeout and then marks the release as failed. Follow progress with kubectl get pods -n ${KUBERNETES_NAMESPACE} instead. Once OpsChain is started again, use --wait as usual.

OpsChain version

You may try to restore the backup to a newer version of OpsChain, but beware of breaking changes between the versions. It's recommended to first restore the backup to the same version of OpsChain as the backup was created with and then upgrade to the latest version. The backup's opschain-backup.json records that version as opschain_version.

Recovery helper pod

The steps below use a helper pod, opschain-db-recovery, that mounts your backup volume read-only at /opt/backup. It is not running by default, and each restore below shows how to start it. If you are recovering into a fresh namespace where the backup volume is empty, copy your off-host backup into the pod first (shown below), then follow the same extract steps.

The helper pod includes age and zstd. During the restore, you paste the backup private key into its shell. The key is never stored in the cluster.

Secret vault

While the database is hibernated, the OpsChain secret vault cannot reach the database, so it logs errors and may restart. This is expected, and it recovers once the database is running again.

Step 1: Ensure the database pod has been initialised​

Right after deploying OpsChain in stopped mode and with a hibernated database, a few pods will be created to initialise the database and will disappear once the initialisation is complete. You can check the status of the pods by running the following command:

kubectl get pods -n ${KUBERNETES_NAMESPACE}

If the database pod is not present in the list and there are no initdb pods present in the list, it's likely that the database has already been initialised and you can continue with the next step.

If you encounter errors or are unsure of the database status, you can always delete the database cluster and redeploy OpsChain, this will recreate and initialise the database again.

Step 2: Start the recovery helper pod​

Now that the data volume exists, start the recovery helper pod with the database's data volume mounted. In your values.yaml set:

db:
  backup:
    recovery:
      enabled: true
      mountDataVolume: true

Then redeploy with --set stopped=true --set stopDatabase=true, keeping the database stopped and hibernated.

Wait for the pod to start, then exec into it:

kubectl rollout status deployment opschain-db-recovery -n ${KUBERNETES_NAMESPACE}
kubectl exec -it deploy/opschain-db-recovery -n ${KUBERNETES_NAMESPACE} -- bash

Your backups are available read-only under /opt/backup. List the physical backups and choose one to restore:

ls -lah /opt/backup/periodic

Each physical backup directory contains base.tar.zst.age. If a backup directory has no such file, see backups taken before encryption.

If the backup volume is empty — for example when recovering into a fresh namespace — copy your off-host backup into the pod from another terminal session instead, then use that path in the next step:

POD=$(kubectl get pod -n ${KUBERNETES_NAMESPACE} -l app=opschain-db-recovery -o jsonpath='{.items[0].metadata.name}')
for file in $(find /path/to/your/backup/20210101_120000 -type f); do
  kubectl cp $file ${KUBERNETES_NAMESPACE}/${POD}:/tmp/$(basename $file)
done

Then check the copy in the recovery pod shell. Every file should report OK:

cd /tmp && sha256sum -c SHA256SUMS

Step 3: Recover the database files​

In the recovery pod shell, set a variable to the backup you chose (a directory under /opt/backup/periodic, or /tmp if you copied one in):

BACKUP_DIR=/opt/backup/periodic/20210101_120000

Clean up the existing PostgreSQL files in the database directory:

rm -rf /var/lib/postgresql/data/pgdata/*

Load the backup private key into the shell. Run the command below and paste the whole contents of opschain-backup-key. The pasted text is not shown. Then press Enter and Ctrl-D:

stty -echo; BACKUP_KEY=$(cat); stty echo

Decrypt and extract the backup:

age -d -i <(printf '%s\n' "$BACKUP_KEY") "$BACKUP_DIR/base.tar.zst.age" | zstd -dc | tar -x -C /var/lib/postgresql/data/pgdata/

Then remove the key from the shell:

unset BACKUP_KEY

The recovery pod runs as the postgres user (UID 26), so the extracted files already have the correct ownership.

Step 4: Verify the database recovery​

Check the extracted files against the backup's manifest:

pg_verifybackup /var/lib/postgresql/data/pgdata

This should print backup successfully verified. If it reports errors, restart from step 3, or choose another backup.

Exit the recovery pod shell. Step 5 removes the pod.

Step 5: Release the data volume and resume the database​

The recovery pod holds the database's data volume, so it must be released before the database can start. In your values.yaml turn the recovery helper back off:

db:
  backup:
    recovery:
      enabled: false
      mountDataVolume: false

Then redeploy without the --set stopDatabase=true flag (so the database resumes), keeping --set stopped=true for now so you can verify the database before the application starts.

Wait for the database to resume. Follow the creation and the logs of the database pod to ensure it successfully loaded the backup. You can follow the logs by running the following command:

kubectl logs -f pod/<cluster-name>-1 -n ${KUBERNETES_NAMESPACE}

Where <cluster-name> is your db.cnpg.clusterName (opschain-db by default).

If the database pod runs normally for a few minutes and the logs show no errors, it's likely that the database recovery was successful.

Step 6: Start OpsChain​

You can now start OpsChain by redeploying without the --set stopped=true flag. Follow the API and worker logs to ensure they start successfully.

In a high availability setup, reset sites and work on this deploy.

If you're using OpsChain's internal LDAP server, you'll need to recreate the users and groups in that server before you can sign in to OpsChain. Refer back to the creating your first user guide on how to create LDAP users again.

Step 7: Clean up​

The recovery pod was disabled in step 5, so there is nothing further to remove. Do not delete the backups from the opschain-db-backup volume — they remain your restore point. If you copied a backup into the pod's /tmp for the recovery, it was discarded automatically when the pod was removed.

Restoring a logical backup​

A logical backup — such as a pre-deploy backup — is a pg_dump export that is loaded into a running database, rather than by replacing the data directory. Unlike a physical restore, you do not need to hibernate the database — it must be running to accept the restore.

Prepare as described at the start of database recovery: uninstall OpsChain, set db.backup.preDeploy.enabled to false, and leave out --wait while OpsChain is stopped. If you can, restore to the OpsChain version the backup was taken with. Skip --set stopDatabase=true and the recovery helper's mountDataVolume. Only a physical restore needs them.

Restore into an empty database

A logical restore must target a freshly initialised, empty opschain database. Restoring on top of a database that OpsChain has already populated is unreliable. In practice, this means starting from a clean cluster: deploy into an empty namespace (or delete and recreate the database cluster), and keep OpsChain stopped so the application does not create its schema before you restore.

Step 1: Prepare an empty database and start the recovery helper​

Deploy with the application stopped using the --set stopped=true flag, without the --wait flag, as Helm cannot finish waiting while OpsChain is stopped.

This starts the database but keeps the API and workers scaled to zero, so CloudNativePG bootstraps an empty opschain database. If the cluster already contains data, first remove it (see the uninstall guide) and redeploy, so a fresh, empty database is created.

Wait for the database to be ready, where <cluster-name> is your db.cnpg.clusterName (opschain-db by default):

kubectl wait --for=condition=Ready cluster/<cluster-name> -n ${KUBERNETES_NAMESPACE} --timeout=15m

Then start the recovery helper pod:

kubectl scale deployment opschain-db-recovery --replicas=1 -n ${KUBERNETES_NAMESPACE}
kubectl rollout status deployment opschain-db-recovery -n ${KUBERNETES_NAMESPACE}
Step 2: Restore the dump​

Exec into the recovery helper — its environment is already configured to connect to the database (host, credentials and TLS):

kubectl exec -it deploy/opschain-db-recovery -n ${KUBERNETES_NAMESPACE} -- bash

Your backups are available read-only under /opt/backup. Pre-deploy backups are in /opt/backup/pre-deploy, and scheduled logical backups are in /opt/backup/periodic. Each logical backup directory contains globals.sql.age and opschain.dump.age.

If the backup volume is empty, for example in a fresh namespace, copy your off-host backup into the pod from another terminal session. Then use /tmp as the backup directory below:

POD=$(kubectl get pod -n ${KUBERNETES_NAMESPACE} -l app=opschain-db-recovery -o jsonpath='{.items[0].metadata.name}')
for file in $(find /path/to/your/backup/20210101_120000 -type f); do
  kubectl cp $file ${KUBERNETES_NAMESPACE}/${POD}:/tmp/$(basename $file)
done

Then check the copy in the recovery pod shell. Every file should report OK:

cd /tmp && sha256sum -c SHA256SUMS

Choose the dump to restore and load the backup private key into the shell. Run the command below and paste the whole contents of opschain-backup-key. The pasted text is not shown. Then press Enter and Ctrl-D:

BACKUP_DIR=/opt/backup/pre-deploy/20210101_120000
stty -echo; BACKUP_KEY=$(cat); stty echo

Restore the global objects (roles), then the database. Errors about roles that already exist can be ignored:

age -d -i <(printf '%s\n' "$BACKUP_KEY") "$BACKUP_DIR/globals.sql.age" | psql -d postgres
age -d -i <(printf '%s\n' "$BACKUP_KEY") "$BACKUP_DIR/opschain.dump.age" | pg_restore -d opschain --clean --if-exists --no-owner

The restore runs in a single process.

Then remove the key from the shell:

unset BACKUP_KEY
Step 3: Start OpsChain​

If you use the OpsChain secret vault, restart its pods so they read the restored database:

kubectl delete pod -l app.kubernetes.io/name=openbao -n ${KUBERNETES_NAMESPACE}

Then redeploy without the --set stopped=true flag to start the application against the restored database. The deploy also stops the recovery helper pod.

In a high availability setup, reset sites and work on this deploy.

Testing a logical backup (non-destructive)​

To confirm a dump is complete and restorable without touching any database, decode it inside the recovery helper. Start the helper and open a shell in it:

kubectl scale deployment opschain-db-recovery --replicas=1 -n ${KUBERNETES_NAMESPACE}
kubectl rollout status deployment opschain-db-recovery -n ${KUBERNETES_NAMESPACE}
kubectl exec -it deploy/opschain-db-recovery -n ${KUBERNETES_NAMESPACE} -- bash

pg_restore -f reads every object in the archive. Load the key into BACKUP_KEY as above, then run:

BACKUP_DIR=/opt/backup/pre-deploy/20210101_120000
# list the archive's contents
age -d -i <(printf '%s\n' "$BACKUP_KEY") "$BACKUP_DIR/opschain.dump.age" | pg_restore -l | head
age -d -i <(printf '%s\n' "$BACKUP_KEY") "$BACKUP_DIR/opschain.dump.age" | pg_restore -f /dev/null && echo 'archive fully decodes'
unset BACKUP_KEY

If it completes without error the dump is intact and restorable. This is the safest way to routinely verify your backups. To check a copy without the private key, use sha256sum -c SHA256SUMS.

Exit the shell and stop the helper when you are done:

kubectl scale deployment opschain-db-recovery --replicas=0 -n ${KUBERNETES_NAMESPACE}

Backups taken before encryption​

Backups taken by an earlier OpsChain version are not encrypted and have no SHA256SUMS file. Follow the same steps, skipping the checksum checks and the private key, and use these commands to extract and verify.

For a physical backup:

tar xzf "$BACKUP_DIR/base.tar.gz" -C /var/lib/postgresql/data/pgdata/
tar xzf "$BACKUP_DIR/pg_wal.tar.gz" -C /var/lib/postgresql/data/pgdata/pg_wal

To verify it, point pg_verifybackup at the manifest beside the tar files:

pg_verifybackup -m "$BACKUP_DIR/backup_manifest" /var/lib/postgresql/data/pgdata

For a logical backup:

psql -d postgres -f "$BACKUP_DIR/globals.sql"
pg_restore -d opschain --clean --if-exists --no-owner "$BACKUP_DIR/opschain.dump"

After you verify your first encrypted backup, delete the old unencrypted backups, including any copies off the host. Retention also removes old backups from the backup volume over time.

Resetting sites and work after a recovery​

In a high availability setup, a backup brings back the site registrations, cordons and active work of the topology it was taken from. A restored site can come up cordoned, and sites that no longer exist stay listed with work that never runs. To clear them, add this flag to the deploy that starts OpsChain (step 6 of a physical restore or step 3 of a logical restore):

--set db.backup.recovery.resetAfterRecovery=true

Before OpsChain starts, a job runs the database migrations and then:

  • cancels active changes, workflow runs and background tasks, and pending changes already assigned to a site.
  • marks running agents as Failed. Start them again once OpsChain is running.
  • removes every site registration and cordon. Live sites register again, uncordoned, within about a minute.
  • deletes the background jobs queued for sites, pods and work that no longer exist.

The job's log is removed when it succeeds. To see its progress and summary, follow it from another shell as the deploy starts:

until kubectl logs -f job/opschain-recovery-reset -n ${KUBERNETES_NAMESPACE} 2>/dev/null; do sleep 1; done

Removed sites and failed agents are recorded in the audit history.

The job has 15 minutes. If it times out, run the same deploy again and it continues where it stopped. The flag is only accepted while OpsChain is stopped and the database is running, and has no effect on a replica site. Leave it out of later deploys.

Kubernetes resources recovery​

Follow your backup tool's restore procedures if you need to restore a snapshot of your OpsChain Kubernetes resources.