Operations

The procedures below operate on an existing high availability topology. They use the same two-cluster example as the replica cluster setup guide: a primary cluster North (opschain-db-cluster-north) and a replica cluster South (opschain-db-cluster-south) that replicates it.

By default, OpsChain automatically maps the PGHOST and PGPORT settings to the primary cluster's definition in the values.yaml file, so when the topology changes the OpsChain instance automatically connects to the new primary cluster.

How "cluster" is used here

An OpsChain site is one complete deployment: a Kubernetes cluster running an API, its workers, and its own database cluster. A high availability topology is several sites, kept in step by replicating one site's database cluster to the others.

Throughout this guide cluster means a site's database cluster, so North and South below name each site's database cluster, and a step that acts on a cluster acts on the site holding it.

Procedures

Once you've defined your topology and deployed your clusters, you can perform manual failover and failback operations between your clusters and add or remove clusters to the topology.

To upgrade OpsChain across a topology, follow upgrading a high availability topology — the order in which clusters are upgraded matters, because they share one database.

Deploy OpsChain in stopped mode

If you don't want to keep an entire OpsChain instance running in the replica clusters, you can opt to only deploy the database and its services such that the replication is still happening, but the OpsChain instance (API, workers, etc.) won't be available. This is useful if you want to save compute resources and can use the primary cluster's OpsChain instance for all operations.

To deploy OpsChain in stopped mode, you can set stopped: true at the top of the values.yaml file. For example:

stopped: true

Alternatively, you can simply pass the --set stopped=true flag to the Helm command used for installing and patching OpsChain.

To startup the OpsChain instance again, you can simply remove the stopped: true setting in the values.yaml file and redeploy the cluster or by passing --set stopped=false to the Helm command used for installing and patching OpsChain.

Hibernating the database

Besides stopping the OpsChain instance, you can also hibernate the database using the stopDatabase: true setting at the top of the values.yaml file or by passing --set stopDatabase=true to the Helm command used for installing and patching OpsChain.

When the database is hibernated, the Kubernetes PersistentVolume (PV) and PersistentVolumeClaim (PVC) will be created for it, but no PostgreSQL instance runs in that cluster. This is useful for recovering a cluster from a physical backup or performing maintenance on the cluster.

To resume the database, you can simply remove the stopDatabase: true setting in the values.yaml file and redeploy the cluster or by passing --set stopDatabase=false to the Helm command used for installing and patching OpsChain.

tip

For faster hibernation switching, you can toggle hibernation on and off by annotating the database cluster with the cnpg.io/hibernation annotation instead of redeploying OpsChain. A deploy only changes the annotation when stopDatabase changes, so resume the database the same way, by setting the annotation back to off. For example:

kubectl annotate cluster opschain-db -n ${KUBERNETES_NAMESPACE} --overwrite cnpg.io/hibernation=on

To resume it:

kubectl annotate cluster opschain-db -n ${KUBERNETES_NAMESPACE} --overwrite cnpg.io/hibernation=off

Failover

A failover is the process of promoting a replica cluster to primary while demoting the current primary to a replica, ensuring data is not lost and that the new primary is ready to serve requests as soon as possible. With a simple declarative approach, you can do a planned failover in a few steps. We'll use the North (primary) and South (replica) clusters from the example topology as an example.

Downtime

Failover requires a short downtime of OpsChain. The estimated downtime is the amount of time it takes to apply the changes to both clusters (primary and replica) + the time it takes for the databases to synchronize + the time it takes to redeploy the OpsChain instance such that it connects to the new primary database.

Changes running during a failover

A change whose actions are running in a change pod when the database fails over loses its database connection and fails. Its step log says the database connection was lost and that the change must be re-run. Re-run it once the failover is complete.

A step running in its own step runner pod (in pod_per_step mode) does not need re-running. Once the failover is complete it resumes from its step runner pod, within about 15 minutes, and its log says it was resumed.

Update the primary cluster to become a replica

In the values.yaml file of the North cluster, set the replica.enabled, replica.primary and replica.source settings.

db:
  cnpg:
    replica:
      enabled: true
      # Now pointing to the South cluster as the primary
      primary: "opschain-db-cluster-south"
      source: "opschain-db-cluster-south"

Deploy the then-primary cluster in stopped mode, as described in the Deploy OpsChain in stopped mode section. Once you patch OpsChain with the updated values.yaml file, the North cluster will be in read-only mode.

Check the status of the North cluster:

kubectl get cluster opschain-db-cluster-north -n ${KUBERNETES_NAMESPACE}

Once the cluster notices it's now a replica, it will provide a demotionToken in its status. This token will be used to synchronize the data from the North cluster to the South cluster. You can obtain the token by running the following command:

kubectl get cluster opschain-db-cluster-north -n ${KUBERNETES_NAMESPACE} -o jsonpath='{.status.demotionToken}'

Update the secondary cluster to become the primary

Using the token obtained from the previous step, configure the replica.enabled, replica.primary, replica.source and replica.promotionToken settings in the values.yaml file of the South cluster.

db:
  cnpg:
    replica:
      enabled: false
      primary: "opschain-db-cluster-south"
      source: "opschain-db-cluster-south"
      promotionToken: "<demotion-token>"

Promoting the cluster

The changes in these fields in the promoted cluster must be applied at exactly the same time, otherwise, data loss may occur and the former primary cluster will need to be rebuilt from scratch.

Once you patch OpsChain with the updated values.yaml file, the South cluster will be promoted to primary, copying over the remaining WAL files from the North cluster, switching the WAL timeline and ensuring the North cluster can now fetch information from it. The OpsChain instance will automatically connect to the new primary database once deployed. Remember to deploy the South cluster without the stopped: true setting if you wish to use the OpsChain instance from that cluster.

Remaining clusters

If you have more than two clusters, you must modify the replica.primary and replica.source fields for each remaining cluster to point to the new primary cluster.

info

Once the failover is complete, the now-replica cluster will still have a stale demotionToken in its status, this token can be safely ignored. The promotionToken can be safely removed from the now-primary cluster's values.yaml file as well.

The secret vault in each cluster restarts on its own when that cluster is deployed with the new primary database.

Verify the failover

You can verify the failover has succeeded in a few different ways. First, ensure all clusters are up and running:

kubectl get cluster <cluster-name> -n ${KUBERNETES_NAMESPACE}

All clusters should show Cluster in healthy state in their status.

You can access the logs for each database instance of a cluster using the following command:

kubectl logs <cluster-name>-1 -n ${KUBERNETES_NAMESPACE}

Where <cluster-name> is the name of the cluster.

And, finally, you can verify via the OpsChain instance that writes are working by creating a change or modifying a project's properties via the UI.

The Topology section of the Database tab in the OpsChain GUI shows the same picture without leaving OpsChain: which site's database cluster now holds the data, how far behind each replica is, and whether each site is running work. OpsChain also records an info:database_cluster:role_changed event for the site that took over.

A promoted site keeps any cordon it had. If the new primary shows as cordoned in the Topology section, uncordon it.

Failback

Once a failover is complete, you may want to restore the original cluster as a primary cluster in your topology. You can do so by running the failover process in reverse.

If you already haven't done so, remove the promotionToken field from the values.yaml file of the now-primary cluster when you're demoting it to a replica.

Cordoning a site

Cordoning a site stops new work from starting there, while everything already running there finishes normally. Use it to keep work off a site you are about to maintain, test or retire, without stopping its API or its database.

To cordon a site, open the Database tab of the administration screen, select Cordon on the site's card in the Topology section, optionally give a reason, and confirm. Select Uncordon on the same card to let the site take new work again. Cordoning and uncordoning need the update permission on the /admin/clusters/cordon authorisation path, which a rule granting update on /admin also gives. To do it through the API instead, send PATCH /api/admin/clusters/<site> with cordoned set to true or false — see the API reference.

While a site is cordoned:

• New changes, background tasks and agents start at other sites. This covers refreshing an asset's actions, concretising a MintModel and starting an agent.
• What is already running at the site carries on there until it finishes, including changes stopped at a wait step, which carry on at that site when they are continued.
• Running agents stay at the site. Stop and start an agent to move it to another site.
• Retrying or re-running a change that ran at the cordoned site starts it at another site. Its runner image is built again there, because each site keeps its own images.
• The site keeps itself running. Its API stays available, its database keeps replicating, and its workers keep doing the site's own housekeeping, so it stays visible on the Database tab.

The site's card shows when it was cordoned, by whom and why, and what is still running there, so you can tell when it has finished its work.

OpsChain refuses to cordon a site if no other site would be left taking work, because new work would then wait until a site is uncordoned. A site counts as taking work when it is not cordoned and its workers have reported in within the last five minutes. If another site's workers are restarting or stopped, wait until its card shows Workers active? as Yes, then try again.

When you cordon or uncordon a site, its workers restart in the background after finishing the jobs they are running, so they stop or start picking up the shared work. Work is already kept off the site while that happens. Because each site has its own concurrency limits, the remaining sites take on the cordoned site's share of new work within their own limits, so more work may wait for a free slot while a site is cordoned.

Cordoning and uncordoning are recorded on the audit history page as info:site:cordoned and info:site:uncordoned events.

Adding new clusters to the topology

To add new clusters to the topology, you can simply specify its configuration in the externalClusters section for each cluster already in the topology, ensuring the new cluster's values.yaml sets replica.enabled to true and sets replica.primary and replica.source to the primary cluster's clusterName.

Once the cluster is deployed, it will start replicating from the source cluster as soon as possible. See the replica cluster setup guide for more information on how to configure the new cluster.

info

Ensure that the source cluster can be reached from the new cluster at all times. If the source cluster is not reachable, the new cluster will not be able to replicate from it and will fail to start up.

Check the external service setting section for more information on how to configure the external service for the new cluster.

A new site registers itself once it starts, so it appears in the Topology section of the Database tab on its own within a minute or two of being deployed.

Recovering a replica cluster

A replica cluster needs to be recovered from scratch if it has been down for a long time, such that the primary has deleted the WAL files the replica needs to fully synchronize. This might happen for various reasons, such as:

• The replica's been offline for a long time
• Network connectivity between the primary and the replica has been interrupted for a long time
• The replication stopped working - can be caused by many factors, such as bad certificates, configuration errors, etc.

To recover a stale replica cluster, you must:

1. Delete the replica cluster via Kubernetes: kubectl delete cluster <cluster-name> -n ${KUBERNETES_NAMESPACE}
2. Ensure the certificates and configurations are well defined and the cluster is defined as a replica and pointing to the primary cluster in its values.yaml file.
3. Redeploy the replica cluster using the same command used for deploying it initially.
4. Once the cluster is redeployed, it will start replicating from the primary cluster as soon as possible.

Re-syncing replication certificates

A replica connects to its primary with the primary's replication certificate and CA, copied into a secret on the replica when it was set up. When the primary's certificates are renewed, the replica can no longer connect, and its database logs show SSL error: certificate verify failed. Copy the new certificates to the replica. The database is not restarted.

1. On the primary, extract the certificates as in step 2 of the replica setup, and copy the files to the replica's host.

2. On the replica, replace the secret's contents and label it, so CloudNativePG reloads the certificates:

   kubectl -n ${KUBERNETES_NAMESPACE} create secret generic opschain-db-cluster-north-certs \
     --from-file=tls.crt=north-tls.crt \
     --from-file=tls.key=north-tls.key \
     --from-file=ca.crt=north-ca.crt \
     --dry-run=client -o yaml | kubectl apply -f -
   kubectl -n ${KUBERNETES_NAMESPACE} label secret opschain-db-cluster-north-certs cnpg.io/reload="" --overwrite

3. Check the replica's database logs. Within a minute they show started streaming WAL from primary.

If the logs then show requested WAL segment ... has already been removed, the replica was disconnected for longer than the primary keeps its WAL, and it must be recovered.

Force secret vault primary to run on a cluster

When using the OpsChain secret vault with multiple clusters, the secret vault that will operate as the primary is whichever got a lock to the database first. During failover, upgrades or redeployment of the secret vault the primary instance might change to a different vault.

You can force the primary to be in a specific cluster by scaling down the other cluster's vaults, leaving only the one you want to be the primary.

# In each non-primary cluster
kubectl scale statefulset opschain-secret-vault --replicas=0 -n ${KUBERNETES_NAMESPACE}

Once the vaults are scaled down, the remaining cluster's vault takes the lock and becomes the primary. To confirm it, run this in that cluster. It lists the vault pod once it is the primary:

# In the primary vault's cluster
kubectl get pods -l app.kubernetes.io/name=openbao,openbao-active=true -n ${KUBERNETES_NAMESPACE}

Then scale the other vaults back up:

# In each non-primary cluster
kubectl scale statefulset opschain-secret-vault --replicas=1 -n ${KUBERNETES_NAMESPACE}

Splitting a cluster

If at any point you want to split your clusters, set replica.enabled to false and set replica.primary and replica.source to the cluster's own clusterName in the values.yaml file of the cluster you want to split, then redeploy it. The cluster will then become a standalone primary cluster with the data it had already replicated and will not replicate from any other cluster. Its secret vault and log aggregator restart on their own to connect to it.

Rejoining a cluster

Note that it is not possible to rejoin a cluster to the topology after splitting it. If you want it to get back into the topology, you will need to delete the cluster and redeploy it as a replica, which it will then replicate from zero.

Disaster recovery

Disaster recovery is the process of recovering a cluster from an unexpected failure. Disasters can happen at any time and at any cluster, primary or replica. It's important to devise a disaster recovery plan that takes into account the following scenarios.

Recovery from a replica cluster failure

If a disaster happens to one of your replica clusters, a full recovery might be necessary, depending on how much time it's been offline and how much data the primary cluster retains, as per the WAL retention settings. If you're unsure, you can try to start up the cluster and follow the database logs to see if it can successfully synchronize with the primary cluster. You can verify the database logs by running the following command:

kubectl logs pod/<cluster-name>-1 -n ${KUBERNETES_NAMESPACE}

In case a full recovery is necessary, you can follow the steps outlined in the Recovering a replica cluster section above.

Recovery from a primary cluster failure

If a disaster happens to the primary cluster, without dedicated backups, a recovery can be achieved by promoting one of the replica clusters to primary and updating the remaining clusters to point to the new primary. It needs to be taken into account whether this is the right decision for your use case, such as not to cause a "split-brain" scenario.

The data loss (RPO) will be subject to the replication delay between the then-primary and the replicas. This can vary depending on the size of the database and many other factors that affect the replication delay, such as the cross-cluster network latency, the size of the WAL files, the number of replica clusters, etc.

When running with multiple replica clusters, it's important to consider that each cluster might be in a different stage of the replication process, meaning that the RPO can vary depending on the cluster selected to become the new primary.

You can check the status of the replication process by running the following command:

kubectl get cluster <cluster-name> -n ${KUBERNETES_NAMESPACE}

Where <cluster-name> is the name of the cluster.

The cluster will show the status of the replication process in its status.

Recovery from a backup

If a disaster happens and instead of promoting a replica you want to restore from a previously created backup, you can follow the steps outlined in the database recovery guide, with a few extra considerations:

• First, deploy every replica cluster with --set stopped=true --set stopDatabase=true, so no site runs work against the database while it is restored.
• Reinstall the primary cluster from the backup as described in the guide, resetting sites and work on the deploy that starts OpsChain. Ensure the primary cluster works as expected.
• Once the primary is up, delete the database cluster on each replica cluster with kubectl delete cluster <cluster-name> -n ${KUBERNETES_NAMESPACE}, where <cluster-name> is that cluster's db.cnpg.clusterName, and redeploy it without stopped and stopDatabase. It then replicates the restored primary from scratch.
• If you are using the OpsChain secret vault, restart its pods in every cluster after redeploying so they pick up the new primary cluster's configuration and reconnect to it — delete them (kubectl delete pod -l app.kubernetes.io/name=openbao -n ${KUBERNETES_NAMESPACE}) and let the StatefulSet recreate them.

With these steps, you will be able to restore your OpsChain instance and replicas to a working state as soon as possible.