Upgrading from 2026-10-08

Upgrading from 2026-10-08 needs an outage and several steps of its own. Carry them out in this order, alongside the upgrade guide. In a high availability topology, the steps apply to every cluster.

Before upgrading

1. Read the important breaking changes in the changelog, and review the authorisation rules, actions and integrations they name.

2. Run the secret vault settings check and fix what it reports. If your global secret vault is external, follow steps 1 to 5 of if your global secret vault is external instead.

3. If you enabled LDAP sign-in in the OpsChain secret vault by hand, record its configuration.

4. Check your nodes can run the new MintModel API, and size it in your values.yaml, or the number of MintModels rendered at once drops to four. If you use a proxy, replace opschain-mintmodel-steps-api with opschain-mintmodel-api in NO_PROXY.

5. Update your values.yaml:

   • remove the vaultAddress, vaultAuthMethod, vaultToken, vaultUsername, vaultPassword and vaultClientOptions values and the OPSCHAIN_VAULT_* variables, as described in what changes for the bundled secret vault
   • remove the rollStaleSecretVaultPods settings
   • set db.backup.encryption.publicKey to an ed25519 SSH public key, the same on every high availability site, unless you turn off scheduled and pre-deploy backups. See backup encryption
   • keep OPSCHAIN_ENCRYPTION_SEED_KEY or mintPressTransportableKey if you have one, and otherwise set OPSCHAIN_ENCRYPTION_SEED_KEY to a 32-character value. See MintPress transportable key
   • if you forward logs to additional outputs that buffer to a file, move each <buffer> section's path under /data

   helm upgrade fails while any of the secret vault values is set, or while a required key is missing.

6. If you restrict network traffic in OpsChain's namespace, allow step runner and agent pods to reach the opschain-api service on port 3000, or the opschain-log-ingest service if you enable a dedicated log ingest deployment. See the log ingest endpoint.

7. We recommend renewing the database certificates, so they last 10 years rather than 90 days.

8. Turn on maintenance mode, wait until nothing is still running, and stop any running agents. Output from steps and agents started by 2026-10-08 is not stored once the upgrade starts.

9. Stop OpsChain.

Upgrade

Run the upgrade command. If your global secret vault was external, follow step 6 of if your global secret vault is external, which deletes a secret before it.

After upgrading

1. Give the registered external secret vaults meaningful names, and review the runner credentials the upgrade created.

2. If your global secret vault was external, finish steps 7 to 9 of if your global secret vault is external.

3. If you enabled LDAP sign-in in the OpsChain secret vault by hand, hand it over to OpsChain.

4. Rebuild each agent's image before starting the agent again.

5. Ask users to recreate API keys that need secret vault access and, where they need it, to register a secret vault credential. See existing API keys.

6. Turn off maintenance mode.

Move to registered external secret vaults

Projects, environments, assets and templates now use an external secret vault by registering it once and setting vault.code, rather than by setting its address and credentials in their own settings. The global secret vault is always the bundled secret vault. As OpsChain starts after the upgrade, it:

• registers every external secret vault that a project, environment, asset, template or template version connects to, with the codes external_vault_1, external_vault_2 and so on and the names External vault 1, External vault 2 and so on. Each level that connected to one is given its vault.code, and keeps its mount path, encryption and password settings. A level that connected to the bundled secret vault is given vault.code: bundled
• turns the secret vault credentials of a project, environment or asset into a runner credential tagged on that level, so its changes keep using the same access. This applies to:
  • a level using the bundled secret vault with its own token, username or password gets a managed runner credential reaching the mount paths in force across that level and the levels below it (for assets, those of their active template version only). OpsChain stops using the level's own vault account, but does not remove it
  • each level of a group that reaches one external secret vault with different credentials gets a supplied runner credential holding its own credentials. The vault keeps one of the sets as its service credentials
  • every other vault the levels below a converted level use gets a credential tagged on it too, so their changes are not rejected. For an external vault this is a copy of the vault's service credentials, which does not follow a later change to them: update the runner credential too when you change the vault's
• removes the vault addresses and credentials from settings history, events and the stored settings of changes. Finished changes are cleaned in the background, which takes roughly 5 to 10 minutes for every 100,000 changes

After upgrading, give the registered vaults meaningful names in the Vault settings section of the system configuration page. Their codes cannot be changed.

Check your secret vault settings before upgrading

Some settings cannot be moved without guessing, so the upgrade stops on them. Find them first with a read-only check against the release you are running now. Download external_vaults_check.rb and run it in the API pod:

kubectl -n ${KUBERNETES_NAMESPACE} exec deploy/opschain-api -i -- /usr/bin/container_start.sh 'rails runner -' < external_vaults_check.rb

The check changes nothing, and never prints credentials. It prints No external secret vault conflicts found when you can upgrade, or otherwise lists the levels to fix under each of the findings below. Either way it then lists the runner credentials the upgrade will create, the level each is tagged on, and the address of each external vault they reach, so you can review them first.

The findings are:

Finding	Fix
The global secret vault is not the bundled vault	Follow if your global secret vault is external.
These levels reach the same external secret vault with different credentials, and a template level or different TLS options stops them becoming runner credentials	Projects, environments and assets with different credentials become runner credentials, but templates and template versions cannot be tagged, and a runner credential cannot carry its own TLS options. Make the listed levels use the same authentication method, credentials, vault.ssl_verify and client options.
These templates use the bundled secret vault with their own token, username or password	Remove the override, or move the template's secrets to an external secret vault and set its address too. Projects, environments and assets with their own credentials become runner credentials instead.
These templates set secret vault connection settings without an address	Set the address on the template or template version too, or remove its connection settings.
An environment adds secret vault connection settings to these templates for its assets	After the upgrade an asset takes its whole connection from its template, so settings it inherits from its environment, such as a namespace or a credential, would be lost. Set those settings on the template or template version too, or remove them from the environment.
These unfinished changes use a secret vault connection no settings level has	Let the changes finish, or cancel them. Changes under a level that becomes a runner credential are not listed.
These levels set a secret vault setting to a secret-vault:// reference	Replace the reference with the value itself.

Run the check again until it finds nothing, then upgrade.

If you upgrade without running the check and the upgrade finds one of these, the new opschain-api pod does not start, and its log lists the same findings. Roll back with helm rollback opschain -n ${KUBERNETES_NAMESPACE}, fix the settings through the release you upgraded from, and upgrade again. Database changes made earlier in the upgrade stay in place; that release ignores them.

Stop OpsChain before upgrading

The upgrade moves the bundled secret vault's credentials out of the settings and into the database. The release you are upgrading from reads them from the settings, so once they have moved its API cannot store or read any secret: storing or viewing a secret in the GUI or API fails, and so does a running change that resolves one. The upgrade stops that release's workers itself, but its API keeps serving until the new opschain-api pod is ready. Stop both before upgrading, so that OpsChain is unavailable for the upgrade rather than failing part way through requests and changes.

Stopping OpsChain is the last step before helm upgrade: finish the check and, if your global secret vault is external, steps 1 to 5 of that procedure first, as they need the API running. Then make sure no changes or workflows are running, and stop the API and its workers:

kubectl -n ${KUBERNETES_NAMESPACE} scale deploy/opschain-api deploy/opschain-api-worker --replicas=0

helm upgrade starts them again with the new release. OpsChain is unavailable from this command until the new opschain-api pod is ready, which includes the time the database migrations take.

If you upgrade with a full values.yaml saved from an earlier release, remove rolloutGuard.migrationVersion from it so the chart's own value is used. An earlier release's value names a migration that is already applied, so the upgrade skips stopping the workers, and the database changes break the workers of the release you are upgrading from.

What changes for the bundled secret vault

• The bundled secret vault's address, authentication method, client options, ssl_verify, token, usernames, passwords and recovery key are no longer settings. The upgrade moves the credentials into the database, and removes the keys from the global settings and their history. Saving one of them in the global settings is rejected, and an OPSCHAIN_VAULT_* connection variable set under api.env or apiWorker.env is ignored.
• Remove the vaultAddress, vaultAuthMethod, vaultToken, vaultUsername, vaultPassword and vaultClientOptions values, and the OPSCHAIN_VAULT_ADDRESS, OPSCHAIN_VAULT_AUTH_METHOD, OPSCHAIN_VAULT_TOKEN, OPSCHAIN_VAULT_USERNAME, OPSCHAIN_VAULT_PASSWORD and OPSCHAIN_VAULT_CLIENT_OPTIONS variables under env, from your values.yaml. helm upgrade fails while any of them is set.
• OpsChain's own secret vault accounts are always called opschain-api and opschain-runner. If you renamed them with vault.username or vault.runner_username, OpsChain creates the accounts under these names as it starts and deletes the renamed ones.
• To relax certificate verification for the bundled secret vault, set the OPSCHAIN_VAULT_SSL_VERIFY environment variable instead. See secret vault requests fail with certificate verify failed.
• To roll back the upgrade, restore the database from a backup taken before it, then roll back the Helm release. helm rollback on its own leaves the release you upgraded from unable to reach the secret vault.

If your global secret vault is external

The global secret vault can no longer be external. Move it to a registered external secret vault, and install the bundled secret vault as the new global vault, as part of the upgrade:

1. Run the check and fix everything it reports except the global secret vault.

2. Record the global secret vault's connection: its address, authentication method, token or username and password, and client options, from the opschain-vault-config secret and the Vault settings section of the system configuration page. Record its mount path and whether it uses double encryption too: if they are not set, OpsChain used the mount path kv/data with double encryption on.

3. Make sure no changes or workflows are running, and do not start any or store any secrets until step 8 is done. From step 6, every level uses the new, empty bundled secret vault.

4. In your values.yaml, set secretVault.unsealKey and global.secretVaultExternalHostName, and remove openbao.global.enabled: false, openbao.server.enabled: false, the vault* values and the OPSCHAIN_VAULT_* variables. Check that the chart renders with helm upgrade --dry-run before going further.

5. Remove the global vault settings that the system configuration stores in the database, using the release you are running now:

   echo 'ConfigurationService.update(ConfigurationService.data.except(:vault), overwrite: true) or raise ConfigurationService.errors.full_messages.join(", ")' | kubectl -n ${KUBERNETES_NAMESPACE} exec deploy/opschain-api -i -- /usr/bin/container_start.sh 'rails runner -'

   OpsChain then reads the same connection from the opschain-vault-config secret, so nothing changes yet. Confirm that the address is unchanged:

   echo 'puts SecretVaultConfig.global.address' | kubectl -n ${KUBERNETES_NAMESPACE} exec deploy/opschain-api -i -- /usr/bin/container_start.sh 'rails runner -'

6. Stop OpsChain, then delete the opschain-vault-config secret and upgrade straight away, with the upgrade command. The opschain-api-worker pods cannot start without the secret, so do not leave time between the two. The bundled secret vault is set up as on a new installation and writes a new secret.

   kubectl -n ${KUBERNETES_NAMESPACE} delete secret opschain-vault-config

7. Register the vault you recorded in step 2, with a code such as main_vault.

8. On every project that should keep using it, or on narrower levels, set vault.code to that code along with the mount path and double encryption you recorded, for example {"vault": {"code": "main_vault", "mount_path": "kv/data", "use_mint_encryption": true}}. Without them, those levels use the bundled secret vault's mount path. The secrets already in the vault, and the secret-vault:// references to them, then work as before.

9. Ask users to register a credential for the vault, as described in secret vault credentials.

Hand over secret vault LDAP sign-in you set up yourself

OpsChain now sets up LDAP sign-in for the OpsChain secret vault itself. If you enabled the secret vault's LDAP sign-in method at ldap/ by hand before upgrading, OpsChain leaves its configuration unchanged and records an error:secret_vault:ldap_auth event saying so each time the API starts. It still applies superuser and block assignments to it. Follow these steps to let OpsChain manage it instead. OpsChain features that sign users in to the secret vault, such as API keys with secret vault access, rely on the token settings OpsChain writes, so hand it over soon after upgrading.

To keep configuring it by hand instead, set vault_ldap_auth.enabled to false. OpsChain then stops managing the secret vault's LDAP sign-in altogether, including superuser and block assignments, and does not sign users in to the secret vault.

What OpsChain changes

Each time the OpsChain API starts, OpsChain writes these values to the ldap/ sign-in method's configuration, replacing the values you set:

Configuration	Value OpsChain writes
url, starttls, discoverdn, request_timeout	the address from the ldap.host, ldap.port and ldap.enable_ssl settings, false, false, and ldap.search_timeout
binddn, bindpass	vault_ldap_auth.bind_dn and vault_ldap_auth.bind_password if set, otherwise ldap.admin and ldap.password
userdn, userattr, userfilter, groupdn, groupfilter	derived from the ldap.user_base, ldap.user_attribute, ldap.users_filter, ldap.group_base, ldap.groups_filter and ldap.group_attribute settings
groupattr	cn
token_policies, token_period	["opschain-vault-user"] and 8 hours
certificate	the certificate authorities in OpsChain's trust store, if it holds any

Any other configuration, such as upndomain, use_token_groups or token_ttl, keeps its current value. The policies and the user and group mappings you created are kept, apart from OpsChain adding or removing its own opschain-vault-admin policy on them. OpsChain overwrites any policy you created named opschain-vault-user or opschain-vault-admin.

Record the configuration before upgrading

Record the sign-in method's configuration and mappings, signed in to the secret vault with a token that can read them, such as the root token:

bao read auth/ldap/config
bao list auth/ldap/groups
bao list auth/ldap/users

The bind password is not shown, so also note the account and password the secret vault binds with.

Hand it over after upgrading

1. Compare the recorded configuration with the table above, and keep each value you still need:

   • A dedicated bind account: set vault_ldap_auth.bind_dn and vault_ldap_auth.bind_password.
   • Any other value OpsChain would replace, such as a different url or userfilter: add it to vault_ldap_auth.config.
   • Policies your token_policies gave every user: map them to the users' groups, or add them to token_policies in vault_ldap_auth.config, keeping opschain-vault-user in the list, for example "token_policies": ["opschain-vault-user", "my-default-policy"]. Without opschain-vault-user, OpsChain cannot create secret vault tokens for its users.
   • Group mappings: OpsChain names groups by their cn, which is also the name it maps superuser assignments to. If your mappings use a different groupattr, recreate each one under the group's cn.

   Edit the vault_ldap_auth settings in the system configuration.

2. Hand the sign-in method over by setting its description to Managed by OpsChain:

   bao auth tune -description="Managed by OpsChain" ldap/

3. Restart the API to apply the configuration:

   kubectl -n ${KUBERNETES_NAMESPACE} rollout restart deployment/opschain-api

4. Check the result with bao read auth/ldap/config, and that no new error:secret_vault:ldap_auth event has been recorded. Sign in with bao login -method=ldap as a member of a mapped group to check that the token receives the group's policies.

Tokens issued before the handover keep their policies until they expire. Ask users to sign in to the secret vault again to receive tokens with the new configuration.

Setting the description back to anything else stops OpsChain writing the configuration, but does not restore your earlier values. Use the configuration you recorded to restore them.

Renewing the database certificates

The CloudNativePG operator creates the database's certificates. With the current operator manifest they last 10 years. Earlier manifests made them last 90 days, and each renewal of a primary site's certificates stopped its replicas until the new certificates were copied to them. Re-apply the manifest and restart the operator to pick up the new lifetime:

kubectl apply -f https://docs.opschain.io/files/downloads/cnpg-operator.yaml --server-side
kubectl -n cnpg-system rollout restart deployment cnpg-controller-manager

Existing certificates keep their expiry date, and CloudNativePG renews them with the new lifetime when they are within 7 days of expiring. Renewing reloads the certificates and does not restart the database. To renew them now:

1. Make the operator treat every certificate as due for renewal, and restart it:

   kubectl -n cnpg-system patch configmap cnpg-controller-manager-config --type merge -p '{"data":{"EXPIRING_CHECK_THRESHOLD":"90"}}'
   kubectl -n cnpg-system rollout restart deployment cnpg-controller-manager

2. Check that the certificates now expire in 10 years:

   kubectl -n ${KUBERNETES_NAMESPACE} get clusters.postgresql.cnpg.io -o jsonpath='{.items[*].status.certificates.expirations}'

3. Remove the setting from step 1 and restart the operator again, so it stops renewing on every check:

   kubectl -n cnpg-system patch configmap cnpg-controller-manager-config --type json -p '[{"op":"remove","path":"/data/EXPIRING_CHECK_THRESHOLD"}]'
   kubectl -n cnpg-system rollout restart deployment cnpg-controller-manager

High availability

Renewing a site's certificates replaces its CA and its replication certificate, so every replica of that site stops replicating until it has the new ones. After renewing, re-sync the replication certificates on each replica. Renewing at one site renews every database cluster that site's operator manages.