MintModel API

The MintModel API renders an asset's MintModel: it turns the template version's MintModel and the asset's properties into the concrete model and the actions OpsChain shows for the asset. It runs as a deployment, opschain-mintmodel-api, that stays up and serves every render in the cluster. OpsChain no longer starts an opschain-mm-… pod for each render, so a render no longer waits for a pod to be scheduled and started.

The MintModel API also computes MintModel diffs, which compare two of an asset's MintModels in the GUI and for templated changes. Each pod keeps a transformer process for diffs alone, so a diff never waits behind the renders in progress.

After following this guide you should know:

• what a default install needs from your nodes before the MintModel API will start
• how to size the deployment for the number of renders you run at once
• what to check before upgrading from a version that rendered each MintModel in its own pod
• how to find out why an install or upgrade was refused by the isolation check

What the nodes must support

By default each render is isolated from the others (see keeping renders apart). That isolation is a prerequisite of a default install. Every node the MintModel API can be scheduled onto needs:

• Linux 5.12 or later
• an x86_64 processor
• unprivileged user namespaces enabled (sysctl user.max_user_namespaces reports a value above 0)

The opschain-mintmodel-api container must also be allowed to run with this security context, which the chart sets for you:

• allowPrivilegeEscalation: true
• an Unconfined seccomp profile
• an Unconfined AppArmor profile

Admission policies can block this security context

A cluster that enforces the Kubernetes baseline or restricted Pod Security Standard on the OpsChain namespace, or a Kyverno or Gatekeeper policy with the same effect, rejects a pod with this security context. Exempt the opschain-mintmodel-api deployment and the opschain-mintmodel-api-isolation-check job from the policy, or turn isolation off.

The sysctl value is a first check only. Something else on the node, such as an AppArmor or SELinux policy that restricts unprivileged user namespaces, can still prevent isolation. The authoritative test is the isolation check OpsChain runs before every install and upgrade. It only tests the node its job lands on, so on a cluster with several nodes it can pass while another node can't isolate renders. An opschain-mintmodel-api pod scheduled onto such a node restarts repeatedly, with the reason in its log.

Restricting ptrace

We recommend setting kernel.yama.ptrace_scope to 1 or higher on every node. At 0, a render could read the memory of the transformer process serving another render in the same pod. RHEL, AlmaLinux and Fedora default to 0.

This is a recommendation rather than a requirement. With a value of 0, or on a kernel without the Yama security module, the install continues and the isolation check and the MintModel API's log both carry a warning:

kernel.yama.ptrace_scope is 0: a render could read a concurrently running transformer's memory; set it to 1 or higher for stronger isolation

To check the current value and set it on a node:

sysctl kernel.yama.ptrace_scope

echo 'kernel.yama.ptrace_scope = 1' | sudo tee /etc/sysctl.d/90-opschain-ptrace.conf
sudo sysctl --system

At 1, a process can only be traced by one of its ancestors, so a user other than root can no longer attach strace or gdb to a process that is already running on that node.

Keeping renders apart

A MintModel's templates and the Ruby files in its lib folder run as part of a render. mintModelApi.isolation decides how far one render is kept from the next:

Value	What each render gets
request (the default)	Its own process and its own view of the filesystem. Anything it writes to /tmp, /var/tmp or /opt/opschain, including the home directory, is discarded when it ends. It can't see the files of the other renders running in the pod.
never	Its own process only. Nothing a render loads or changes in memory carries over to the next, but files it writes stay in the pod and are visible to later renders.

Under both values each render also gets a transformer process of its own, which is stopped when the render ends and is never used again.

Isolation assumes trusted MintModels

Isolation keeps renders from interfering with each other by accident. It is not a security boundary between them: every render in a pod runs as the same user. It assumes the MintModels in your project repositories are trusted, reviewed code.

With isolation: request, concurrent renders are kept from each other's files and memory: each render's process is non-dumpable, so another render can't read its memory or environment. Because every render in a pod runs as the same user, the code in one render could still send signals to another render's processes. This affects availability, not confidentiality.

The isolation check

With isolation: request, the chart runs a job named opschain-mintmodel-api-isolation-check at the start of every helm install and helm upgrade. It runs the MintModel API image with the same security context as the deployment and tries to isolate a render on the node it is scheduled onto. It does not check your other nodes. If it can't isolate the render, the job fails, and the install or upgrade fails with it before any OpsChain component is replaced.

The job's log names the reason:

kubectl -n ${KUBERNETES_NAMESPACE} logs job/opschain-mintmodel-api-isolation-check

Log message	What to check
unprivileged user namespaces are not available	user.max_user_namespaces is 0 on the node, or the pod is not running with the security context listed above.
mount_setattr (Linux 5.12+) or nested mount namespaces are not available	The node's kernel is older than 5.12.

If the job's pod was never created, an admission policy rejected its security context. kubectl -n ${KUBERNETES_NAMESPACE} describe job opschain-mintmodel-api-isolation-check shows the rejection in its events.

The job is removed once it succeeds and is kept when it fails, so its log is still there after a failed install or upgrade. It does not run while OpsChain is stopped.

The MintModel API repeats the check each time its container starts, and refuses to start if it fails. It never falls back to running renders without isolation. A node the check did not run on, or one that stops meeting the requirements after the install, for example a node added to the cluster with an older kernel, therefore shows up as an opschain-mintmodel-api pod that restarts repeatedly, with the reason in its log:

kubectl -n ${KUBERNETES_NAMESPACE} logs deployment/opschain-mintmodel-api

Checking your nodes before upgrading

The job runs on only one node. To run the same check on every node before you install or upgrade, download check-mintmodel-api-nodes.sh and run it with kubectl access to your cluster, giving the OpsChain version you are installing:

bash check-mintmodel-api-nodes.sh -v <version> -n ${KUBERNETES_NAMESPACE}

It starts one short-lived pod on each node, with the same image and security context as the job, prints PASS or FAIL and the reason for each node, and exits 1 if any node fails. Use -l <selector> to check only the nodes your nodeSelector chooses. The pods pull the image with the opschain-image-secret secret if the namespace has one, or the nodes' own registry credentials otherwise; use -s to name another pull secret, or -i for an image from your own registry.

Turning isolation off

If your nodes can't meet the requirements, set isolation to never in your values.yaml and redeploy:

mintModelApi:
  isolation: never

The isolation check is then skipped, and the container runs with allowPrivilegeEscalation: false and the default seccomp and AppArmor profiles. Files a render writes inside the pod stay there for later renders to read until the pod is replaced.

Sizing the deployment

The deployment is configured under mintModelApi in values.yaml. Redeploy to apply a change.

replicas, concurrency, resources and transformer.heap can also be changed while OpsChain is running, from the administration screen's Deployments tab, the opschain admin deployments CLI commands, or the API. A change made that way applies to that site only and is not written back to your values.yaml, so a later upgrade that changes the same value returns the deployment to it. See changing the sizing at runtime.

mintModelApi:
  enabled: true
  replicas: 2
  concurrency: 2
  isolation: request
  resources:
    requests:
      memory: 512Mi
    limits:
      memory: 6400Mi
  transformer:
    spares: 1
    heap: 1536m
  networkPolicy:
    enabled: true
  terminationGracePeriodSeconds: 630

Value	Default	Purpose
enabled	true	Deploys the MintModel API. Leave it true: with false, MintModels cannot be rendered, because OpsChain has no other way to render them.
replicas	2	The number of opschain-mintmodel-api pods. Must be 1 or greater unless OpsChain is stopped: with no pods, OpsChain has nowhere to render MintModels.
concurrency	2	How many renders one pod runs at the same time. Each pod also keeps one more transformer process for MintModel diffs. Must be 1 or greater.
isolation	request	request or never. See keeping renders apart.
resources	512Mi request, 6400Mi limit	The memory and CPU each pod requests and may use, in the same shape Kubernetes uses. See how much memory a pod needs.
transformer.spares	1	How many idle transformer processes each pod keeps started so that a render does not wait for one to start, between 0 and concurrency. See spare transformers.
transformer.heap	1536m	The largest Java heap each transformer process may use, as a number followed by k, m or g in either case. These are read as Java reads them: kibibytes, mebibytes and gibibytes, so 1536m is 1536Mi. Transformers use the G1 garbage collector.
networkPolicy.enabled	true	Restricts who can reach the MintModel API. See who can call the MintModel API.
terminationGracePeriodSeconds	630	How long a stopping pod may keep running to finish the renders it has in progress. See upgrades and restarts.

How many renders run at once

Each cluster renders up to replicas × concurrency MintModels at the same time: four with the defaults. Further renders wait their turn and start as running ones finish.

OpsChain reads this figure from the running opschain-mintmodel-api deployment and reports it as the concurrent.mintmodel_limit setting, which can no longer be edited. Only pods that are ready and running the deployment's current specification are counted, so the figure is lower while pods are starting, cannot be scheduled or are being replaced. Because it is read from the deployment rather than from your values.yaml, it follows a change made at runtime from the Deployments tab, the CLI or the API, or made any other way, for example with kubectl scale, within about a minute of the new pods becoming ready. To change how many renders run at once, change replicas or concurrency:

• Lower either of them to reduce the load renders put on the cluster, and on the OpsChain API and secret vault when they read secrets.
• Raise either of them to render more at once. Raising replicas adds pods, each with its own memory limit. Raising concurrency makes each pod do more, so raise its memory limit to match.

In a high availability deployment each cluster has its own deployment, sized by that cluster's values.yaml.

While OpsChain is stopped the deployment has no pods and concurrent.mintmodel_limit is 0. Renders requested in the meantime wait until it is started again.

How much memory a pod needs

All the renders in a pod share its memory limit. One container runs the Ruby server, which runs each render in its own process and starts a Java transformer process for each render in progress. It keeps transformer.spares idle transformers started, and they count within concurrency. It also keeps one more transformer for MintModel diffs, so a diff never waits behind renders. A pod therefore runs at most concurrency + 1 transformer processes. The memory limit must cover:

100Mi + concurrency x 320Mi + (concurrency + 1) x (transformer.heap + 320Mi)

That is 100Mi for the server, 320Mi for each render process, and each transformer's heap plus 320Mi of Java overhead. With the defaults it is 100Mi + 2 x 320Mi + 3 x (1536Mi + 320Mi) = 6308Mi, and the default limit of 6400Mi (6.25Gi) adds headroom above that. The chart checks this: an install or upgrade whose resources.limits.memory is too small fails, and the error gives the size needed. Work the formula through again and raise resources.limits.memory whenever you raise concurrency or transformer.heap. For example, with concurrency at 4 and the default heap it is 100Mi + 4 x 320Mi + 5 x 1856Mi = 10660Mi.

Write resources.limits.memory as a Kubernetes memory quantity, such as 6400Mi or 6.25Gi. Mi and Gi are the usual units, and plain bytes, M and G also work. Units are case sensitive: a lower-case m means thousandths of a byte in Kubernetes, so 6400m, and units such as mi, are rejected with a message saying so. transformer.heap is the exception, and accepts k, m and g in either case, as Java does.

The default heap of 1536m renders a large SOA MintModel, with many hosts and steps, within the default render timeout. A MintModel that needs more fails within about a minute with The MintModel transformer ran out of Java heap; raise transformer.heap, and resources.limits.memory with it, if this happens. The memory a render may use is no longer set for each project, environment or asset: the pod_templates.mintmodel_api setting has been removed, and mintModelApi.resources applies to every render in the cluster.

When a render runs out of memory

Renders share the pod's memory, so one render can use more while the others use less. If the renders in progress take the pod close to its memory limit, the MintModel API stops the render using the most memory once the pod reaches 90% of its limit, and the other renders carry on. Without this, Kubernetes would kill the whole container, and every render in it would fail with it.

The stopped render fails with an out of memory error that gives how much memory it and the pod were using, and asks you to raise mintModelApi.resources.limits.memory or lower mintModelApi.concurrency. Do one of these if renders fail this way regularly. To do it straight away, without an upgrade, change the sizing at runtime, and make the same change in your values.yaml so the next upgrade keeps it. A render stopped for running out of memory is not retried, because the same MintModel usually needs the same memory again, unless you turn on the mintmodel_retry_out_of_memory setting.

A transformer can also run out of its own Java heap, which is set by transformer.heap rather than the pod's memory limit. The MintModel API stops it as soon as its heap is full and it is spending most of its time collecting garbage, so the render fails within about a minute with The MintModel transformer ran out of Java heap rather than running on until the render timeout. Raise transformer.heap, and resources.limits.memory to match the formula, if this happens. It is never retried, even with mintmodel_retry_out_of_memory on, because the same MintModel runs out of the same heap again.

Changing the sizing at runtime

To scale or resize the MintModel API without a helm upgrade, for example while renders are failing because they run out of memory, use the Scale and Resources buttons on the opschain-mintmodel-api row of the Deployments tab. Both open with the deployment's current values. Scale changes replicas, up to 5. Resources changes resources, concurrency and transformer.heap. The CLI's opschain admin deployments scale and opschain admin deployments resources commands, and the POST /api/admin/deployments/opschain-mintmodel-api/scale and .../resources endpoints, do the same.

• OpsChain applies the same memory rule as the chart: a memory limit below what the new concurrency and transformer.heap need is rejected, and the error gives the size needed. It applies no upper limit of its own, so a pod asking for more memory than any node can give stays pending while the existing pods keep serving.
• The number of renders this site runs at once rises as the new pods become ready. A render that arrives while new pods are starting waits for a free slot.
• transformer.spares cannot be changed this way, and concurrency cannot be set below it. Change it in values.yaml and redeploy.
• The change applies to the site whose API you are connected to. In a high availability deployment, change each site.
• The change is not written back to values.yaml. An upgrade that changes the same value in the chart returns the deployment to it, so make a change you want to keep in values.yaml as well.

Each change is recorded as an event (api:deployments:scale or api:deployments:resources) giving who made it, and for a resize the values the deployment had before.

Before upgrading from per-render pods

Versions before the MintModel API rendered each MintModel in its own pod, and the concurrent.mintmodel_limit setting decided how many ran at once. From the upgrade on, the stored setting is ignored, and it is removed in a later release: the limit is replicas × concurrency, four with the defaults. An installation that ran 10 or 30 renders at once would drop to four unless you size the deployment first. Before you upgrade:

1. Note your current limit: open the system configuration page in the GUI and find concurrent.mintmodel_limit in the global settings.
2. Set mintModelApi.replicas and mintModelApi.concurrency in your values.yaml so that replicas × concurrency is at least that limit. Each extra replica is another pod for the scheduler to place, so prefer raising concurrency on clusters with few nodes.
3. Set mintModelApi.resources.limits.memory from the formula in how much memory a pod needs, for your concurrency and transformer.heap.
4. Check for render settings that no longer apply. Per-render pods took their memory and CPU from pod_templates.mintmodel_api.resources, which could be set globally or for a project, environment, asset or template, and their Java heap was 75% of that memory limit (1536m with the default 2Gi). The upgrade removes those settings. Every render in the cluster now shares mintModelApi.resources, and each transformer's heap is mintModelApi.transformer.heap (1536m by default). If you raised the memory limit for nodes with large MintModels, raise transformer.heap to about 75% of that limit, and raise resources.limits.memory to match the formula in how much memory a pod needs.
5. Check where renders will run. Per-render pods ran on the nodes chosen by the runner.node_selector setting, with the runner pods. The opschain-mintmodel-api pods run where your other OpsChain components run, on the nodes chosen by nodeSelector in your values.yaml, at the opschain-system priority. Neither ever supported tolerations or affinity.
6. If your values.yaml sets mintModelStepsApi, move any env or resources you rely on to mintModelApi. The mintModelStepsApi block is ignored from this version.
7. If you use a proxy, replace opschain-mintmodel-steps-api with opschain-mintmodel-api in your NO_PROXY list. See determine the no_proxy service list.

The upgrade records an event naming your old limit and the capacity the deployment gives instead, so you can confirm the two match afterwards. It also removes every pod_templates.mintmodel_api setting, wherever it was set, because the memory a render may use is now set by mintModelApi.resources for the whole cluster. It records an event for each one that differed from the default, giving the old value and where it was set.

Spare transformers

Every transformer process serves one render and is then stopped, so each render needs a freshly started one. transformer.spares decides whether that start happens ahead of the render or during it:

• With spares at 1 or more, a pod reports ready once a spare has started. A render takes the waiting spare and the pod starts a replacement straight away.
• With spares at 0, a pod reports ready as soon as its server is up, and every render waits about one to two seconds for its transformer to start.

Before a spare is ready, it runs one transform of a small model bundled with the MintModel API, so that Java has already loaded and compiled the code a transform needs. This makes the first transform it serves noticeably faster, and an idle spare uses about 500Mi, within its share of the memory formula. A transformer started because no spare was ready skips this step, so its render does not wait for it.

Setting spares to 0 lowers the memory an idle pod uses, at the cost of that wait on every render.

When a render fails

A render that fails for a reason other than its MintModel, for example because it overran the mintmodel_render_timeout deadline or its process was killed, is retried up to three times. A render stopped for running out of memory is not retried unless mintmodel_retry_out_of_memory is on.

If OpsChain can't connect to the MintModel API, it tries twice more, waiting up to two seconds between attempts, before giving up. While one pod is restarted or replaced, renders reach the other ready pods, so this does not usually fail the render. When no opschain-mintmodel-api pod is serving at all, the render fails, usually within a few seconds, with the MintModel API is unavailable, check that the opschain-mintmodel-api pods are running and ready. That failure is not stored against the asset, so its MintModel is rendered again once the MintModel API is back.

Upgrades and restarts

An upgrade replaces the opschain-mintmodel-api pods only when their specification changes, for example when the OpsChain version or a mintModelApi value changes. Otherwise they keep running.

When a pod is stopped, by an upgrade, by lowering replicas or by draining its node, it first waits 10 seconds so that the Kubernetes service stops sending it new renders. The renders it has in progress then finish, for up to terminationGracePeriodSeconds in all. A render may take up to the mintmodel_render_timeout, 300 seconds by default, and then as long again to resolve its steps, so the default of 630 covers twice the default render timeout and that wait. If you raise mintmodel_render_timeout, raise mintModelApi.terminationGracePeriodSeconds by twice the amount, or renders that run that long are cut off when a pod is replaced.

Who can call the MintModel API

Renders carry the asset's properties and a key that lets the render read secrets through the OpsChain API, so the MintModel API is closed to everything but OpsChain itself:

• With networkPolicy.enabled: true, a Kubernetes network policy named opschain-mintmodel-api accepts connections to the MintModel API only from the opschain-api and opschain-api-worker pods. It relies on your cluster's network plugin enforcing network policies, as K3s does by default.
• Every request must present a token that the chart generates once per install and stores in the opschain-mintmodel-api-token secret. This applies whether or not the network policy is enabled.

Set networkPolicy.enabled to false only if your cluster manages network policies another way and the generated one conflicts with it.

Secret vault access in renders

A render holds no secret vault credentials. When the MintModel ERB reads the secret vault, the read goes through the OpsChain API, as it does for an action:

• A direct call in the ERB, such as SecretVault.get, uses the asset's vault: the vault settings of the asset, or the nearest level above it that sets them.
• MintPress's PasswordVault accepts the same calls as SecretVault, get, get_password, remove, remove_path and vault, and reaches the same vault.
• As in an action, a call may store a value in the vault, for example a generated secret or a default: value for a secret that does not exist.
• A secret-vault:// value under the mintpress property, including mintmodel_overrides, is resolved before the ERB runs, through the vault of the level that supplied it. The ERB receives the value exactly as SecretVault.get would return it: the encrypted value, or the plain value when the vault holds it unencrypted.
• A secret-vault:// value anywhere else in the properties is not resolved during a render, and reaches the ERB as the reference text.

A MintModel is rendered when its asset is concretised, so a value changed in the vault reaches the MintModel only when the asset is next concretised.

Checking it is running

kubectl -n ${KUBERNETES_NAMESPACE} get deployment opschain-mintmodel-api
kubectl -n ${KUBERNETES_NAMESPACE} get pods -l app=opschain-mintmodel-api

Every replica should be Ready. With transformer.spares at 1 or more, a pod that is running but not ready is still starting its first spare transformer.

Kubernetes checks each pod's /readyz endpoint every 10 seconds and takes the pod out of service after three failures in a row. It checks /healthz every 30 seconds and restarts the container after three failures in a row. A pod whose render process manager (its render zygote) has died answers both with 503 and is restarted, and the renders sent to it in the meantime are retried on another pod.

A pod that keeps restarting has one of three causes. Its log from before the restart, and its events, show which:

kubectl -n ${KUBERNETES_NAMESPACE} logs <pod name> --previous
kubectl -n ${KUBERNETES_NAMESPACE} describe pod <pod name>

Cause	What to look for
Its node fails the isolation check.	One of the messages listed under the isolation check, in the log.
Its render zygote died.	Failed liveness probes answering 503 in the pod's events.
Its transformer processes keep failing to start.	the transformer on port … failed to start (… in a row); its output was: … in the log, followed by the transformer's own output, which gives the reason.