Log ingestion

Understand how OpsChain stores the log output of your changes, workflow runs, agents and background tasks, and what to look for if log output goes missing. After following this guide you should know:

• how each kind of pod gets its log output into OpsChain
• what happens to log output while OpsChain's API or database is unavailable
• which events report log output that could not be stored
• how to size the log buffer with the log_buffer_size setting
• how to run log ingestion in a deployment of its own, if LimePoint support ask you to

Copying your logs to external log storage is a separate step, covered in the log forwarding guide.

How log output is stored

Every log line is stored in OpsChain's database before OpsChain treats it as delivered. A line that is part of a step's output is stored before the step's final status is recorded, so once a step, change or workflow run has finished, its log is complete — reading it straight away returns every line.

How a line reaches the database depends on where it was produced:

• OpsChain's own messages. The status lines, errors and progress messages that the API and API worker add to a log are stored before the code that logged them carries on. They are stored independently of the work being logged, so a message logged by an operation that later fails is still kept.
• Change worker and action-generation pods. When pod_per_change_step is false, the whole change runs in one change worker pod, and that pod writes its steps' output straight to the database. Pods that derive an asset's actions from its template version do the same.
• Step runner and agent pods. When each step runs in a pod of its own, and for agents, the pod collects its output with Fluent Bit and sends it to OpsChain's log ingest endpoint. OpsChain replies only once the lines are stored, and the pod does not exit until every line it produced has been acknowledged.
• Remote runners. A remote runner sends its log output to OpsChain over its existing connection, and OpsChain stores it before acknowledging it. The remote runner waits for that acknowledgement before reporting the step's result.

Each line is stored as one line however long it is. A long line is no longer split into several shorter ones.

The log aggregator is no longer part of this path. It only receives copies of the stored lines, to send to any additional outputs you have configured.

When OpsChain is unavailable

OpsChain is designed to keep log output while the API or the database is briefly unavailable, for example during an upgrade or a high availability failover, rather than lose it or fail the work that produced it.

Step runner and agent pods

While the log ingest endpoint cannot be reached, or cannot store lines, Fluent Bit in a step runner or agent pod keeps retrying, waiting up to 30 seconds between attempts, for as long as it takes. Lines waiting to be sent are kept in the pod's log-spool volume, an emptyDir volume on the node.

Because the pod does not exit until its lines are stored, a step whose action has finished stays running until OpsChain can accept the rest of its log. It then finishes normally.

The amount of log output a pod keeps while waiting is limited by the log_buffer_size setting, 256Mi by default. When a pod has more than that waiting, it drops its oldest waiting lines to make room for new ones. OpsChain notices the gap in the lines it later receives and raises a warning:log_lines:dropped event against the step or agent, recording how many lines were lost. The check is not immediate: the event can take up to about half an hour to appear for a step, and up to about a day for an agent that has been quiet.

OpsChain rejects a batch of lines that can never be stored, such as one that is not valid or is larger than 128 MB, and the pod drops it rather than retrying. Each rejection raises an error:log_ingest:rejected_batch event.

A pod's lines are accepted while its step is running and for 15 minutes after the step finishes (or after the agent stops), so the end of a pod's log is still stored if the pod takes a little longer to send it. Lines that arrive after that are discarded.

The lines a step runner or agent pod produces are also written to the pod's own output, so while the pod exists you can see them with kubectl logs, including any that could not be stored.

Change worker and action-generation pods

A change worker or action-generation pod holds the lines it has not yet stored in memory, and keeps retrying for up to five minutes while the database is unavailable. Lines that still cannot be stored after that are lost. The memory it uses for this is also limited by log_buffer_size. If the limit is reached, the pod drops lines and adds a line to the log saying how many were dropped:

===> 120 log line(s) were dropped because the log buffer was full

When a step finishes with dropped or lost lines in its log, OpsChain raises a warning:log_lines:dropped event against the step recording how many.

OpsChain's own messages

When the database cannot store one of OpsChain's own log messages, OpsChain retries for about 15 seconds. A message that still cannot be stored never stops a step from moving on or from being failed. Instead, OpsChain raises an error:step:log_not_stored event against the step, at most once an hour for each step, and the step continues.

Events

These system events report log output that was not stored, or could not be forwarded:

Event	Raised when
warning:log_lines:dropped	A pod dropped or lost log lines before they were stored, usually because its log buffer filled during an outage. The dropped_count data key records how many. Recorded against the step or agent.
error:log_ingest:rejected_batch	OpsChain rejected a batch of log lines that could never be stored, so the pod dropped it. The event message gives the reason. Throttled to one event per hour for each pod and reason.
error:step:log_not_stored	One of OpsChain's own log messages for a step could not be stored. The step carried on regardless. Throttled to one event per hour for each step.
warning:log_lines:legacy_pods_running	When it starts, OpsChain found pods created by an earlier version of OpsChain that are still running. Their log output is not stored. The pod_names and cluster_name data keys identify them.
error:log_forwarding:failed	Stored log lines could not be sent to the log aggregator for your additional outputs. See log forwarding.
warning:log_forwarding:dropped	Stored log lines waiting to be sent to the log aggregator were discarded because too many were waiting. The lines remain in OpsChain. See log forwarding.

You can subscribe to any of these events to be notified when they occur.

Sizing the log buffer

The log_buffer_size global setting limits how much log output each pod keeps while it waits for OpsChain to store it. It takes a Kubernetes quantity, such as 256Mi (the default) or 1Gi.

• In a step runner or agent pod, it is the size of the on-disk spool. The pod's log-spool volume is given a size limit of one and a half times this value, so the node must have that much ephemeral storage available for each such pod.
• In a change worker or action-generation pod, it is the memory used to hold lines that have not yet been stored, so leave room for it within the pod's memory limit.

Raise it if you see warning:log_lines:dropped events, or log lines dropped because the log buffer was full, after an outage that you expect to recover from. A change to the setting applies to pods started after it is saved. Pods that are already running keep the value they started with.

The log ingest endpoint

Step runner and agent pods send their log output to OpsChain's log ingest endpoint, POST /api/log_ingest/v1/batches. The endpoint is for OpsChain's own pods only — it is not part of the public API, and you never need to call it.

Each pod is given its own token when it is created. A token only allows the pod to add lines to the log of its own step or agent, and is removed from the environment that your action code runs in. Batches are sent in plain HTTP inside the cluster, to the opschain-api service on port 3000 or, if you have enabled a dedicated log ingest deployment, to the opschain-log-ingest service. Pods send to these services directly, ignoring any proxy settings, so if you restrict network traffic in OpsChain's namespace, allow step runner and agent pods to reach whichever of these services is in use.

A batch that a pod sends again — because the reply to an earlier attempt was lost, for example — is recognised and is not stored twice.

Dedicated log ingest deployment

Only do this if LimePoint support tell you to

Most installations should leave log ingestion in the API. Only enable a dedicated log ingest deployment if LimePoint support have asked you to.

By default, the API pods store the log output that step runner and agent pods send them. On an installation where large volumes of log output compete with the API's other work, log ingestion can instead run in a deployment of its own, opschain-log-ingest. It runs the same image as the API.

To enable it, set logIngest.enabled to true in your values.yaml, and redeploy OpsChain:

logIngest:
  enabled: true
  replicas: 1
  resources:
    requests:
      cpu: 200m
      memory: 1Gi
    limits:
      memory: 2Gi

Value	Default	Description
logIngest.enabled	false	Whether to run log ingestion in the opschain-log-ingest deployment.
logIngest.replicas	1	The number of opschain-log-ingest pods.
logIngest.resources	200m CPU and 1Gi memory requested, 2Gi memory limit	The resources of each opschain-log-ingest pod.
logIngest.env	none	Additional environment variables for the log ingest container.

Once it is enabled, step runner and agent pods started from then on send their log output to the opschain-log-ingest service. Pods started before the change keep sending to the API, which still accepts them. In a high availability deployment, each cluster's values.yaml controls that cluster's deployment.

Turning it off again

When logIngest.enabled is set back to false, the opschain-log-ingest deployment is removed. A pod started while it was enabled, such as a long-running agent, cannot store any further log output until it is restarted. Restart any such agents after turning the deployment off.