Skip to main content
Version: Next

Log forwarding

Understand how OpsChain's log aggregator works and, if required, configure it to forward build, change and workflow run logs to external log storage. After following this guide you should know how to:

  • add plugins to the OpsChain log aggregator
  • configure build, change and workflow run logs to be sent to external log storage
  • build and use your own log aggregator image

Introduction​

OpsChain uses Fluentd as its log aggregator. Fluentd provides an extensive framework that allows for custom developed and pre-built plugins to be used to forward logs to external log storage solutions. This guide describes the capabilities of OpsChain's log aggregator and provides the steps on how to configure OpsChain to forward build, change and workflow run logs to a Splunk HTTPS Event Collector.

The log aggregator does not store OpsChain's logs. Every log line is stored in OpsChain's database first, as described in the log ingestion guide, and the log aggregator receives a copy of each stored line to send to the additional outputs you configure. Without any additional outputs, it only writes the lines it receives to its own output.

A Kubernetes network policy, opschain-log-aggregator, accepts connections to the log aggregator only from the opschain-api and opschain-api-worker pods.

Capabilities​

OpsChain's log aggregator is provided with a 1GB PVC, mounted at /data, for buffering logs. OpsChain does not use any of it itself, so the whole volume is available to buffer logs for your additional outputs. Refer to Fluentd's buffer configuration guide on how to configure the buffer for your outputs, and keep the buffer path under /data so the buffered logs survive a restart of the log aggregator.

PVC size

If you need to increase the PVC size, you can do so by modifying your values.yaml file and setting the logAggregator.volume.size value to the desired size and (re)deploying OpsChain.

logAggregator:
  volume:
    size: 2Gi

For example, to configure the buffer for the Splunk output plugin, dropping oldest chunks on an overflow and compressing the chunks using zstd, use a configuration similar to the example below:

  <store>
    @type splunk_hec

    host splunk.myco.com
    port 8088
    token <Splunk HEC token>
    use_ssl false

    <buffer>
      @type file
      path /data/my-company/splunk

      retry_forever true
      total_limit_size 100MB
      overflow_action drop_oldest_chunk
      compress zstd
    </buffer>
  </store>
info

It's up to you to ensure your configuration is valid. Please refer to Fluentd's config file syntax guide for more information.

You can check the log aggregator's logs for errors by running the following command:

kubectl logs -n ${KUBERNETES_NAMESPACE} deployment/opschain-log-aggregator -f

Automated restarts​

OpsChain will automatically restart the log aggregator deployment if it's stuck or unresponsive.

While the log aggregator is down, OpsChain carries on as normal: agents, builds, changes and workflow runs still start, and their logs are still stored and visible in OpsChain. Only forwarding to your additional outputs is delayed. See when the log aggregator is unavailable.

If Fluentd never recovers, you can check its logs and events to investigate the issue.

How stored log lines reach the log aggregator​

When a log line is stored, OpsChain also queues a copy of it in the database to be forwarded. A background job on each site sends the queued lines to that site's log aggregator in batches, and removes them from the queue once the log aggregator has acknowledged them. In a high availability deployment, the sites share one queue, so a line may be forwarded by any site's log aggregator — configure the same additional outputs on every site (the settings method does this for you).

Lines are forwarded at least once. If the log aggregator receives a batch but OpsChain does not receive its acknowledgement, the batch is sent again, so an additional output can occasionally receive the same line twice.

When the log aggregator is unavailable​

While lines cannot be sent to the log aggregator, OpsChain keeps them queued and retries, waiting up to a minute between attempts. Once the log aggregator is reachable again, the queued lines are forwarded, oldest first. While forwarding is failing, OpsChain raises an error:log_forwarding:failed event, at most once an hour for each site.

The number of lines that can wait in the queue is limited by the log_forwarding.max_queued_lines setting, 2,000,000 by default. When more lines than that are waiting, the oldest are removed from the queue and will not be forwarded, and OpsChain raises a warning:log_forwarding:dropped event recording how many. The lines themselves remain stored in OpsChain — they are only missing from your additional outputs.

Configure additional outputs​

OpsChain provides multiple options to configure additional outputs for Fluentd, choose the one that best suits your needs.

However you decide to configure the additional outputs, the configuration will be stored in a Kubernetes ConfigMap and mounted into the log-aggregator pod at runtime. You'll need to supply the entire <store></store> string of your configuration to this setting.

The configuration you provide to these settings will be read by Fluentd in the context of the copy output plugin which OpsChain takes advantage of to send the OpsChain logs to multiple outputs.

Under the copy configuration, each <store> directive added instructs Fluentd to send the log entry to an additional target.

note

The specific configuration to include in additionalOutputConfig will depend on the plugin @type used. Please see Fluentd's config file syntax guide, and the relevant plugin manual for further information.

Via settings - post-installation​

You can configure OpsChain settings pre-installation via Helm chart or post-installation via the UI configuration page.

Using the UI configuration page - post-installation​

The simplest way to manage additional outputs is via OpsChain's UI configuration page. For example, to enable sending the logs to Splunk, just paste the entire <store></store> section into the setting and save.

  <store>
    @type splunk_hec

    host splunk.myco.com
    port 8088
    token <Splunk HEC token>
    use_ssl false
  </store>

The log aggregator picks the change up on its own, without restarting: it watches its configuration and asks Fluentd to reload, so log forwarding continues uninterrupted while the new outputs take effect. When in a high availability deployment, the configuration is applied to every site.

Via Helm chart - pre-installation​

The OpsChain Helm chart allows you to provide the additional configuration via the logAggregator.additionalOutputConfig value before installing OpsChain. An example configuration is provided below:

logAggregator:
  additionalOutputConfig: |-
    <store>
      @type splunk_hec

      host splunk.myco.com
      port 8088
      token <Splunk HEC token>
      use_ssl false
    </store>

note

When this method is used, the configuration will not appear in OpsChain's UI configuration page. You can check the ConfigMap to verify the configuration that is being applied.

kubectl get configmap opschain-log-aggregator-additional-output-config -n ${KUBERNETES_NAMESPACE} -o jsonpath='{.data.output\.conf}'

Build your own log aggregator image​

If you need more flexibility than the options provided by the UI configuration page or the Helm chart, you can build your own log aggregator image based on the existing limepoint/opschain-log-aggregator image.

Most Fluentd output plugins can be installed by using the fluent-gem install command. For example, to install the Splunk output plugin your Dockerfile might look like this:

ARG OPSCHAIN_VERSION
FROM limepoint/opschain-log-aggregator:${OPSCHAIN_VERSION}

RUN fluent-gem install fluent-plugin-splunk-enterprise

You may also use the custom Dockerfile to include your company's private CA certificate if the output plugin you are using requires it to verify the TLS connection to your logging infrastructure.

ARG OPSCHAIN_VERSION
FROM limepoint/opschain-log-aggregator:${OPSCHAIN_VERSION}

RUN fluent-gem install fluent-plugin-splunk-enterprise

# add your company's private CA certificate
COPY myco-cacert.pem /etc/ssl/myco-cacert.pem

Once you have added the required customisations to the Dockerfile, build and push the image to your private image registry. An example is provided below:

OPSCHAIN_VERSION='2022-04-11' # EXAMPLE ONLY - To find the current version for your OpsChain instance, you can run the `opschain info` CLI command
docker build --build-arg OPSCHAIN_VERSION="${OPSCHAIN_VERSION}" --tag "image-registry.myco.com/myco/opschain-log-aggregator:${OPSCHAIN_VERSION}-1" .
docker push "image-registry.myco.com/myco/opschain-log-aggregator:${OPSCHAIN_VERSION}-1"
# builds and pushes an image tagged as image-registry.myco.com/myco/opschain-log-aggregator:2022-04-11-1

Configure OpsChain to use your custom log aggregator image​

Once you have built and pushed your custom log aggregator image to your private registry, you can tell OpsChain to use it by overriding the logAggregator.image value in the OpsChain Helm chart.

logAggregator:
  image: image-registry.myco.com/myco/opschain-log-aggregator:2022-04-11-1

If your internal registry requires credentials to pull this image, update the OpsChain imagePullSecret config to allow this image to be pulled. For example:

kubectl edit -n ${KUBERNETES_NAMESPACE} secret opschain-image-secret
danger

Modify the base64 encoded .dockerconfigjson value to add the additional credentials. Don't remove the existing ones.

Once you have configured OpsChain to use your image, you can use the files you added in your additional outputs configuration:

  <store>
    @type splunk_hec

    hec_host splunk.myco.com
    hec_port 8088
    hec_token <Splunk HEC token>
    use_ssl true
    ssl_verify false
    # Using the custom CA certificate
    ca_file /etc/ssl/myco-cacert.pem
  </store>