SIEM/audit export

Included from Basic tier

The push half of the audit lane, shipped as a base capability — never behind a pack: the server streams its event feed to a customer SIEM so security teams wire OpsChain's governance evidence into their existing alerting and retention, and — the otlp_http kind — its correlation chains to a trace receiver so observability and governance towers see each unit of work as one trace. The pull half — evidence-ledger segment export — is a separate surface (opschain docs governance).

The pieces

• SIEM connections (opschain admin siem, v2 SiemService; in the GUI, Administration → Operate → SIEM connections, with a record page per connection that carries its delivery health, the Test verb and the link to its usage dashboard): the admin-managed forwarder records. Three kinds:
  • splunk_hec — Splunk HTTP Event Collector. Events POST to the collector URL as concatenated JSON envelopes authenticated by a write-only HEC token (encrypted at rest; reads serve secret_set only, and no response, audit row or error ever echoes it).
  • syslog — CEF over syslog, UDP/TCP/TLS (TLS verifies against the system roots), to any CEF-speaking receiver (QRadar, ArcSight, Sentinel collectors, rsyslog…).
  • otlp_http — the correlation chain as OpenTelemetry traces over OTLP/HTTP (POST <endpoint>/v1/traces, protocol protobuf by default or json), to Grafana Tempo, Jaeger, Datadog, an OpenTelemetry Collector or a governance tower's trace collector. The token (optional — collectors are often unauthenticated) travels as Authorization: Bearer or in a named auth_header (a tenant header such as X-Scope-OrgID), with the same write-only custody as the HEC token; TLS trust is ssl_verify (default on) plus an optional ssl_ca_cert (PEM inline or a server-host file path). See "Traces" below.
• The forwarding engine: one durable cursor over the events table per connection. Every server replica sweeps; a FOR UPDATE SKIP LOCKED claim on the connection's delivery state is the cross-cluster lease, so N replicas never double-send. Batches deliver at-least-once in (created_at, id) order and the cursor advances (compare-and-swap) only after the receiver acknowledged; the scan trails real time by a short settle window so late-committing rows are never skipped.
• The event-type filter: an optional prefix list per connection (warn:, api:security:, …; a trailing * is accepted). Empty forwards every event.

Payloads

Forwarded payloads carry only the catalogued event row and a fixed envelope — events are secret-free by the standing redaction rule, and the forwarder adds nothing beyond them. Receivers dedup on the explicit event id.

Splunk HEC (one JSON document per event, concatenated per batch):

{"time": 1793615400.000, "host": "<server host>", "source": "opschain",
 "sourcetype": "opschain:event",
 "event": {"id": "<event uuid>", "type": "warn:steps:slow",
           "created_at": "2026-11-02T10:30:00Z", "username": "…",
           "node_path": "/projects/demo", "data": { …the event's data… }}}

CEF (one record per event; over TCP/TLS newline-framed, one datagram each over UDP, each behind a <PRI> + RFC3339 timestamp + host syslog header):

CEF:0|LimePoint|OpsChain|<version>|<event type>|<event type>|<severity>|
  externalId=<event uuid> rt=<epoch ms> suser=<username>
  cs1Label=nodePath cs1=<node path> msg=<the event's data JSON>

Severity maps from the event-type prefix: error 9, warn 6, audit 5, api 3, everything else 1 (syslog priority follows: err/warning/info on facility local0).

Traces — the otlp_http kind

A correlation chain already is a trace: one unit of work, its causes and effects, with timing. The otlp_http kind exports it as one, on the same forwarder — the same durable cursor, cross-replica lease, event-type filter, batch and at-least-once order as the other kinds; only the sender differs. It serves the observability and governance-tower buyer beside the SIEM lanes; it is export only — nothing is read back, nothing on the decide path changes, and the ledger stays the record.

The mapping.

• One trace per chain. The trace id is the chain key's 16 UUID bytes verbatim — coalesce(correlation_id, id), the read-time chain rule — so every event of a chain, across batches and replicas, lands in one trace, and a chain's trace id can be computed from its correlation id alone.
• One synthetic root span per chain, per batch. Named by the unit of work: change <id> when the chain root is a change, run <id> when it is a workflow run, else chain <correlation id>. It starts at the earliest event of the chain seen in the batch and ends at the latest. Its span id is derived from the chain key (the first eight bytes of SHA-256 over a fixed domain prefix and the UUID bytes), so every batch that touches the chain re-emits the same root and receivers merge it.
• One span per event. Name = the event type; start = end = the event time (an event is an instant); span id = the first eight bytes of SHA-256 over the event id's UUID bytes. The parent is the recorded causing event's span (caused_by_event_id) where the chain records one, else the root — spans are flat under the root otherwise, and no causality is ever invented. error:* events carry an error status.
• Idempotent. Every id is a pure function of a stored id, so an at-least-once re-send is byte-identical and receivers dedupe on span id.
• Attributes are identifiers from the catalogued row, under opschain.*: opschain.event.id, .event.type, .event.created_at, .node_path, .correlation_id, .caused_by_event_id, .chain_depth, .source.type / .source.id, .initiated_by.type, .principal_id (the acting principal's opaque id — the audit envelope's, else the initiator's when a user initiated the event), .change_id / .run_id (from the event's source or its data document), .agent / .agent.id on agent events, and on model-call events .model.provider, .model.name, .model.status, .model.input_tokens, .model.output_tokens. The root carries .correlation_id, .chain.kind and the change or run id. Usernames and emails are never attributes: the row's username column and the data document never enter the trace — only identifiers do.
• The evidence anchor. An event an evidence-ledger attestation cites as audit_event evidence carries opschain.evidence.hash (the citing ledger entry's row hash — the hash-chain position) and opschain.evidence.seq, so a receiver can anchor the span back to the tamper-evident record and the sealed segment that comes to cover that seq (opschain docs governance). Uncited events carry no anchor.
• Resource: service.name=opschain, service.version, service.instance.id (the server replica id, else the host), opschain.estate (the licensee named by the active licence) and opschain.installation_id (the licence's installation id).

Encodings. protocol: protobuf (default) sends application/x-protobuf — an ExportTraceServiceRequest encoded with the OpenTelemetry proto bindings. protocol: json sends application/json per the OTLP/JSON mapping (camelCase, hex trace and span ids, 64-bit integers as strings, enums as numbers) for receivers without a protobuf path.

Receivers.

# Grafana Tempo (OTLP/HTTP on :4318; multi-tenant Tempo takes the tenant in X-Scope-OrgID)
opschain admin siem create tempo --kind otlp_http \
  --endpoint-url https://tempo.example.com:4318 \
  --auth-header X-Scope-OrgID --secret-env TEMPO_TENANT

# Jaeger (all-in-one or collector with OTLP enabled, :4318)
opschain admin siem create jaeger --kind otlp_http \
  --endpoint-url http://jaeger.example.com:4318 --ssl-verify=false

# An OpenTelemetry Collector (otlphttp receiver) fronting a governance tower
# or an APM — the collector carries the vendor credential; the connection
# forwards only the governance families
opschain admin siem create tower --kind otlp_http \
  --endpoint-url https://otel-collector.example.com:4318 \
  --ssl-ca-cert /etc/opschain/collector-ca.pem \
  --secret-env COLLECTOR_BEARER --event-filter 'audit:,api:,warn:agent:,info:agent:'

A collector pipeline for the last example:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  otlphttp/tower:
    endpoint: https://traces.tower.example.com
    headers: {authorization: "Bearer ${TOWER_TOKEN}"}
service:
  pipelines:
    traces: {receivers: [otlp], exporters: [otlphttp/tower]}

opschain admin siem test <code> sends one synthetic info:siem:test span (its own chain) through the real encoder and sender.

The usage metrics — opschain.provider.*

Every delivery is counted on the connection's usage row (opschain docs provider-usage): a SIEM kind's events_shipped (per batch) and ship_failures, an otlp_http connection's spans_exported and export_failures, and after each delivery the backlog_depth gauge — the filtered events still past the advanced cursor, counted up to 100,000. opschain admin providers usage --kind siem|otlp --code <code> reads them; the connection's record opens its operational dashboard.

The otlp_http kind is also the metrics export: once a minute, on the cron leader, every provider usage minute row written since the previous minute goes to every enabled otlp_http connection on /v1/metrics (same credential, same protocol, same TLS posture as the traces), so an external dashboard shows the same numbers as the GUI page. One metric per usage metric, named opschain.provider.<metric> — opschain.provider.reads, opschain.provider.tokens, opschain.provider.events_shipped, opschain.provider.api_errors, opschain.provider.backlog_depth … (the full set per family is on the provider usage page) — with the attributes provider.kind (the family: secrets, transport, model, image_build, image_registry, identity, siem, otlp, github, jsm), provider.code and provider.detail where the row has one (the model, the image). A counter or value metric is a cumulative monotonic sum whose start time is its minute bucket's start — a new minute is a new series start, a re-exported minute a larger total of the same start — and a gauge is a gauge; the resource is the trace lane's (service.name=opschain, the replica, the estate). A receiver that refuses the metrics is noticed once per failure streak and retried next minute; the trace cursor is never affected.

Delivery health and failure

Every admin read carries the connection's health: last_delivered_at, lag_seconds (how far the cursor trails the newest event), consecutive_failures and last_error. A refused batch retries with backoff (30s → 2m → 8m → 30m) and never advances the cursor, so nothing is lost while a receiver is down; a sustained failure streak lands a throttled warn:siem:delivery_failing event (one per connection per window) that notifications and triggers can act on. An HTTP receiver's refusal is recorded as a named failure carrying the status (the HEC collector answered 403, the OTLP receiver answered 401) — never the receiver's body and never the token. opschain admin siem test probes a receiver with one synthetic info:siem:test event through the real formatter and sender.

Access and audit

Every verb sits behind the admin-plane authorization determination; writes and tests are audited as api:siem_connections:create|update|destroy|test — never carrying the receiver token. Deleting a connection removes its cursor; disabling pauses the cursor in place for a later resume.