Notification

Included from Basic tier

The interest-based notification system (waves N1–N4): an operator declares what they care about (a scope and conditions) and the platform matches, authorises, groups, renders and delivers — orthogonal to execution, never embedded in a change or workflow. The surface is API v2 only (NotificationService), strictly scoped to the authenticated caller's own rows, and PostgreSQL-backed (the embedded SQLite store answers the typed ERROR_REASON_STORE_POSTGRES_REQUIRED).

Three resources:

• Subscription — Scope × Conditions × Route (× Controls). The scope is either estate with a node-path selector in the authz-rule path grammar (/projects/foo, /projects/*/environments/prod, ** any depth; a scope covers its whole subtree, "" is root), or platform for the admin-object happenings (runners, agents, licence, users/groups, maintenance, platform posture, database). Conditions: object_kinds, event_kinds (exact event types, a trailing : matching as a prefix), state_predicates (started, succeeded, failed, cancelled, waiting_on_gate), a severity_floor (info < warn < error) and attribute_predicates — {field, pattern} pairs that only ever narrow: field names an event attribute (action_path is the only field this wave; an unknown field is refused), and pattern is a glob in the scope-segment grammar (*, ?, [...]) matched against the attribute's full string value, so * crosses the : separators of an action path — *start* covers web:start:run. Every predicate must match, and an event that carries no value for the attribute (e.g. a node event with no run) never matches a subscription holding a predicate on it. Controls: a route reference (absent = the always-on in-app default), a cadence (default/realtime/hourly/daily — default is the default posture: realtime for failures/errors and approvals-waiting, daily digest for all else; the explicit values override it per route), enabled, muted and snoozed_until.

• Route — a per-user delivery target: in_app, or a connector-backed kind bound to an admin-owned channel connector of the same kind — webhook, whose config_json names the connector_id, a target url inside that connector's allowlist, and a payload template (generic default, slack, teams); or slack / teams (base channel kinds), whose config_json names the connector_id only — the connector holds the incoming-webhook URL (a capability secret, HMAC n/a) and the kind implies the payload shape (Slack Block Kit / Teams Workflows Adaptive Card). See reference/resources/notification-connector.

• Notification — one dispatched record per (user, event): the in-app feed row and the delivery-evidence record in one. thread_key groups one thread per change/workflow run; summary is rendered from an allowlisted field set only (event type, classification, node path, action path) so resolved secrets and PII never enter a payload; deep_link carries the reader to the authz-checked subject.

• MandatedSubscription — org policy: a subscription shape an administrator attaches to one group, whose members cannot delete, mute, snooze or disable it. Group membership resolves through the LDAP directory mirror (the approver-group posture). Members see their mandates via ListMandatedNotificationSubscriptions; the subscription verbs aimed at a mandate that binds the caller answer the typed ERROR_REASON_MANDATED_SUBSCRIPTION refusal naming the policy (a mandate that does not bind the caller stays a 404). Mandate rows dispatch in-app and realtime, and a matching mandate's id is stamped onto the dispatched row (mandate_id) even when the member's own subscription won the (user, event) claim — the delivery evidence records the org's coverage either way. Administration (ListNotificationMandates / CreateNotificationMandate / UpdateNotificationMandate / DeleteNotificationMandate on NotificationAdminService) sits behind the /admin grant AND the packs:security-governance entitlement (the typed ERROR_REASON_PACK_ENTITLEMENT_REQUIRED refusal names the pack); enforcement of existing mandates never consults the licence — the controls are free, the admin surfaces are the pack. Instance-wide bounds: unique names, 500 mandates.

• TeamSubscription / TeamRoute (FAQ team-channel gap) — group-owned twins of the subscription and the webhook route: one subscription owned by a GROUP (same Scope × Conditions grammar, attribute predicates included), delivered ONCE per event through a group-owned webhook route bound to an admin connector — never fanned out per member, so a shared Slack/Teams channel gets one post however large the team. Members see their groups' rows read-only on /notification_settings ("Team watches"); management sits behind the per-group admin determination (updatable on /admin/groups/<group>, which every /admin-plane grant covers — a scoped rule can delegate one group). mirror_in_app optionally lands an in-app mirror row per member (matched_by: team; off by default — mandates remain the guaranteed-in-app instrument). The authz overlay evaluates the GROUP's own path-RBAC read grants — never a union of members' personal grants: creating/pointing a group subscription at an estate scope requires the group to hold read on the scope's literal prefix (404-on-deny), the engine matches events against the group principal's grants, and dispatch re-checks the same (a revocation records a suppressed group row). The group claim row (user_id NULL, subscription_id = the team subscription, migration 00062's partial unique claim) is the delivery-evidence record naming the group subscription; QueryNotificationDeliveries carries its group_name. Team verbs: ListTeamNotificationSubscriptions / CreateTeamNotificationSubscription / UpdateTeamNotificationSubscription / DeleteTeamNotificationSubscription and the route CRUD (ListTeamNotificationRoutes / CreateTeamNotificationRoute / UpdateTeamNotificationRoute / DeleteTeamNotificationRoute; connector-backed only — webhook, slack or teams; there is no group in-app feed). Group membership resolves through the provider-group snapshots folded with the local group memberships.

Authorisation

Every call is caller's-own-rows — no request names a user. Creating an estate subscription additionally requires read on the scope's literal prefix path (404-on-deny); a platform subscription requires the admin grant (/admin updatable, permission_denied). The dispatcher applies an authorisation overlay: match results are intersected with the subscriber's read authz at match time, and dispatch re-checks against a fresh snapshot — a revocation in between records a suppressed row (redacted to the event id) instead of a delivery, and losing access stops the notifications.

Zero-config defaults

With no subscriptions at all, every user implicitly gets (in-app only): runs they created reaching a terminal state (matched_by: my_activity), and approvals waiting on them (matched_by: approval):

• Step wait gates — the gate's named approvers, resolved directory-group members, or the initiator for a plain wait step.
• Approval checklists — when a subject enters the approval funnel (a governed change or workflow run held pending_approval, a trust raise, a release promotion, a transport grant; a validity lapse that re-gates the subject counts as a fresh entry), every identity the approvals inbox would show it waiting on you to: named on a checklist line or a member of a group the line names, with that line still unsatisfied, the requester and change owner excluded exactly as the inbox excludes them. The row reads <Kind> awaiting your approval: <target> · <action> — requested by <requester> (kinds: Change, Workflow run, Trust raise, Release promotion, Transport grant), its deep_link is the inbox (/governance/reviews), its state is waiting_on_gate (so the realtime cadence applies to routed copies) and its authorisation is the inbox's own — read on the checklist target, not on the event's node path. The ceremony's closure (released, rejected, expired, break-glassed) reaches the same identities as <Kind> approval <outcome>: <target> · <action> in the same subject thread (thread_key is the run for a change/workflow run, otherwise <run_kind>:<subject id>), superseding the pending call to action. A user who is also subscribed to the subject keeps their subscription's route and attribution but receives the approver wording and link.

My-activity preference

The my-activity arm honours a per-user opt-down, read and written via GetNotificationPreferences / UpdateNotificationPreferences (caller's-own-rows; the GUI surfaces it at the top of /notification_settings as a pinned pseudo-subscription):

my_activity	Effect on the caller's own runs
all	Every terminal state notifies (the zero-config default — a user with no stored row gets this).
failures_only	Only the failed terminal state (errored/failed runs); successes and cancellations land no row.
off	No my-activity rows at all.

An unknown value is refused (invalid_argument). The preference governs only the my-activity arm: approvals-waiting notifications are actionable, not informational, and keep flowing even at off; explicit subscriptions and mandated subscriptions are likewise untouched. The engine reads the opt-down rows fresh each sweep, so an edit applies within one sweep interval.

Dispatch

The notification engine is a cursor-based consumer of the typed event stream (the trigger-engine idiom): every replica sweeps; exactly-once across the HA topology comes from the UNIQUE (user_id, event_id) claim — concurrent sweepers race one set-based insert and the loser skips. The cursor seeds at the newest event on first boot, so history is never replayed against later subscriptions. In-app delivery is the persisted row (delivery_status: delivered); a persistently failing engine emits warn:notifications:engine. A connector-routed match (webhook, slack or teams) lands queued instead and the delivery sweeper (every replica; the claim moves next_attempt_at forward under FOR UPDATE SKIP LOCKED, so the shared database is the cross-cluster lease) sends the payload — signed for the webhook kind; posted unsigned to the connector's stored capability URL for slack/teams — with retry/backoff — up to 5 attempts (30s/2m/8m/30m) — then dead-letters the row as failed with the reason in delivery_error. The feed row stays readable in-app whatever its egress fate.

Noise controls

• Cadence and digest windows — the dispatcher stamps every row with its resolved cadence and, for hourly/daily, the aligned digest_window_ends_at; a queued webhook-routed row's next_attempt_at defers to the window's close, so the N3 delivery sweeper honours cadence and can aggregate due rows per (route, window). The in-app feed row itself always lands realtime.
• Thread grouping — one updating row per change/workflow run: a new row in a (user, thread) supersedes the thread's older unarchived rows (superseded_at), which stay as delivery evidence and in the include_superseded/thread-history view but leave the default feed, the unread count and the WatchNotifications stream.
• Flood collapse — 5 failure rows within 5 minutes on one thread collapse to that thread's surviving row, its collapsed_count and summary (N failures on <path>) updating in place as the flood grows; a flood spanning several threads under one subscription collapses to a single row naming the subscription's scope as the blast radius (N failures across <scope>). Recomputed set-based from stored rows each sweep, so concurrent cluster sweeps converge.
• The live bell — WatchNotifications streams fresh rows (never archived or superseded ones) with resume cursors plus unread-count ticks, on the WatchEvents poll pace. The stream opens with an immediate count-only tick carrying the current unread count, so the bell knows the subscription is live before any row lands.

Delivery evidence and retention

QueryNotificationDeliveries (admin + packs:security-governance) pages the delivery-evidence records newest first over a keyset cursor: who was told what, when, on which channel, with what outcome — metadata only, never rendered secret content. Filters (conjunctive): username, channel_kind, delivery_status, subscription_id, mandate_id, and a from/to creation window.

Retention is tier-derived (a funnel lever, never a mintable limit):

Tier	Feed	Evidence	Tunable
Basic	30 days	90 days	No — fixed
Enterprise	90 days (default)	400 days (default)	Yes — GetNotificationRetention / UpdateNotificationRetention

The Enterprise tunables live in the stored global settings document's notifications subtree (feed_retention_days / evidence_retention_days, each 1–3650), written as a new settings version through UpdateNotificationRetention with evidence ≥ feed enforced at write (and defensively re-clamped at read, so a hand-edited document cannot invert them). Both retention RPCs are pack-gated like the evidence query; the reaper itself is not.

The retention reaper (serve/notifyretention.go, the delivery-sweeper idiom: every replica, set-based, cluster-convergent) sweeps hourly plus once at boot. Feed expiry archives a row (archived_at) — it leaves every feed, thread, unread and stream view but survives as delivery evidence; evidence expiry deletes the row outright. Deletion is fail-safe towards retention: a licence or settings resolution failure skips the sweep, and the reaper's hold seam lets legal-hold rework suspend both clocks without touching the mechanics (a guard seam, not an implementation). A sweep that removed anything records a notifications:retention:sweep event with the counts and the policy.

Bounds

Bound	Value	Breach
Subscriptions per user	200	invalid_argument ("subscription cap reached")
Routes per user	25	invalid_argument ("route cap reached")
Team subscriptions per group	200	invalid_argument ("subscription cap reached")
Team routes per group	25	invalid_argument ("route cap reached")
Name length	255 characters	invalid_argument field violation
Feed page	page_size ≤ 1000 (default 100)	clamped

Fields (Notification)

Field	Type	Meaning
event_id / event_type	string	The spine event dispatched for.
severity / state / object_kind	string	The engine's classification of the event.
matched_by	string	subscription, my_activity, approval, mandate or team (a group claim row or a member mirror).
subscription_id	string	The matched subscription; absent for the implicit defaults.
mandate_id	string	The mandated subscription that (also) covered this dispatch — the evidence attribution; absent when no mandate matched.
subject_type / subject_id / node_path	string	The subject's polymorphic reference and node path.
thread_key	string	One thread per change/workflow run (change:<id> / workflow_run:<id>), else kind × node path.
summary / deep_link	string	The allowlisted one-line rendering and the subject link.
channel_kind / delivery_status	string	in_app, webhook, slack or teams; queued → delivered | failed (dead-letter) | suppressed.
delivery_error	string	Why a delivery failed or was suppressed — a fixed reason string, never payload content.
cadence / digest_window_ends_at	string / timestamp	The resolved cadence and, for a digest cadence, when its window closes (a queued webhook row's next_attempt_at defers to it).
collapsed_count	int	How many rows this one speaks for (the flood blast radius; 1 normally).
superseded_at	timestamp	Set when a newer thread row replaced this one in the default feed view.
read_at / archived_at / delivered_at / created_at	timestamp	The feed and evidence stamps.

Mutations record api:notification_subscriptions:*, api:notification_routes:*, api:notifications:read|archive, api:notification_preferences:update, api:notification_team_subscriptions:create|update|destroy, api:notification_team_routes:create|update|destroy, api:notification_mandates:create|update|destroy and api:notification_retention:update audit events (names, groups and counts only — never summaries or documents). Storage is migrations 00046, 00047 (webhook delivery bookkeeping), 00048 (the N2 noise columns), 00051 (the N4 mandate_id attribution plus the evidence/retention indexes) and 00052 (the per-user notification_preferences opt-down rows), 00062 (the group-owned rows and the per-team claim index) and 00068 (the chat kinds widening the sweeper's due partial index); notification_channel_connectors serves the channel connectors and the slack/teams kinds. The parked v1 families (event_filters, event_subscribers, notification_jobs and their notification_filters/notification_subscribers aliases) are removed — a clean break with no importer: re-declare an interest as a subscription, and what a v1 job/channel pair delivered is now a route (in_app, or webhook via a channel connector).