diff --git a/BACKLOG.md b/BACKLOG.md index 5857963..b87a48b 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -253,10 +253,16 @@ Split (issue #11 closed) into two independently-demoable slices per §13 — the **Outcome:** Beheer portal lets an admin view ZTC catalogi (read-only first), and manage the ACL's default-fill configuration via a CRUD UI. MFA on the medewerker realm enforced. -### S-16 · OpenTelemetry traces + Grafana dashboard +### S-16 · OpenTelemetry traces + Grafana dashboard *(split — #17 closed)* **Outcome:** Traces span portal → BFF → Domain → ACL → OpenZaak and portal → BFF → Domain → Flowable. Grafana dashboards pre-built for golden signals. +Split into independently deployable sub-slices (CLAUDE.md §13): + +- **S-16a** (#122) · Observability backplane — Grafana Tempo + Prometheus + Grafana in compose, datasources auto-provisioned (ADR-0023). No collector; config baked into built images. +- **S-16b** (#123) · Distributed traces across the five .NET services (OTLP → Tempo; nginx `traceparent` passthrough). Depends on S-16a. +- **S-16c** (#124) · Prometheus metrics + golden-signal Grafana dashboards. Depends on S-16a. + ### S-17 · Quartz.NET scheduler — herregistratie reminder sweep ✅ **Outcome:** Daily Quartz.NET cron job finds inscriptions within 90 days of their herregistratie deadline and reminds each (flag on the aggregate + log). No outbound notification and no domain event in v1 — the reminder is the persisted flag, surfaced on the read model (ADR-0022, #120). Quartz fires time-triggered sweeps; the existing pumps stay as queue-drainers. diff --git a/docs/architecture/adr-0023-observability-stack.md b/docs/architecture/adr-0023-observability-stack.md new file mode 100644 index 0000000..7adb034 --- /dev/null +++ b/docs/architecture/adr-0023-observability-stack.md @@ -0,0 +1,74 @@ +# ADR-0023: Grafana-native observability stack (Tempo + Prometheus + Grafana) + +- **Status:** Accepted +- **Date:** 2026-07-23 +- **Deciders:** Respellion engineering +- **Slice:** S-16a (#122), first of the S-16 (#17) split + +## Context + +The PRD calls for "OpenTelemetry traces, Prometheus metrics; a local Grafana with +pre-built dashboards" (§80). S-16 was split (CLAUDE.md §13) into a backplane slice +(this one), distributed tracing (#123), and metrics + dashboards (#124). The +backplane must stand up first: a local, CI-friendly place for traces and metrics to +land, viewable in one UI, reaching green health within the 3-minute compose budget. + +Two shape decisions are non-obvious enough to record. + +## Decision + +**Run a Grafana-native stack — Grafana Tempo (traces) + Prometheus (metrics) + +Grafana (UI) — with the services exporting OTLP straight to Tempo (no collector), +and ship the config baked into small built images.** + +### Trace backend: Tempo (not Jaeger) + +Tempo keeps everything under one Grafana pane alongside metrics (and later logs), +which is exactly the "local Grafana with dashboards" the PRD asks for. Jaeger would +add a second UI and a second mental model for no benefit at this scale. + +### No OTLP collector + +Tempo ingests OTLP directly (gRPC 4317 / HTTP 4318) and Prometheus scrapes each +service's `/metrics`, so a collector would be a hop that processes nothing. Skipped. +If we later need fan-out, tail sampling, or log processing, a collector is an +additive change — the services already speak OTLP. + +### Config baked into built images, not config volumes + +The upstream Common Ground modules (OpenZaak, NRC, Keycloak, Flowable) run as +**verbatim** images and get their config streamed into external named volumes by +`infra/seed-config.sh`, because bind mounts don't reach sibling containers on the +CI runner (see `docs/runbooks/gitea-actions-gotchas.md`). The observability tools +are **not** peer modules we must run verbatim, so we take the simpler path: a +three-line `Dockerfile` per tool that `COPY`s its config in. This reaches sibling +containers everywhere (docker, podman, CI) with no seed step, no `CFG_VOLS` entry, +and no Makefile sprawl. + +### Verified, not assumed + +`infra/run-observability-check.sh` (the `verify-observability` step, run early in CI +`verify-stack`) asks Grafana to reach both datasources — Prometheus via its health +method, Tempo via the datasource proxy (Tempo's Grafana plugin implements no health +method) — so the check proves the datasources are actually wired, not merely that +containers started. The containers are not in `WAIT_SVCS`; the check polls Grafana +itself, so no in-image healthcheck tool is required. + +## Consequences + +**Positive** + +- One UI for traces + metrics + (future) logs. Config is versioned in + `infra/observability/` and self-contained in the images. +- Backplane is independent of app instrumentation — #123 and #124 build on it. + +**Negative / costs** + +- Three more images built each CI run (kept small; not on the health-gate list). +- Storage is ephemeral container fs — a demo backplane, not a retention target. + Object storage for Tempo / remote-write for Prometheus is a later concern. + +## Coupling rules touched (CLAUDE.md §8) + +None. The stack is passive infrastructure: services *push* OTLP and *expose* +`/metrics`; nothing in the stack calls into a service or a peer module. diff --git a/docs/demo-script.md b/docs/demo-script.md index eeebfbb..c4b7492 100644 --- a/docs/demo-script.md +++ b/docs/demo-script.md @@ -5,6 +5,31 @@ copy-pasteable walkthrough against a local `make up` stack. --- +## S-16a — observability backplane: Tempo + Prometheus + Grafana (#122, ADR-0023) + +**Outcome:** the compose stack now includes a Grafana-native observability backplane — **Tempo** (OTLP +trace ingest on 4317/4318), **Prometheus**, and **Grafana** with both datasources auto-provisioned. +Nothing is instrumented yet (traces land in S-16b, metrics + dashboards in S-16c); this slice stands the +backplane up and proves Grafana can reach both datasources. Config is baked into small built images +(`infra/observability/`) — no collector, no config-volume seeding. + +```bash +# 1. Bring the stack up, then assert the backplane is live (Grafana healthy + Tempo/Prometheus +# datasources reachable through Grafana). This is a CI verify-stack step. +make up +make verify-observability # → ✓ Grafana healthy ✓ Prometheus reachable ✓ Tempo reachable + +# 2. Or just the backplane, no full stack needed (no external egress): +docker compose -f infra/docker-compose.yml up -d --build tempo prometheus grafana +open http://localhost:3000 # Grafana (admin/admin) → Connections → Data sources: Prometheus + Tempo +open http://localhost:9090 # Prometheus +``` + +**The path:** services will export OTLP → **Tempo:4317** and expose `/metrics` ← **Prometheus** scrapes; +**Grafana** (:3000) reads both via provisioned datasources with fixed uids `tempo` / `prometheus`. + +--- + ## S-17 — herregistratie reminder sweep on a Quartz cron (#18, ADR-0022) **Outcome:** an inscription (INGESCHREVEN) now carries the moment it was entered in the register, from