not
4274fd30d1
## What & why S-16a, the first of the **S-16 split** (#17 closed → #122/#123/#124, §13). Stands up a local, CI-friendly observability backplane so traces (S-16b) and metrics (S-16c) have somewhere to land, viewable in one Grafana. - **Grafana Tempo** — OTLP trace ingest (gRPC 4317 / HTTP 4318), local storage. - **Prometheus** — scrapes itself for now; service `/metrics` targets arrive in S-16c. - **Grafana** — Tempo + Prometheus datasources auto-provisioned with fixed uids (`tempo`, `prometheus`), exposed on :3000. All three are small **built images** with config baked in (`infra/observability/`), on the existing `cg` network. **No OTLP collector** (Tempo ingests OTLP directly; Prometheus scrapes) and **no config-volume seeding** — the tools aren't verbatim CG peer modules, so a 3-line `COPY` Dockerfile is the simpler path that still reaches sibling containers on the CI runner (**ADR-0023**). ### Verified, not assumed `make verify-observability` (new CI `verify-stack` step, run early) asks Grafana to reach both datasources — Prometheus via its health method, Tempo via the datasource proxy (Tempo's plugin implements no health method) — so it proves the datasources are wired, not merely that containers booted. Validated locally against the three containers (no external egress): Grafana healthy, both datasources reachable. Closes #122 ## Definition of Done - [x] Failing test committed first (`verify-observability` fails with no backplane). - [x] Implementation makes it pass; verified locally. - [x] Conventional Commits referencing the issue (`refs #122`). - [ ] CI green — awaiting Gitea Actions (verify-stack now includes the observability step; `docker compose config` validates locally). - [ ] `docker compose up` reaches green health within 3 min — new containers are lightweight and off the health-gate list. - [x] Docs — ADR-0023, demo-script, BACKLOG sync. - [x] ADR added — `docs/architecture/adr-0023-observability-stack.md`. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - **No app changes** — this is pure infra; the five services are untouched (instrumentation is #123/#124). - **Ports:** Grafana 3000 (admin/admin, anonymous viewer on), Prometheus 9090; Tempo internal to `cg`. - **CI:** the three containers are added to the failure log-dump list; deliberately **not** added to `WAIT_SVCS` (the check polls Grafana itself, so no in-image healthcheck tool is needed). Trades ~3 small image builds per run. - **Next:** #123 wires OTLP export + `AddAspNetCoreInstrumentation`/`AddHttpClientInstrumentation` into the five hosts so a request becomes one connected trace in Tempo. Reviewed-on: #125
3.3 KiB
ADR-0023: Grafana-native observability stack (Tempo + Prometheus + Grafana)
- Status: Accepted
- Date: 2026-07-23
- Deciders: Respellion engineering
- Slice: S-16a (#122), first of the S-16 (#17) split
Context
The PRD calls for "OpenTelemetry traces, Prometheus metrics; a local Grafana with pre-built dashboards" (§80). S-16 was split (CLAUDE.md §13) into a backplane slice (this one), distributed tracing (#123), and metrics + dashboards (#124). The backplane must stand up first: a local, CI-friendly place for traces and metrics to land, viewable in one UI, reaching green health within the 3-minute compose budget.
Two shape decisions are non-obvious enough to record.
Decision
Run a Grafana-native stack — Grafana Tempo (traces) + Prometheus (metrics) + Grafana (UI) — with the services exporting OTLP straight to Tempo (no collector), and ship the config baked into small built images.
Trace backend: Tempo (not Jaeger)
Tempo keeps everything under one Grafana pane alongside metrics (and later logs), which is exactly the "local Grafana with dashboards" the PRD asks for. Jaeger would add a second UI and a second mental model for no benefit at this scale.
No OTLP collector
Tempo ingests OTLP directly (gRPC 4317 / HTTP 4318) and Prometheus scrapes each
service's /metrics, so a collector would be a hop that processes nothing. Skipped.
If we later need fan-out, tail sampling, or log processing, a collector is an
additive change — the services already speak OTLP.
Config baked into built images, not config volumes
The upstream Common Ground modules (OpenZaak, NRC, Keycloak, Flowable) run as
verbatim images and get their config streamed into external named volumes by
infra/seed-config.sh, because bind mounts don't reach sibling containers on the
CI runner (see docs/runbooks/gitea-actions-gotchas.md). The observability tools
are not peer modules we must run verbatim, so we take the simpler path: a
three-line Dockerfile per tool that COPYs its config in. This reaches sibling
containers everywhere (docker, podman, CI) with no seed step, no CFG_VOLS entry,
and no Makefile sprawl.
Verified, not assumed
infra/run-observability-check.sh (the verify-observability step, run early in CI
verify-stack) asks Grafana to reach both datasources — Prometheus via its health
method, Tempo via the datasource proxy (Tempo's Grafana plugin implements no health
method) — so the check proves the datasources are actually wired, not merely that
containers started. The containers are not in WAIT_SVCS; the check polls Grafana
itself, so no in-image healthcheck tool is required.
Consequences
Positive
- One UI for traces + metrics + (future) logs. Config is versioned in
infra/observability/and self-contained in the images. - Backplane is independent of app instrumentation — #123 and #124 build on it.
Negative / costs
- Three more images built each CI run (kept small; not on the health-gate list).
- Storage is ephemeral container fs — a demo backplane, not a retention target. Object storage for Tempo / remote-write for Prometheus is a later concern.
Coupling rules touched (CLAUDE.md §8)
None. The stack is passive infrastructure: services push OTLP and expose
/metrics; nothing in the stack calls into a service or a peer module.