feat(infra): observability backplane — Tempo + Prometheus + Grafana (S-16a, closes #122) #125

2026-07-23T10:48:42Z

not commented

2026-07-23 10:48:42 +00:00

What & why

S-16a, the first of the S-16 split (#17 closed → #122/#123/#124, §13). Stands up a local, CI-friendly observability backplane so traces (S-16b) and metrics (S-16c) have somewhere to land, viewable in one Grafana.

Grafana Tempo — OTLP trace ingest (gRPC 4317 / HTTP 4318), local storage.
Prometheus — scrapes itself for now; service /metrics targets arrive in S-16c.
Grafana — Tempo + Prometheus datasources auto-provisioned with fixed uids (tempo, prometheus), exposed on :3000.

All three are small built images with config baked in (infra/observability/), on the existing cg network. No OTLP collector (Tempo ingests OTLP directly; Prometheus scrapes) and no config-volume seeding — the tools aren't verbatim CG peer modules, so a 3-line COPY Dockerfile is the simpler path that still reaches sibling containers on the CI runner (ADR-0023).

Verified, not assumed

make verify-observability (new CI verify-stack step, run early) asks Grafana to reach both datasources — Prometheus via its health method, Tempo via the datasource proxy (Tempo's plugin implements no health method) — so it proves the datasources are wired, not merely that containers booted. Validated locally against the three containers (no external egress): Grafana healthy, both datasources reachable.

Closes #122

Definition of Done

Failing test committed first (verify-observability fails with no backplane).
Implementation makes it pass; verified locally.
Conventional Commits referencing the issue (refs #122).
CI green — awaiting Gitea Actions (verify-stack now includes the observability step; docker compose config validates locally).
docker compose up reaches green health within 3 min — new containers are lightweight and off the health-gate list.
Docs — ADR-0023, demo-script, BACKLOG sync.
ADR added — docs/architecture/adr-0023-observability-stack.md.
Demo note in docs/demo-script.md.

Notes for reviewers

No app changes — this is pure infra; the five services are untouched (instrumentation is #123/#124).
Ports: Grafana 3000 (admin/admin, anonymous viewer on), Prometheus 9090; Tempo internal to cg.
CI: the three containers are added to the failure log-dump list; deliberately not added to WAIT_SVCS (the check polls Grafana itself, so no in-image healthcheck tool is needed). Trades ~3 small image builds per run.
Next: #123 wires OTLP export + AddAspNetCoreInstrumentation/AddHttpClientInstrumentation into the five hosts so a request becomes one connected trace in Tempo.

## What & why S-16a, the first of the **S-16 split** (#17 closed → #122/#123/#124, §13). Stands up a local, CI-friendly observability backplane so traces (S-16b) and metrics (S-16c) have somewhere to land, viewable in one Grafana. - **Grafana Tempo** — OTLP trace ingest (gRPC 4317 / HTTP 4318), local storage. - **Prometheus** — scrapes itself for now; service `/metrics` targets arrive in S-16c. - **Grafana** — Tempo + Prometheus datasources auto-provisioned with fixed uids (`tempo`, `prometheus`), exposed on :3000. All three are small **built images** with config baked in (`infra/observability/`), on the existing `cg` network. **No OTLP collector** (Tempo ingests OTLP directly; Prometheus scrapes) and **no config-volume seeding** — the tools aren't verbatim CG peer modules, so a 3-line `COPY` Dockerfile is the simpler path that still reaches sibling containers on the CI runner (**ADR-0023**). ### Verified, not assumed `make verify-observability` (new CI `verify-stack` step, run early) asks Grafana to reach both datasources — Prometheus via its health method, Tempo via the datasource proxy (Tempo's plugin implements no health method) — so it proves the datasources are wired, not merely that containers booted. Validated locally against the three containers (no external egress): Grafana healthy, both datasources reachable. Closes #122 ## Definition of Done - [x] Failing test committed first (`verify-observability` fails with no backplane). - [x] Implementation makes it pass; verified locally. - [x] Conventional Commits referencing the issue (`refs #122`). - [ ] CI green — awaiting Gitea Actions (verify-stack now includes the observability step; `docker compose config` validates locally). - [ ] `docker compose up` reaches green health within 3 min — new containers are lightweight and off the health-gate list. - [x] Docs — ADR-0023, demo-script, BACKLOG sync. - [x] ADR added — `docs/architecture/adr-0023-observability-stack.md`. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - **No app changes** — this is pure infra; the five services are untouched (instrumentation is #123/#124). - **Ports:** Grafana 3000 (admin/admin, anonymous viewer on), Prometheus 9090; Tempo internal to `cg`. - **CI:** the three containers are added to the failure log-dump list; deliberately **not** added to `WAIT_SVCS` (the check polls Grafana itself, so no in-image healthcheck tool is needed). Trades ~3 small image builds per run. - **Next:** #123 wires OTLP export + `AddAspNetCoreInstrumentation`/`AddHttpClientInstrumentation` into the five hosts so a request becomes one connected trace in Tempo.

not added this to the Iteration 3 — Beheer & Observability milestone 2026-07-23 10:48:42 +00:00

not added 3 commits 2026-07-23 10:48:43 +00:00

test(infra): verify-observability asserts Grafana + Tempo/Prometheus datasources are live (refs #122 ) f12e9ce746

Polls Grafana's datasource-health endpoints so it proves Grafana can actually
reach Tempo and Prometheus over the cg network, not merely that the containers
started. Fails until the backplane is wired (next commit).

refs #122

feat(infra): Tempo + Prometheus + Grafana observability backplane in compose (refs #122 ) f29a8598fd

Three built images on the cg network with config baked in: Tempo (OTLP ingest,
ports 4317/4318), Prometheus, and Grafana with both datasources auto-provisioned.
Config lives in infra/observability/. The verify-observability step runs early in
CI verify-stack; the three containers are added to the failure log-dump list.
Tempo's Grafana plugin has no datasource /health method, so the check proves
Tempo reachability through Grafana's datasource proxy instead.

refs #122

docs: ADR-0023 + demo note + backlog sync for the observability backplane (refs #122 )

CI / lint (pull_request) Successful in 1m23s

Details

CI / build (pull_request) Successful in 1m9s

Details

CI / unit (pull_request) Successful in 1m17s

Details

CI / frontend (pull_request) Successful in 2m43s

Details

CI / mutation (pull_request) Successful in 5m56s

Details

CI / verify-stack (pull_request) Failing after 8m6s

Details

43c6c8e0d2

ADR-0023 records the Grafana-native stack (Tempo, no collector, config baked into
images); demo-script and BACKLOG describe S-16a and the S-16 split.

refs #122

not added the type:slice area:infra labels 2026-07-23 10:48:55 +00:00

not added 1 commit 2026-07-23 11:55:07 +00:00

fix(infra): run the observability check inside the compose network (refs #122 )

CI / lint (pull_request) Successful in 1m25s

Details

CI / build (pull_request) Successful in 1m9s

Details

CI / unit (pull_request) Successful in 1m19s

Details

CI / frontend (pull_request) Successful in 2m47s

Details

CI / mutation (pull_request) Successful in 6m9s

Details

CI / verify-stack (pull_request) Successful in 9m20s

Details

56a5fe014c

The stack runs as sibling containers on the CI runner's host daemon, so its
published ports aren't on the runner job's localhost — verify-observability hit
localhost:3000 and failed. Reach Grafana by container IP through a throwaway curl
container on the stack network, mirroring run-bff-check.sh et al. Parses with grep
(the curl image has no python).

refs #122

not merged commit 4274fd30d1 into main

2026-07-23 12:26:23 +00:00

not referenced this issue from a commit

2026-07-23 12:26:23 +00:00

feat(infra): observability backplane — Tempo + Prometheus + Grafana (S-16a, closes #122) (#125)

not referenced this pull request

2026-07-23 12:38:16 +00:00

feat(obs): distributed traces across the .NET services (S-16b, closes #123) #126

not referenced this issue from a commit

2026-07-23 14:38:28 +00:00

feat(obs): distributed traces across the .NET services (S-16b, closes #123) (#126)

Sign in to join this conversation.