feat(obs): Prometheus metrics on /metrics + golden-signal Grafana dashboard (closes #124) (#129)
CI / build (push) Successful in 1m47s
CI / lint (push) Successful in 1m59s
CI / unit (push) Successful in 1m57s
CI / frontend (push) Successful in 3m58s
CI / mutation (push) Successful in 6m57s
CI / verify-stack (push) Successful in 8m29s

## What & why

S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service.

Closes #124

### How

- Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files.
- `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name.
- `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`).
- `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`.

### Dependency (CLAUDE.md §13/§14)

Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**.

## Definition of Done

- [x] Linked Gitea issue (#124).
- [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`).
- [x] Implementation makes the test pass.
- [ ] CI green — pending Gitea Actions run.
- [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023).
- [x] Docs updated — demo-script S-16c entry.
- [x] ADR added — ADR-0024.
- [x] Demo note in `docs/demo-script.md`.

## Notes for reviewers

- `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top.
- `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code.
- Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #129
This commit was merged in pull request #129.
This commit is contained in:
not
2026-07-24 08:31:41 +00:00
parent 6771fccf47
commit d5dfbdc0b2
22 changed files with 418 additions and 11 deletions
@@ -0,0 +1,53 @@
# ADR-0024: Expose OTel metrics with the (prerelease) Prometheus AspNetCore exporter
- **Status:** Accepted
- **Date:** 2026-07-24
- **Deciders:** Respellion engineering
- **Slice:** S-16c (#124), last of the S-16 (#17) split
## Context
ADR-0023 already fixed the shape of metrics collection: **Prometheus scrapes each
service's `/metrics`** (pull, no collector). S-16c implements it. That needs a package
that turns the OpenTelemetry `MeterProvider` into a Prometheus scrape endpoint inside
ASP.NET Core. The canonical one is `OpenTelemetry.Exporter.Prometheus.AspNetCore`
(`AddPrometheusExporter()` + `app.MapPrometheusScrapingEndpoint()`).
The catch: that exporter has **never had a stable release** — the whole OTel .NET
Prometheus exporter line is versioned `-beta` (we pin `1.17.0-beta.1`, matched to the
`1.17.0` core we already use). Adding it is a new dependency (CLAUDE.md §14), and taking
a prerelease package into all five services is the decision worth recording.
## Decision
**Add `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` to the five .NET
services and expose `/metrics` with it.**
- What it gives us: the OTel-native pull endpoint, so the meters we already register for
tracing-adjacent instrumentation surface as Prometheus text with zero extra plumbing.
- What we'd write to replace it: a hand-rolled `IMetricsListener`/`MeterListener` that
formats Prometheus exposition text — real work, and a reimplementation of a widely-used
library for no gain.
- Risk it adds: a prerelease API that can shift between betas. Contained: it is only
wired in `Program.cs` (two calls per service, excluded from mutation), the version is
pinned, and `verify-metrics` proves the endpoint + scrape actually work each CI run.
The alternative — pushing metrics over OTLP to a collector that re-exposes them — was
already rejected in ADR-0023 (no collector hop). Not revisited here.
## Consequences
**Positive**
- Golden-signal metrics on `/metrics` with the standard OTel names
(`http_server_request_duration_seconds`, `dotnet_*`), scraped straight by Prometheus.
- No collector, no bespoke exposition code.
**Negative / costs**
- A `-beta` package in production services. Mitigated by the pin + the `verify-metrics`
CI gate; upgrading tracks the OTel core version bumps.
## Coupling rules touched (CLAUDE.md §8)
None. Metrics are passive: Prometheus pulls; no service calls into the stack.