feat(obs): golden-signal metrics on /metrics + Prometheus scrape + Grafana dashboard (S-16c, refs #124)
CI / lint (pull_request) Successful in 5m7s
CI / unit (pull_request) Successful in 2m2s
CI / frontend (pull_request) Successful in 3m58s
CI / mutation (pull_request) Successful in 6m30s
CI / verify-stack (pull_request) Successful in 9m34s
CI / build (pull_request) Successful in 4m57s
CI / lint (pull_request) Successful in 5m7s
CI / unit (pull_request) Successful in 2m2s
CI / frontend (pull_request) Successful in 3m58s
CI / mutation (pull_request) Successful in 6m30s
CI / verify-stack (pull_request) Successful in 9m34s
CI / build (pull_request) Successful in 4m57s
Wire OTel metrics into the four remaining .NET services (acl, domain, event-subscriber, projection-api) exactly as the BFF: ASP.NET Core + HttpClient instrumentation + the built-in System.Runtime meter, exposed at /metrics via the Prometheus AspNetCore exporter (ADR-0024). Prometheus scrapes one job per service; Grafana ships a pre-built 'Request path — golden signals' dashboard (traffic/errors/latency/saturation). A verify-metrics CI step proves the endpoints are scraped end to end.
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
# ADR-0024: Expose OTel metrics with the (prerelease) Prometheus AspNetCore exporter
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-07-24
|
||||
- **Deciders:** Respellion engineering
|
||||
- **Slice:** S-16c (#124), last of the S-16 (#17) split
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0023 already fixed the shape of metrics collection: **Prometheus scrapes each
|
||||
service's `/metrics`** (pull, no collector). S-16c implements it. That needs a package
|
||||
that turns the OpenTelemetry `MeterProvider` into a Prometheus scrape endpoint inside
|
||||
ASP.NET Core. The canonical one is `OpenTelemetry.Exporter.Prometheus.AspNetCore`
|
||||
(`AddPrometheusExporter()` + `app.MapPrometheusScrapingEndpoint()`).
|
||||
|
||||
The catch: that exporter has **never had a stable release** — the whole OTel .NET
|
||||
Prometheus exporter line is versioned `-beta` (we pin `1.17.0-beta.1`, matched to the
|
||||
`1.17.0` core we already use). Adding it is a new dependency (CLAUDE.md §14), and taking
|
||||
a prerelease package into all five services is the decision worth recording.
|
||||
|
||||
## Decision
|
||||
|
||||
**Add `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` to the five .NET
|
||||
services and expose `/metrics` with it.**
|
||||
|
||||
- What it gives us: the OTel-native pull endpoint, so the meters we already register for
|
||||
tracing-adjacent instrumentation surface as Prometheus text with zero extra plumbing.
|
||||
- What we'd write to replace it: a hand-rolled `IMetricsListener`/`MeterListener` that
|
||||
formats Prometheus exposition text — real work, and a reimplementation of a widely-used
|
||||
library for no gain.
|
||||
- Risk it adds: a prerelease API that can shift between betas. Contained: it is only
|
||||
wired in `Program.cs` (two calls per service, excluded from mutation), the version is
|
||||
pinned, and `verify-metrics` proves the endpoint + scrape actually work each CI run.
|
||||
|
||||
The alternative — pushing metrics over OTLP to a collector that re-exposes them — was
|
||||
already rejected in ADR-0023 (no collector hop). Not revisited here.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- Golden-signal metrics on `/metrics` with the standard OTel names
|
||||
(`http_server_request_duration_seconds`, `dotnet_*`), scraped straight by Prometheus.
|
||||
- No collector, no bespoke exposition code.
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- A `-beta` package in production services. Mitigated by the pin + the `verify-metrics`
|
||||
CI gate; upgrading tracks the OTel core version bumps.
|
||||
|
||||
## Coupling rules touched (CLAUDE.md §8)
|
||||
|
||||
None. Metrics are passive: Prometheus pulls; no service calls into the stack.
|
||||
@@ -5,6 +5,34 @@ copy-pasteable walkthrough against a local `make up` stack.
|
||||
|
||||
---
|
||||
|
||||
## S-16c — Prometheus metrics + golden-signal Grafana dashboard (#124, ADR-0023)
|
||||
|
||||
**Outcome:** the five .NET services now expose OpenTelemetry metrics in Prometheus format at `/metrics`
|
||||
— ASP.NET Core + `HttpClient` instrumentation plus the built-in `System.Runtime` meter. Prometheus
|
||||
scrapes each service (one job per service), and a **pre-built Grafana dashboard** — *Request path —
|
||||
golden signals* — plots the four golden signals: **traffic** (req/s), **errors** (5xx/s), **latency**
|
||||
(p95 request duration), and **saturation** (CPU cores in use), split by service. It populates under load.
|
||||
|
||||
```bash
|
||||
# 1. Automated (a CI verify-stack step): generate BFF traffic and assert Prometheus scraped the
|
||||
# golden-signal metric from every service.
|
||||
make verify-metrics # → OK — targets up: [...]; request metric scraped from: [...]
|
||||
|
||||
# 2. By hand: drive the stack, generate some load, then open the dashboard.
|
||||
make up
|
||||
for i in $(seq 1 50); do curl -s localhost:8080/openbaar/register >/dev/null; done # BFF → projection-api
|
||||
open http://localhost:3000 # Grafana → Dashboards → "Request path — golden signals"
|
||||
open http://localhost:9090/targets # Prometheus → every service target UP
|
||||
```
|
||||
|
||||
**The path:** each host adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation +
|
||||
AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`; Prometheus scrapes
|
||||
`<service>:8080/metrics` (config in `infra/observability/prometheus/prometheus.yml`); Grafana ships the
|
||||
dashboard via provisioning against the fixed `prometheus` datasource uid. No metrics are pushed over
|
||||
OTLP — Prometheus pulls, so there is no collector hop (ADR-0023).
|
||||
|
||||
---
|
||||
|
||||
## S-16b — distributed traces across the .NET services (#123, ADR-0023)
|
||||
|
||||
**Outcome:** the five .NET services (BFF, Domain, ACL, projection-api, event-subscriber) now emit
|
||||
|
||||
Reference in New Issue
Block a user