## What & why S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service. Closes #124 ### How - Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files. - `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name. - `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`). - `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`. ### Dependency (CLAUDE.md §13/§14) Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**. ## Definition of Done - [x] Linked Gitea issue (#124). - [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`). - [x] Implementation makes the test pass. - [ ] CI green — pending Gitea Actions run. - [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023). - [x] Docs updated — demo-script S-16c entry. - [x] ADR added — ADR-0024. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top. - `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code. - Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #129
This commit was merged in pull request #129.
This commit is contained in:
Executable
+75
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env python3
|
||||
"""S-16c (#124): prove the golden-signal metrics pipeline works end to end.
|
||||
|
||||
Generate anonymous BFF traffic (GET /openbaar/register — no auth, no OpenZaak egress),
|
||||
then query Prometheus and assert (1) every .NET service's scrape target is UP, and (2)
|
||||
the http.server.request.duration histogram is actually being scraped — i.e. the services
|
||||
expose /metrics AND Prometheus collects it, which is exactly what the golden-signal
|
||||
dashboard reads.
|
||||
|
||||
Stdlib only (urllib/json) so it runs in a bare python:3-slim container in-network.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
BFF = os.environ["BFF"] # http://<bff-ip>:8080
|
||||
PROM = os.environ["PROMETHEUS"] # http://<prometheus-ip>:9090
|
||||
TIMEOUT = int(os.environ.get("METRICS_TIMEOUT", "90"))
|
||||
SERVICES = {"acl", "domain", "bff", "event-subscriber", "projection-api"}
|
||||
|
||||
|
||||
def _get(url):
|
||||
with urllib.request.urlopen(url, timeout=10) as r:
|
||||
return r.read()
|
||||
|
||||
|
||||
def generate_traffic():
|
||||
for _ in range(3):
|
||||
try:
|
||||
_get(f"{BFF}/openbaar/register")
|
||||
except urllib.error.HTTPError:
|
||||
pass # a non-2xx still records an http.server metric
|
||||
|
||||
|
||||
def query(promql):
|
||||
q = urllib.parse.quote(promql)
|
||||
try:
|
||||
data = json.loads(_get(f"{PROM}/api/v1/query?query={q}"))
|
||||
except Exception:
|
||||
return []
|
||||
return data.get("data", {}).get("result", [])
|
||||
|
||||
|
||||
def jobs_up():
|
||||
return {r["metric"].get("job") for r in query("up == 1")}
|
||||
|
||||
|
||||
def jobs_with_request_metric():
|
||||
return {r["metric"].get("job")
|
||||
for r in query("http_server_request_duration_seconds_count")}
|
||||
|
||||
|
||||
def main():
|
||||
deadline = time.time() + TIMEOUT
|
||||
while time.time() < deadline:
|
||||
generate_traffic()
|
||||
up = jobs_up()
|
||||
scraped = jobs_with_request_metric()
|
||||
if SERVICES.issubset(up) and SERVICES.issubset(scraped):
|
||||
print(f"OK — targets up: {sorted(up & SERVICES)}; "
|
||||
f"request metric scraped from: {sorted(scraped & SERVICES)}")
|
||||
return 0
|
||||
time.sleep(3)
|
||||
print(f"FAIL — up: {sorted(jobs_up() & SERVICES)}; "
|
||||
f"request metric from: {sorted(jobs_with_request_metric() & SERVICES)}; "
|
||||
f"expected all of {sorted(SERVICES)}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user