not
d5dfbdc0b2
## What & why S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service. Closes #124 ### How - Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files. - `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name. - `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`). - `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`. ### Dependency (CLAUDE.md §13/§14) Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**. ## Definition of Done - [x] Linked Gitea issue (#124). - [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`). - [x] Implementation makes the test pass. - [ ] CI green — pending Gitea Actions run. - [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023). - [x] Docs updated — demo-script S-16c entry. - [x] ADR added — ADR-0024. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top. - `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code. - Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #129
29 lines
1.4 KiB
Bash
Executable File
29 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# S-16c (#124): assert the golden-signal metrics pipeline works — the .NET services expose
|
|
# /metrics and Prometheus scrapes them — against an ALREADY-RUNNING full stack. Runs the
|
|
# driver in a python:3-slim container on the stack network (services reached by container IP;
|
|
# the runner can't reach published ports — gitea-actions-gotchas.md §5/§6). Does NOT manage
|
|
# the stack lifecycle.
|
|
set -euo pipefail
|
|
|
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
|
|
ip() { docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$1"; }
|
|
|
|
bff="$(docker ps -q --filter 'name=[-_]bff[-_]' | head -1)"
|
|
prom="$(docker ps -q --filter 'name=[-_]prometheus[-_]' | head -1)"
|
|
[ -n "$bff" ] && [ -n "$prom" ] || { echo "ERROR: bff and/or prometheus not running — bring the stack up first" >&2; exit 1; }
|
|
net="$(docker inspect -f '{{range $k,$_ := .NetworkSettings.Networks}}{{$k}}{{"\n"}}{{end}}' "$bff" | head -1)"
|
|
bff_ip="$(ip "$bff")"; prom_ip="$(ip "$prom")"
|
|
echo ">> network=$net bff=$bff_ip prometheus=$prom_ip"
|
|
|
|
cid="$(docker create --network "$net" \
|
|
-e "BFF=http://$bff_ip:8080" -e "PROMETHEUS=http://$prom_ip:9090" \
|
|
-e "METRICS_TIMEOUT=${METRICS_TIMEOUT:-90}" \
|
|
python:3-slim python /metrics-check.py)"
|
|
docker cp "$here/metrics-check.py" "$cid:/metrics-check.py" >/dev/null
|
|
rc=0; docker start -a "$cid" || rc=$?
|
|
docker rm -f "$cid" >/dev/null
|
|
exit $rc
|