feat(obs): Prometheus metrics on /metrics + golden-signal Grafana dashboard (closes #124) (#129)

## What & why

S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service.

Closes #124

### How

- Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files.
- `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name.
- `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`).
- `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`.

### Dependency (CLAUDE.md §13/§14)

Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**.

## Definition of Done

- [x] Linked Gitea issue (#124).
- [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`).
- [x] Implementation makes the test pass.
- [ ] CI green — pending Gitea Actions run.
- [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023).
- [x] Docs updated — demo-script S-16c entry.
- [x] ADR added — ADR-0024.
- [x] Demo note in `docs/demo-script.md`.

## Notes for reviewers

- `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top.
- `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code.
- Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #129

services/acl/Acl.Api/Acl.Api.csproj

@@ -7,6 +7,7 @@
 
   <ItemGroup>
     <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.17.0" />
+    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.17.0-beta.1" />
     <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.17.0" />

services/acl/Acl.Api/Program.cs

@@ -1,5 +1,6 @@
 using Acl.Application;
 using Acl.Infrastructure;
+using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
 using OpenTelemetry.Trace;
 
@@ -14,7 +15,16 @@ builder.Services.AddOpenTelemetry()
     .WithTracing(tracing => tracing
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
         .AddHttpClientInstrumentation()
-        .AddOtlpExporter());
+        .AddOtlpExporter())
+    // OpenTelemetry metrics (S-16c, ADR-0023): golden signals for the request path —
+    // http.server.request.duration (traffic/errors/latency) + http.client.* for downstream hops, plus
+    // the built-in System.Runtime meter for saturation (GC, CPU, thread pool). Prometheus scrapes these
+    // from /metrics (mapped below); metrics aren't pushed over OTLP, so no collector hop (ADR-0023).
+    .WithMetrics(metrics => metrics
+        .AddAspNetCoreInstrumentation()
+        .AddHttpClientInstrumentation()
+        .AddMeter("System.Runtime")
+        .AddPrometheusExporter());
 
 builder.Services.AddSingleton<IClock, SystemClock>();
 builder.Services.AddSingleton(sp => sp.GetRequiredService<IConfiguration>()
@@ -32,6 +42,9 @@ var app = builder.Build();
 
 app.MapGet("/health", () => "Healthy");
 
+// Prometheus scrape endpoint (S-16c): exposes the OTel metrics above in Prometheus text format.
+app.MapPrometheusScrapingEndpoint();
+
 // The ACL's single operation, exposed as a service endpoint.
 app.MapPost("/zaken", async (OpenZaakRequest body, AclService acl, CancellationToken ct) =>
 {

services/bff/Bff.Api/Bff.Api.csproj

@@ -11,6 +11,7 @@
     <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.8" />
     <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="10.0.8" />
     <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.17.0" />
+    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.17.0-beta.1" />
     <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.17.0" />

services/bff/Bff.Api/Program.cs

@@ -3,6 +3,7 @@ using System.Text.Json;
 using System.Text.Json.Serialization;
 using Bff.Api;
 using Microsoft.AspNetCore.Authentication.JwtBearer;
+using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
 using OpenTelemetry.Trace;
 
@@ -18,7 +19,16 @@ builder.Services.AddOpenTelemetry()
     .WithTracing(tracing => tracing
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
         .AddHttpClientInstrumentation()
-        .AddOtlpExporter());
+        .AddOtlpExporter())
+    // OpenTelemetry metrics (S-16c, ADR-0023): the golden signals for the request path —
+    // http.server.request.duration (traffic/errors/latency) + http.client.* for the downstream hops,
+    // plus the built-in System.Runtime meter for saturation (GC, CPU, thread pool). Prometheus scrapes
+    // these from /metrics (mapped below); no OTLP push for metrics, so no collector hop (ADR-0023).
+    .WithMetrics(metrics => metrics
+        .AddAspNetCoreInstrumentation()
+        .AddHttpClientInstrumentation()
+        .AddMeter("System.Runtime")
+        .AddPrometheusExporter());
 
 var keycloakAuthority = builder.Configuration["Keycloak:Authority"]
     ?? throw new InvalidOperationException("Missing configuration 'Keycloak:Authority'");
@@ -82,6 +92,9 @@ app.UseAuthentication();
 app.UseAuthorization();
 
 app.MapHealthChecks("/health");
+
+// Prometheus scrape endpoint (S-16c): exposes the OTel metrics above in Prometheus text format.
+app.MapPrometheusScrapingEndpoint();
 app.MapOpenApi();
 
 // Self-service submit: requires a valid digid token; the bsn comes from the token, not the body,

services/bff/Bff.Tests/MetricsEndpointTests.cs

@@ -0,0 +1,28 @@
+using System.Net;
+using Microsoft.AspNetCore.Mvc.Testing;
+
+namespace Bff.Tests;
+
+/// <summary>
+/// S-16c (#124): the service exposes OTel HTTP-server metrics in Prometheus text format at /metrics,
+/// so Prometheus can scrape the golden signals (traffic, errors, latency) for the request path.
+/// </summary>
+public class MetricsEndpointTests(WebApplicationFactory<Program> factory)
+    : IClassFixture<WebApplicationFactory<Program>>
+{
+    [Fact]
+    public async Task Metrics_endpoint_exposes_http_server_request_duration_after_traffic()
+    {
+        var client = factory.CreateClient();
+
+        // One request produces an http.server.request.duration measurement...
+        await client.GetAsync("/health");
+
+        // ...which the /metrics scrape endpoint then exposes in Prometheus text format.
+        var response = await client.GetAsync("/metrics");
+
+        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
+        var body = await response.Content.ReadAsStringAsync();
+        Assert.Contains("http_server_request_duration", body);
+    }
+}

services/domain/Big.Api/Big.Api.csproj

@@ -7,6 +7,7 @@
 
   <ItemGroup>
     <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.17.0" />
+    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.17.0-beta.1" />
     <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.17.0" />

services/domain/Big.Api/Program.cs

@@ -1,6 +1,7 @@
 using Big.Application;
 using Big.Domain;
 using Big.Infrastructure;
+using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
 using OpenTelemetry.Trace;
 using Quartz;
@@ -18,7 +19,16 @@ builder.Services.AddOpenTelemetry()
     .WithTracing(tracing => tracing
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
         .AddHttpClientInstrumentation()
-        .AddOtlpExporter());
+        .AddOtlpExporter())
+    // OpenTelemetry metrics (S-16c, ADR-0023): golden signals for the request path —
+    // http.server.request.duration (traffic/errors/latency) + http.client.* for downstream hops, plus
+    // the built-in System.Runtime meter for saturation (GC, CPU, thread pool). Prometheus scrapes these
+    // from /metrics (mapped below); metrics aren't pushed over OTLP, so no collector hop (ADR-0023).
+    .WithMetrics(metrics => metrics
+        .AddAspNetCoreInstrumentation()
+        .AddHttpClientInstrumentation()
+        .AddMeter("System.Runtime")
+        .AddPrometheusExporter());
 
 // Options bound from configuration (compose sets Flowable__* and Acl__* env vars).
 builder.Services.AddSingleton(sp => sp.GetRequiredService<IConfiguration>()
@@ -84,6 +94,9 @@ var app = builder.Build();
 
 app.MapGet("/health", () => "Healthy");
 
+// Prometheus scrape endpoint (S-16c): exposes the OTel metrics above in Prometheus text format.
+app.MapPrometheusScrapingEndpoint();
+
 // Submit a registration. The aggregate is created (INGEDIEND) and the registratie process started;
 // the zaak is opened later, off the request path, by the worker — so this returns 202 Accepted with
 // a location to read the registration's progress (ADR-0009, eventual consistency).

services/event-subscriber/EventSubscriber.Api/EventSubscriber.Api.csproj

@@ -7,6 +7,7 @@
 
   <ItemGroup>
     <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.17.0" />
+    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.17.0-beta.1" />
     <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.17.0" />

services/event-subscriber/EventSubscriber.Api/Program.cs

@@ -1,5 +1,6 @@
 using System.Text.Json;
 using EventSubscriber.Application;
+using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
 using OpenTelemetry.Trace;
 using Projection.ReadModel;
@@ -15,7 +16,16 @@ builder.Services.AddOpenTelemetry()
     .WithTracing(tracing => tracing
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
         .AddHttpClientInstrumentation()
-        .AddOtlpExporter());
+        .AddOtlpExporter())
+    // OpenTelemetry metrics (S-16c, ADR-0023): golden signals for the request path —
+    // http.server.request.duration (traffic/errors/latency) + http.client.* for downstream hops, plus
+    // the built-in System.Runtime meter for saturation (GC, CPU, thread pool). Prometheus scrapes these
+    // from /metrics (mapped below); metrics aren't pushed over OTLP, so no collector hop (ADR-0023).
+    .WithMetrics(metrics => metrics
+        .AddAspNetCoreInstrumentation()
+        .AddHttpClientInstrumentation()
+        .AddMeter("System.Runtime")
+        .AddPrometheusExporter());
 
 var connectionString = builder.Configuration.GetConnectionString("Projection")
     ?? throw new InvalidOperationException("Missing connection string 'ConnectionStrings:Projection'");
@@ -41,6 +51,9 @@ await app.Services.MigrateProjectionAsync();
 
 app.MapGet("/health", () => "Healthy");
 
+// Prometheus scrape endpoint (S-16c): exposes the OTel metrics above in Prometheus text format.
+app.MapPrometheusScrapingEndpoint();
+
 // The NRC abonnement callback. Open Notificaties POSTs a notification here; we project it.
 // Auth-on-callback is mandatory: the auth check runs *before* the body is read, so NRC's
 // registration probe (a POST without the configured Authorization, and without a valid

services/projection-api/ProjectionApi.Api/Program.cs

@@ -1,4 +1,5 @@
 using Microsoft.EntityFrameworkCore;
+using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
 using OpenTelemetry.Trace;
 using Projection.ReadModel;
@@ -14,7 +15,16 @@ builder.Services.AddOpenTelemetry()
     .WithTracing(tracing => tracing
         .AddAspNetCoreInstrumentation(o => o.Filter = ctx => ctx.Request.Path != "/health")
         .AddHttpClientInstrumentation()
-        .AddOtlpExporter());
+        .AddOtlpExporter())
+    // OpenTelemetry metrics (S-16c, ADR-0023): golden signals for the request path —
+    // http.server.request.duration (traffic/errors/latency) + http.client.* for downstream hops, plus
+    // the built-in System.Runtime meter for saturation (GC, CPU, thread pool). Prometheus scrapes these
+    // from /metrics (mapped below); metrics aren't pushed over OTLP, so no collector hop (ADR-0023).
+    .WithMetrics(metrics => metrics
+        .AddAspNetCoreInstrumentation()
+        .AddHttpClientInstrumentation()
+        .AddMeter("System.Runtime")
+        .AddPrometheusExporter());
 
 var connectionString = builder.Configuration.GetConnectionString("Projection")
     ?? throw new InvalidOperationException("Missing connection string 'ConnectionStrings:Projection'");
@@ -30,6 +40,9 @@ await app.Services.MigrateProjectionAsync();
 
 app.MapGet("/health", () => "Healthy");
 
+// Prometheus scrape endpoint (S-16c): exposes the OTel metrics above in Prometheus text format.
+app.MapPrometheusScrapingEndpoint();
+
 // The read side of the projection. Public-safe field filtering is tightened in S-09; for now
 // the minimal projection only carries id + status (bsn/naam deferred — ADR-0008).
 app.MapGet("/register", async (ProjectionDbContext db, CancellationToken ct) =>

services/projection-api/ProjectionApi.Api/ProjectionApi.Api.csproj

@@ -6,6 +6,7 @@
 
   <ItemGroup>
     <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.17.0" />
+    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.17.0-beta.1" />
     <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.17.0" />
     <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.17.0" />