## What & why S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service. Closes #124 ### How - Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files. - `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name. - `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`). - `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`. ### Dependency (CLAUDE.md §13/§14) Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**. ## Definition of Done - [x] Linked Gitea issue (#124). - [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`). - [x] Implementation makes the test pass. - [ ] CI green — pending Gitea Actions run. - [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023). - [x] Docs updated — demo-script S-16c entry. - [x] ADR added — ADR-0024. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top. - `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code. - Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #129
184 lines
7.7 KiB
YAML
184 lines
7.7 KiB
YAML
name: CI
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches: [main]
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
# Supersede stale runs: a new push to the same branch/PR cancels the previous run, so the runner's
|
|
# concurrency slots aren't spent on commits nobody is waiting for (refs #127).
|
|
concurrency:
|
|
group: ci-${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
# Self-hosted runner — see docs/runbooks/ci.md for the runner setup.
|
|
# `uses:` are absolute, tag-pinned URLs (CLAUDE.md §8.7 / §15).
|
|
|
|
# Each job calls a `make` target — the same one developers run locally
|
|
# (`make ci`). The Makefile is the single source of truth; see docs/runbooks/ci.md.
|
|
|
|
jobs:
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
- uses: https://github.com/actions/setup-dotnet@v4
|
|
with:
|
|
dotnet-version: '10.0.x'
|
|
# Cache the NuGet package store so each .NET job restores from disk, not the network. There are
|
|
# no lock files (so setup-dotnet's built-in cache doesn't apply); key on the project files. @v3
|
|
# avoids the GHES guard that breaks @v4 on Gitea (gitea-actions-gotchas.md); cache is best-effort
|
|
# — a miss just restores from the network. See issue #73.
|
|
- uses: https://github.com/actions/cache@v3
|
|
with:
|
|
path: ~/.nuget/packages
|
|
key: nuget-${{ runner.os }}-${{ hashFiles('**/*.csproj') }}
|
|
restore-keys: |
|
|
nuget-${{ runner.os }}-
|
|
- run: make lint
|
|
|
|
build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
- uses: https://github.com/actions/setup-dotnet@v4
|
|
with:
|
|
dotnet-version: '10.0.x'
|
|
- uses: https://github.com/actions/cache@v3
|
|
with:
|
|
path: ~/.nuget/packages
|
|
key: nuget-${{ runner.os }}-${{ hashFiles('**/*.csproj') }}
|
|
restore-keys: |
|
|
nuget-${{ runner.os }}-
|
|
- run: make build
|
|
|
|
unit:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
- uses: https://github.com/actions/setup-dotnet@v4
|
|
with:
|
|
dotnet-version: '10.0.x'
|
|
- uses: https://github.com/actions/cache@v3
|
|
with:
|
|
path: ~/.nuget/packages
|
|
key: nuget-${{ runner.os }}-${{ hashFiles('**/*.csproj') }}
|
|
restore-keys: |
|
|
nuget-${{ runner.os }}-
|
|
- run: make unit
|
|
|
|
# Frontend (Nx/Angular) lane: install with pnpm, then Nx lint + test + build.
|
|
frontend:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
- uses: https://github.com/pnpm/action-setup@v4
|
|
with:
|
|
version: 11
|
|
- uses: https://github.com/actions/setup-node@v4
|
|
with:
|
|
node-version: '24'
|
|
cache: 'pnpm'
|
|
- run: make frontend
|
|
|
|
mutation:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
- uses: https://github.com/actions/setup-dotnet@v4
|
|
with:
|
|
dotnet-version: '10.0.x'
|
|
- uses: https://github.com/actions/cache@v3
|
|
with:
|
|
path: ~/.nuget/packages
|
|
key: nuget-${{ runner.os }}-${{ hashFiles('**/*.csproj') }}
|
|
restore-keys: |
|
|
nuget-${{ runner.os }}-
|
|
- run: make mutation
|
|
# Publish the Stryker HTML reports. `if: always()` uploads them even when the
|
|
# ratchet fails — that is exactly when you want to inspect the survivors.
|
|
# `continue-on-error` keeps the upload best-effort: the mutation *gate* is the
|
|
# ratchet (make mutation's exit code), not the report, so a Gitea artifact-backend
|
|
# 500 must not fail the job (gitea-actions-gotchas.md §4). Glob handles Stryker's
|
|
# non-deterministic StrykerOutput/<timestamp>/ dir. Pinned @v3: @v4's bundled
|
|
# @actions/artifact hard-aborts on non-github.com (GHES guard) — see the runbook.
|
|
- uses: https://github.com/actions/upload-artifact@v3
|
|
if: always()
|
|
continue-on-error: true
|
|
with:
|
|
name: acl-mutation-report
|
|
path: services/acl/StrykerOutput/**/reports/mutation-report.html
|
|
if-no-files-found: warn
|
|
- uses: https://github.com/actions/upload-artifact@v3
|
|
if: always()
|
|
continue-on-error: true
|
|
with:
|
|
name: event-subscriber-mutation-report
|
|
path: services/event-subscriber/StrykerOutput/**/reports/mutation-report.html
|
|
if-no-files-found: warn
|
|
- uses: https://github.com/actions/upload-artifact@v3
|
|
if: always()
|
|
continue-on-error: true
|
|
with:
|
|
name: domain-mutation-report
|
|
path: services/domain/StrykerOutput/**/reports/mutation-report.html
|
|
if-no-files-found: warn
|
|
- uses: https://github.com/actions/upload-artifact@v3
|
|
if: always()
|
|
continue-on-error: true
|
|
with:
|
|
name: bff-mutation-report
|
|
path: services/bff/StrykerOutput/**/reports/mutation-report.html
|
|
if-no-files-found: warn
|
|
|
|
# One stage for every check that needs the live stack. Booting OpenZaak once (instead
|
|
# of once per job) is the cheapest layout (issue #58). No setup-dotnet: the ACL test runs
|
|
# in a built image and everything reaches services by container IP. Needs Docker + egress
|
|
# (base images, nuget, selectielijst.openzaak.nl).
|
|
#
|
|
# `needs: [mutation]` is NOT a data dependency — it serialises the two memory-heavy jobs so
|
|
# they never co-schedule now the runner has capacity >1. A concurrent Stryker run + full-stack
|
|
# bring-up + Playwright browser on one host is what OOMs the e2e (commit d5e5fa2, #126). The
|
|
# light .NET/frontend jobs have no `needs`, so they still parallelise up to runner capacity.
|
|
# `if: !cancelled()` keeps verify-stack running even when the mutation ratchet fails (so we don't
|
|
# lose its signal) while still honouring run cancellation from the concurrency group above.
|
|
verify-stack:
|
|
needs: [mutation]
|
|
if: ${{ !cancelled() }}
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
# Bring the full stack up + wait for health — this also is the DoD "compose up
|
|
# reaches green health" smoke (it replaces the old compose-smoke job).
|
|
- name: Bring up the full stack & wait for health
|
|
run: make verify-up
|
|
- name: Observability backplane (Grafana + Tempo + Prometheus datasources)
|
|
run: OBS_TIMEOUT=180 make verify-observability
|
|
- name: ACL ↔ OpenZaak integration tests
|
|
run: make verify-acl
|
|
- name: OpenZaak → NRC notification delivery
|
|
run: make verify-nrc
|
|
- name: OpenZaak → NRC → Event Subscriber → projection-api
|
|
run: make verify-projection
|
|
- name: Domain → Flowable → ACL → OpenZaak
|
|
run: make verify-domain
|
|
- name: BFF → Keycloak + domain + projection
|
|
run: make verify-bff
|
|
- name: Distributed traces reach Tempo (one connected trace across services)
|
|
run: TRACING_TIMEOUT=120 make verify-tracing
|
|
- name: Golden-signal metrics scraped by Prometheus (/metrics on every service)
|
|
run: METRICS_TIMEOUT=120 make verify-metrics
|
|
- name: Self-service e2e (Playwright, login → submit → success)
|
|
run: make verify-e2e
|
|
# Log dump must precede teardown (which removes the containers).
|
|
- name: Dump container logs on failure
|
|
if: failure()
|
|
run: docker compose -f infra/docker-compose.yml logs --no-color --tail=100 oz-init openzaak nrc-init nrc-web nrc-celery nrc-beat flowable-db flowable-rest flowable-init keycloak acl bff domain projection-db event-subscriber projection-api self-service openbaar behandel tempo prometheus grafana 2>&1 || true
|
|
- name: Tear down
|
|
if: always()
|
|
run: make down
|