arch(acl): ADR-0005 adopt Stryker.NET for mutation testing (refs #47)
All checks were successful
CI / lint (pull_request) Successful in 51s
CI / build (pull_request) Successful in 42s
CI / unit (pull_request) Successful in 45s
CI / mutation (pull_request) Successful in 1m24s
CI / compose-smoke (pull_request) Successful in 4m1s

Record the decision to adopt Stryker.NET (pinned local tool, solution mode on
Acl.slnx) and to set the first repo-wide mutation baseline on the ACL: observed
95%, enforced break threshold 90%. Document the ratchet, local run, and report
location in the CI runbook; add the ADR to the docs nav.

Proposed in #51 (adr-proposal). Refs #47.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-25 14:59:41 +02:00
parent e8510bf9c3
commit 7ecc184111
3 changed files with 96 additions and 1 deletions

View File

@@ -0,0 +1,68 @@
# ADR-0005: Stryker.NET for mutation testing, baseline on the ACL
- **Status:** Accepted
- **Date:** 2026-06-25
- **Deciders:** Respellion engineering
- **Relates to:** S-04b (#47); proposed in #51; supports CLAUDE.md §5 (mutation ratchet) and §3 (Definition of Done)
## Context
CLAUDE.md §5 mandates Stryker on every PR with a **ratchet**: CI fails on a regression
below the established baseline, and the baseline only ever moves up. §3 lists "mutation
(ratchet)" as a Definition-of-Done gate for **every** slice. Yet no baseline existed — so,
strictly, no slice could satisfy that gate. S-04b establishes it.
The ACL is the natural place to set the first baseline: it is the first service with real
branching logic — `OpenZaakGateway` (HTTP contract, geo CRS headers, error handling),
`ZgwToken` (HS256 JWT minting), and the `AclService` default-fill mapping. We need a tool
that:
- mutates C# and runs the existing xUnit suite per mutant,
- is reproducible (same version locally and in CI, no global install),
- understands this repo's `.slnx` solution format (used repo-wide),
- emits a break threshold CI can gate on.
## Decision
**Use [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) (`dotnet-stryker`),
pinned as a local dotnet tool**, configured in solution mode against `Acl.slnx`.
- Pinned in `.config/dotnet-tools.json` (v4.15.0); `dotnet tool restore` makes
`make mutation` reproducible from a fresh clone, locally and in CI — no global install.
- **Solution mode** (`stryker-config.json``solution: Acl.slnx`) mutates the two projects
under test (`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` is untested and skipped.
Stryker 4.15 reads `.slnx` directly, so no throwaway `.sln` shim is needed.
- A `mutation` make target runs it; it is wired into `make ci` and a parallel Gitea Actions
`mutation` job, keeping `make ci` an exact mirror of the pipeline.
**Baseline:** writing S-04b's tests surfaced that the ACL suite was thin — the initial
score was **35%** (survivors: unasserted CRS headers, null guards, error paths, and JWT
claims). Those tests were strengthened (killing the mutants honestly rather than lowering
the bar), raising the score to **95%**. The enforced `break` threshold is set to **90%**
one-mutant headroom over the ~20-mutant surface, since a single mutant is ≈5%.
## Consequences
- **Positive:** test *strength* is gated, not just coverage; the ratchet protects the ACL's
ZGW contract logic; the baseline is repo-wide and ratchets upward per §5.
- **Cost:** a new dependency (`dotnet-stryker`) and a slower CI job than unit tests (~25 s on
the small ACL). Pinned + tool-restored, so reproducible.
- **One accepted survivor:** a mutation of the empty-response *exception message string*.
Asserting exception message text is brittle and the behaviour (type + control flow) is
unchanged — treated as an equivalent mutant, not a test gap.
- **Commitment:** later slices ratchet the threshold up deliberately, never down (§5). New
services add their own mutation run as they gain branching logic (BFF, Domain, …).
- **Replaceable by:** no realistic .NET alternative — Stryker.NET is the tool §5 already
names; the fallback is no mutation testing, which §5 forbids.
## Alternatives considered
- **Global `dotnet tool install -g`** — rejected: not reproducible/pinned per clone; the
local manifest gives every checkout and the CI runner the same version.
- **Mutate the whole `register-referentie.slnx`** — rejected for this slice: scopes the
baseline to services with no logic yet (BFF skeleton), diluting the signal. Each service
opts in as it gains logic.
- **Application-only scope** — rejected: would leave `Acl.Infrastructure`'s HTTP/JWT logic —
the riskiest code — unguarded by the ratchet.
- **Coverage gate instead of mutation** — rejected: line coverage does not measure whether
tests would *catch* a regression; that is the whole point of §5.

View File

@@ -16,6 +16,7 @@ and CI cannot drift:
| `lint` | `make lint``dotnet format … --verify-no-changes` | .NET 10 SDK |
| `build` | `make build``dotnet build … -c Release` | .NET 10 SDK |
| `unit` | `make unit``dotnet test … -c Release` | .NET 10 SDK |
| `mutation` | `make mutation``dotnet tool restore``dotnet stryker` (ACL) | .NET 10 SDK |
| `compose-smoke` | `make smoke` → seed config volumes → `up -d` (full stack) → `up --wait` durable services → `down` | container engine + compose v2 |
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
@@ -30,6 +31,30 @@ Actions resolves them from GitHub.
> bare `docker compose up` no longer self-seeds; use `make up`. See
> [gitea-actions-gotchas.md](gitea-actions-gotchas.md).
## Mutation testing (the ratchet)
The `mutation` job enforces test *strength*, not just coverage (CLAUDE.md §5).
[Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) is pinned as a local
dotnet tool (`.config/dotnet-tools.json`), so it runs identically locally and in CI:
```bash
make mutation # dotnet tool restore + dotnet stryker on the ACL
```
Config lives in [`services/acl/stryker-config.json`](../../services/acl/stryker-config.json).
It runs in **solution mode** against `Acl.slnx`, mutating the two projects under test
(`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` has no tests and is skipped.
**Baseline (the ratchet):** the ACL is the first service with branching logic, so it
sets the repo-wide baseline. Observed score **95%**; enforced `break` threshold **90%**
(one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the
job — when the score drops below `break`. Per §5 the baseline only moves **up**, and only
as a slice's stated outcome; never lower it. New services add their own mutation run as
they gain logic.
The HTML report is written to `services/acl/StrykerOutput/<timestamp>/reports/` (git-ignored);
open it to see survived vs. killed mutants.
## Running the stack locally without `make` (Windows / Docker Desktop)
`make` and the bash helpers assume a Unix shell. To bring the whole stack up on a
@@ -55,8 +80,9 @@ the containerized CI runner). Keep the two files in sync.
Until the runner exists, run the full pipeline yourself before pushing:
```bash
make ci # lint + build + unit + smoke — what the pipeline runs
make ci # lint + build + unit + mutation + smoke — what the pipeline runs
make lint # or a single stage
make mutation # Stryker.NET ratchet on the ACL
make smoke # compose up --wait, curl /health, tear down
```

View File

@@ -26,6 +26,7 @@ nav:
- "ADR-0002: Catalogus design": architecture/adr-0002-catalogus-design.md
- "ADR-0003: ACL default-fill": architecture/adr-0003-default-fill.md
- "ADR-0004: BDD framework": architecture/adr-0004-bdd-framework.md
- "ADR-0005: Mutation testing": architecture/adr-0005-mutation-testing.md
- Working in Gitea: gitea-workflow.md
- Runbooks:
- CI: runbooks/ci.md