diff --git a/docs/architecture/adr-0005-mutation-testing.md b/docs/architecture/adr-0005-mutation-testing.md new file mode 100644 index 0000000..23f11e9 --- /dev/null +++ b/docs/architecture/adr-0005-mutation-testing.md @@ -0,0 +1,68 @@ +# ADR-0005: Stryker.NET for mutation testing, baseline on the ACL + +- **Status:** Accepted +- **Date:** 2026-06-25 +- **Deciders:** Respellion engineering +- **Relates to:** S-04b (#47); proposed in #51; supports CLAUDE.md §5 (mutation ratchet) and §3 (Definition of Done) + +## Context + +CLAUDE.md §5 mandates Stryker on every PR with a **ratchet**: CI fails on a regression +below the established baseline, and the baseline only ever moves up. §3 lists "mutation +(ratchet)" as a Definition-of-Done gate for **every** slice. Yet no baseline existed — so, +strictly, no slice could satisfy that gate. S-04b establishes it. + +The ACL is the natural place to set the first baseline: it is the first service with real +branching logic — `OpenZaakGateway` (HTTP contract, geo CRS headers, error handling), +`ZgwToken` (HS256 JWT minting), and the `AclService` default-fill mapping. We need a tool +that: + +- mutates C# and runs the existing xUnit suite per mutant, +- is reproducible (same version locally and in CI, no global install), +- understands this repo's `.slnx` solution format (used repo-wide), +- emits a break threshold CI can gate on. + +## Decision + +**Use [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) (`dotnet-stryker`), +pinned as a local dotnet tool**, configured in solution mode against `Acl.slnx`. + +- Pinned in `.config/dotnet-tools.json` (v4.15.0); `dotnet tool restore` makes + `make mutation` reproducible from a fresh clone, locally and in CI — no global install. +- **Solution mode** (`stryker-config.json` → `solution: Acl.slnx`) mutates the two projects + under test (`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` is untested and skipped. + Stryker 4.15 reads `.slnx` directly, so no throwaway `.sln` shim is needed. +- A `mutation` make target runs it; it is wired into `make ci` and a parallel Gitea Actions + `mutation` job, keeping `make ci` an exact mirror of the pipeline. + +**Baseline:** writing S-04b's tests surfaced that the ACL suite was thin — the initial +score was **35%** (survivors: unasserted CRS headers, null guards, error paths, and JWT +claims). Those tests were strengthened (killing the mutants honestly rather than lowering +the bar), raising the score to **95%**. The enforced `break` threshold is set to **90%** — +one-mutant headroom over the ~20-mutant surface, since a single mutant is ≈5%. + +## Consequences + +- **Positive:** test *strength* is gated, not just coverage; the ratchet protects the ACL's + ZGW contract logic; the baseline is repo-wide and ratchets upward per §5. +- **Cost:** a new dependency (`dotnet-stryker`) and a slower CI job than unit tests (~25 s on + the small ACL). Pinned + tool-restored, so reproducible. +- **One accepted survivor:** a mutation of the empty-response *exception message string*. + Asserting exception message text is brittle and the behaviour (type + control flow) is + unchanged — treated as an equivalent mutant, not a test gap. +- **Commitment:** later slices ratchet the threshold up deliberately, never down (§5). New + services add their own mutation run as they gain branching logic (BFF, Domain, …). +- **Replaceable by:** no realistic .NET alternative — Stryker.NET is the tool §5 already + names; the fallback is no mutation testing, which §5 forbids. + +## Alternatives considered + +- **Global `dotnet tool install -g`** — rejected: not reproducible/pinned per clone; the + local manifest gives every checkout and the CI runner the same version. +- **Mutate the whole `register-referentie.slnx`** — rejected for this slice: scopes the + baseline to services with no logic yet (BFF skeleton), diluting the signal. Each service + opts in as it gains logic. +- **Application-only scope** — rejected: would leave `Acl.Infrastructure`'s HTTP/JWT logic — + the riskiest code — unguarded by the ratchet. +- **Coverage gate instead of mutation** — rejected: line coverage does not measure whether + tests would *catch* a regression; that is the whole point of §5. diff --git a/docs/runbooks/ci.md b/docs/runbooks/ci.md index 704a6c8..9c11583 100644 --- a/docs/runbooks/ci.md +++ b/docs/runbooks/ci.md @@ -16,6 +16,7 @@ and CI cannot drift: | `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK | | `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK | | `unit` | `make unit` → `dotnet test … -c Release` | .NET 10 SDK | +| `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL) | .NET 10 SDK | | `compose-smoke` | `make smoke` → seed config volumes → `up -d` (full stack) → `up --wait` durable services → `down` | container engine + compose v2 | All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`, @@ -30,6 +31,30 @@ Actions resolves them from GitHub. > bare `docker compose up` no longer self-seeds; use `make up`. See > [gitea-actions-gotchas.md](gitea-actions-gotchas.md). +## Mutation testing (the ratchet) + +The `mutation` job enforces test *strength*, not just coverage (CLAUDE.md §5). +[Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) is pinned as a local +dotnet tool (`.config/dotnet-tools.json`), so it runs identically locally and in CI: + +```bash +make mutation # dotnet tool restore + dotnet stryker on the ACL +``` + +Config lives in [`services/acl/stryker-config.json`](../../services/acl/stryker-config.json). +It runs in **solution mode** against `Acl.slnx`, mutating the two projects under test +(`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` has no tests and is skipped. + +**Baseline (the ratchet):** the ACL is the first service with branching logic, so it +sets the repo-wide baseline. Observed score **95%**; enforced `break` threshold **90%** +(one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the +job — when the score drops below `break`. Per §5 the baseline only moves **up**, and only +as a slice's stated outcome; never lower it. New services add their own mutation run as +they gain logic. + +The HTML report is written to `services/acl/StrykerOutput//reports/` (git-ignored); +open it to see survived vs. killed mutants. + ## Running the stack locally without `make` (Windows / Docker Desktop) `make` and the bash helpers assume a Unix shell. To bring the whole stack up on a @@ -55,8 +80,9 @@ the containerized CI runner). Keep the two files in sync. Until the runner exists, run the full pipeline yourself before pushing: ```bash -make ci # lint + build + unit + smoke — what the pipeline runs +make ci # lint + build + unit + mutation + smoke — what the pipeline runs make lint # or a single stage +make mutation # Stryker.NET ratchet on the ACL make smoke # compose up --wait, curl /health, tear down ``` diff --git a/mkdocs.yml b/mkdocs.yml index 92e4c5b..33f4935 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -26,6 +26,7 @@ nav: - "ADR-0002: Catalogus design": architecture/adr-0002-catalogus-design.md - "ADR-0003: ACL default-fill": architecture/adr-0003-default-fill.md - "ADR-0004: BDD framework": architecture/adr-0004-bdd-framework.md + - "ADR-0005: Mutation testing": architecture/adr-0005-mutation-testing.md - Working in Gitea: gitea-workflow.md - Runbooks: - CI: runbooks/ci.md