arch(acl): ADR-0005 adopt Stryker.NET for mutation testing (refs #47)
Record the decision to adopt Stryker.NET (pinned local tool, solution mode on Acl.slnx) and to set the first repo-wide mutation baseline on the ACL: observed 95%, enforced break threshold 90%. Document the ratchet, local run, and report location in the CI runbook; add the ADR to the docs nav. Proposed in #51 (adr-proposal). Refs #47. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
68
docs/architecture/adr-0005-mutation-testing.md
Normal file
68
docs/architecture/adr-0005-mutation-testing.md
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
# ADR-0005: Stryker.NET for mutation testing, baseline on the ACL
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-06-25
|
||||||
|
- **Deciders:** Respellion engineering
|
||||||
|
- **Relates to:** S-04b (#47); proposed in #51; supports CLAUDE.md §5 (mutation ratchet) and §3 (Definition of Done)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
CLAUDE.md §5 mandates Stryker on every PR with a **ratchet**: CI fails on a regression
|
||||||
|
below the established baseline, and the baseline only ever moves up. §3 lists "mutation
|
||||||
|
(ratchet)" as a Definition-of-Done gate for **every** slice. Yet no baseline existed — so,
|
||||||
|
strictly, no slice could satisfy that gate. S-04b establishes it.
|
||||||
|
|
||||||
|
The ACL is the natural place to set the first baseline: it is the first service with real
|
||||||
|
branching logic — `OpenZaakGateway` (HTTP contract, geo CRS headers, error handling),
|
||||||
|
`ZgwToken` (HS256 JWT minting), and the `AclService` default-fill mapping. We need a tool
|
||||||
|
that:
|
||||||
|
|
||||||
|
- mutates C# and runs the existing xUnit suite per mutant,
|
||||||
|
- is reproducible (same version locally and in CI, no global install),
|
||||||
|
- understands this repo's `.slnx` solution format (used repo-wide),
|
||||||
|
- emits a break threshold CI can gate on.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
**Use [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) (`dotnet-stryker`),
|
||||||
|
pinned as a local dotnet tool**, configured in solution mode against `Acl.slnx`.
|
||||||
|
|
||||||
|
- Pinned in `.config/dotnet-tools.json` (v4.15.0); `dotnet tool restore` makes
|
||||||
|
`make mutation` reproducible from a fresh clone, locally and in CI — no global install.
|
||||||
|
- **Solution mode** (`stryker-config.json` → `solution: Acl.slnx`) mutates the two projects
|
||||||
|
under test (`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` is untested and skipped.
|
||||||
|
Stryker 4.15 reads `.slnx` directly, so no throwaway `.sln` shim is needed.
|
||||||
|
- A `mutation` make target runs it; it is wired into `make ci` and a parallel Gitea Actions
|
||||||
|
`mutation` job, keeping `make ci` an exact mirror of the pipeline.
|
||||||
|
|
||||||
|
**Baseline:** writing S-04b's tests surfaced that the ACL suite was thin — the initial
|
||||||
|
score was **35%** (survivors: unasserted CRS headers, null guards, error paths, and JWT
|
||||||
|
claims). Those tests were strengthened (killing the mutants honestly rather than lowering
|
||||||
|
the bar), raising the score to **95%**. The enforced `break` threshold is set to **90%** —
|
||||||
|
one-mutant headroom over the ~20-mutant surface, since a single mutant is ≈5%.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **Positive:** test *strength* is gated, not just coverage; the ratchet protects the ACL's
|
||||||
|
ZGW contract logic; the baseline is repo-wide and ratchets upward per §5.
|
||||||
|
- **Cost:** a new dependency (`dotnet-stryker`) and a slower CI job than unit tests (~25 s on
|
||||||
|
the small ACL). Pinned + tool-restored, so reproducible.
|
||||||
|
- **One accepted survivor:** a mutation of the empty-response *exception message string*.
|
||||||
|
Asserting exception message text is brittle and the behaviour (type + control flow) is
|
||||||
|
unchanged — treated as an equivalent mutant, not a test gap.
|
||||||
|
- **Commitment:** later slices ratchet the threshold up deliberately, never down (§5). New
|
||||||
|
services add their own mutation run as they gain branching logic (BFF, Domain, …).
|
||||||
|
- **Replaceable by:** no realistic .NET alternative — Stryker.NET is the tool §5 already
|
||||||
|
names; the fallback is no mutation testing, which §5 forbids.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- **Global `dotnet tool install -g`** — rejected: not reproducible/pinned per clone; the
|
||||||
|
local manifest gives every checkout and the CI runner the same version.
|
||||||
|
- **Mutate the whole `register-referentie.slnx`** — rejected for this slice: scopes the
|
||||||
|
baseline to services with no logic yet (BFF skeleton), diluting the signal. Each service
|
||||||
|
opts in as it gains logic.
|
||||||
|
- **Application-only scope** — rejected: would leave `Acl.Infrastructure`'s HTTP/JWT logic —
|
||||||
|
the riskiest code — unguarded by the ratchet.
|
||||||
|
- **Coverage gate instead of mutation** — rejected: line coverage does not measure whether
|
||||||
|
tests would *catch* a regression; that is the whole point of §5.
|
||||||
@@ -16,6 +16,7 @@ and CI cannot drift:
|
|||||||
| `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK |
|
| `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK |
|
||||||
| `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK |
|
| `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK |
|
||||||
| `unit` | `make unit` → `dotnet test … -c Release` | .NET 10 SDK |
|
| `unit` | `make unit` → `dotnet test … -c Release` | .NET 10 SDK |
|
||||||
|
| `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL) | .NET 10 SDK |
|
||||||
| `compose-smoke` | `make smoke` → seed config volumes → `up -d` (full stack) → `up --wait` durable services → `down` | container engine + compose v2 |
|
| `compose-smoke` | `make smoke` → seed config volumes → `up -d` (full stack) → `up --wait` durable services → `down` | container engine + compose v2 |
|
||||||
|
|
||||||
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
|
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
|
||||||
@@ -30,6 +31,30 @@ Actions resolves them from GitHub.
|
|||||||
> bare `docker compose up` no longer self-seeds; use `make up`. See
|
> bare `docker compose up` no longer self-seeds; use `make up`. See
|
||||||
> [gitea-actions-gotchas.md](gitea-actions-gotchas.md).
|
> [gitea-actions-gotchas.md](gitea-actions-gotchas.md).
|
||||||
|
|
||||||
|
## Mutation testing (the ratchet)
|
||||||
|
|
||||||
|
The `mutation` job enforces test *strength*, not just coverage (CLAUDE.md §5).
|
||||||
|
[Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) is pinned as a local
|
||||||
|
dotnet tool (`.config/dotnet-tools.json`), so it runs identically locally and in CI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make mutation # dotnet tool restore + dotnet stryker on the ACL
|
||||||
|
```
|
||||||
|
|
||||||
|
Config lives in [`services/acl/stryker-config.json`](../../services/acl/stryker-config.json).
|
||||||
|
It runs in **solution mode** against `Acl.slnx`, mutating the two projects under test
|
||||||
|
(`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` has no tests and is skipped.
|
||||||
|
|
||||||
|
**Baseline (the ratchet):** the ACL is the first service with branching logic, so it
|
||||||
|
sets the repo-wide baseline. Observed score **95%**; enforced `break` threshold **90%**
|
||||||
|
(one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the
|
||||||
|
job — when the score drops below `break`. Per §5 the baseline only moves **up**, and only
|
||||||
|
as a slice's stated outcome; never lower it. New services add their own mutation run as
|
||||||
|
they gain logic.
|
||||||
|
|
||||||
|
The HTML report is written to `services/acl/StrykerOutput/<timestamp>/reports/` (git-ignored);
|
||||||
|
open it to see survived vs. killed mutants.
|
||||||
|
|
||||||
## Running the stack locally without `make` (Windows / Docker Desktop)
|
## Running the stack locally without `make` (Windows / Docker Desktop)
|
||||||
|
|
||||||
`make` and the bash helpers assume a Unix shell. To bring the whole stack up on a
|
`make` and the bash helpers assume a Unix shell. To bring the whole stack up on a
|
||||||
@@ -55,8 +80,9 @@ the containerized CI runner). Keep the two files in sync.
|
|||||||
Until the runner exists, run the full pipeline yourself before pushing:
|
Until the runner exists, run the full pipeline yourself before pushing:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make ci # lint + build + unit + smoke — what the pipeline runs
|
make ci # lint + build + unit + mutation + smoke — what the pipeline runs
|
||||||
make lint # or a single stage
|
make lint # or a single stage
|
||||||
|
make mutation # Stryker.NET ratchet on the ACL
|
||||||
make smoke # compose up --wait, curl /health, tear down
|
make smoke # compose up --wait, curl /health, tear down
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ nav:
|
|||||||
- "ADR-0002: Catalogus design": architecture/adr-0002-catalogus-design.md
|
- "ADR-0002: Catalogus design": architecture/adr-0002-catalogus-design.md
|
||||||
- "ADR-0003: ACL default-fill": architecture/adr-0003-default-fill.md
|
- "ADR-0003: ACL default-fill": architecture/adr-0003-default-fill.md
|
||||||
- "ADR-0004: BDD framework": architecture/adr-0004-bdd-framework.md
|
- "ADR-0004: BDD framework": architecture/adr-0004-bdd-framework.md
|
||||||
|
- "ADR-0005: Mutation testing": architecture/adr-0005-mutation-testing.md
|
||||||
- Working in Gitea: gitea-workflow.md
|
- Working in Gitea: gitea-workflow.md
|
||||||
- Runbooks:
|
- Runbooks:
|
||||||
- CI: runbooks/ci.md
|
- CI: runbooks/ci.md
|
||||||
|
|||||||
Reference in New Issue
Block a user