Niek Otten
7ecc184111
Record the decision to adopt Stryker.NET (pinned local tool, solution mode on Acl.slnx) and to set the first repo-wide mutation baseline on the ACL: observed 95%, enforced break threshold 90%. Document the ratchet, local run, and report location in the CI runbook; add the ADR to the docs nav. Proposed in #51 (adr-proposal). Refs #47. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
69 lines
3.9 KiB
Markdown
69 lines
3.9 KiB
Markdown
# ADR-0005: Stryker.NET for mutation testing, baseline on the ACL
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-06-25
|
|
- **Deciders:** Respellion engineering
|
|
- **Relates to:** S-04b (#47); proposed in #51; supports CLAUDE.md §5 (mutation ratchet) and §3 (Definition of Done)
|
|
|
|
## Context
|
|
|
|
CLAUDE.md §5 mandates Stryker on every PR with a **ratchet**: CI fails on a regression
|
|
below the established baseline, and the baseline only ever moves up. §3 lists "mutation
|
|
(ratchet)" as a Definition-of-Done gate for **every** slice. Yet no baseline existed — so,
|
|
strictly, no slice could satisfy that gate. S-04b establishes it.
|
|
|
|
The ACL is the natural place to set the first baseline: it is the first service with real
|
|
branching logic — `OpenZaakGateway` (HTTP contract, geo CRS headers, error handling),
|
|
`ZgwToken` (HS256 JWT minting), and the `AclService` default-fill mapping. We need a tool
|
|
that:
|
|
|
|
- mutates C# and runs the existing xUnit suite per mutant,
|
|
- is reproducible (same version locally and in CI, no global install),
|
|
- understands this repo's `.slnx` solution format (used repo-wide),
|
|
- emits a break threshold CI can gate on.
|
|
|
|
## Decision
|
|
|
|
**Use [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) (`dotnet-stryker`),
|
|
pinned as a local dotnet tool**, configured in solution mode against `Acl.slnx`.
|
|
|
|
- Pinned in `.config/dotnet-tools.json` (v4.15.0); `dotnet tool restore` makes
|
|
`make mutation` reproducible from a fresh clone, locally and in CI — no global install.
|
|
- **Solution mode** (`stryker-config.json` → `solution: Acl.slnx`) mutates the two projects
|
|
under test (`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` is untested and skipped.
|
|
Stryker 4.15 reads `.slnx` directly, so no throwaway `.sln` shim is needed.
|
|
- A `mutation` make target runs it; it is wired into `make ci` and a parallel Gitea Actions
|
|
`mutation` job, keeping `make ci` an exact mirror of the pipeline.
|
|
|
|
**Baseline:** writing S-04b's tests surfaced that the ACL suite was thin — the initial
|
|
score was **35%** (survivors: unasserted CRS headers, null guards, error paths, and JWT
|
|
claims). Those tests were strengthened (killing the mutants honestly rather than lowering
|
|
the bar), raising the score to **95%**. The enforced `break` threshold is set to **90%** —
|
|
one-mutant headroom over the ~20-mutant surface, since a single mutant is ≈5%.
|
|
|
|
## Consequences
|
|
|
|
- **Positive:** test *strength* is gated, not just coverage; the ratchet protects the ACL's
|
|
ZGW contract logic; the baseline is repo-wide and ratchets upward per §5.
|
|
- **Cost:** a new dependency (`dotnet-stryker`) and a slower CI job than unit tests (~25 s on
|
|
the small ACL). Pinned + tool-restored, so reproducible.
|
|
- **One accepted survivor:** a mutation of the empty-response *exception message string*.
|
|
Asserting exception message text is brittle and the behaviour (type + control flow) is
|
|
unchanged — treated as an equivalent mutant, not a test gap.
|
|
- **Commitment:** later slices ratchet the threshold up deliberately, never down (§5). New
|
|
services add their own mutation run as they gain branching logic (BFF, Domain, …).
|
|
- **Replaceable by:** no realistic .NET alternative — Stryker.NET is the tool §5 already
|
|
names; the fallback is no mutation testing, which §5 forbids.
|
|
|
|
## Alternatives considered
|
|
|
|
- **Global `dotnet tool install -g`** — rejected: not reproducible/pinned per clone; the
|
|
local manifest gives every checkout and the CI runner the same version.
|
|
- **Mutate the whole `register-referentie.slnx`** — rejected for this slice: scopes the
|
|
baseline to services with no logic yet (BFF skeleton), diluting the signal. Each service
|
|
opts in as it gains logic.
|
|
- **Application-only scope** — rejected: would leave `Acl.Infrastructure`'s HTTP/JWT logic —
|
|
the riskiest code — unguarded by the ratchet.
|
|
- **Coverage gate instead of mutation** — rejected: line coverage does not measure whether
|
|
tests would *catch* a regression; that is the whole point of §5.
|