eho/register-referentie
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Stryker runs the ACL integration tests in its initial run — the mutation score is measured with 8 failing tests #174

New Issue
Open
opened 2026-09-10 10:08:23 +00:00 by not · 0 comments
not commented 2026-09-10 10:08:23 +00:00
Contributor
Copy Link
Copy Source

Every mutation run logs, twice:

[WRN] 8 tests are failing. Stryker will continue but outcome will be impacted.
[INF] Number of tests found: 94 for project …/Acl.Application/Acl.Application.csproj

Those 8 are Acl.IntegrationTests — they talk to a real OpenZaak (ADR-0006) and cannot pass in the mutation job, which brings up no stack. Confirmed locally:

$ dotnet test services/acl/Acl.Tests/Acl.Tests.csproj     # 86 passed
$ dotnet test services/acl/Acl.slnx
Passed!  - Failed: 0, Passed: 86 …  Acl.Tests.dll
Failed!  - Failed: 8, Passed:  0 …  Acl.IntegrationTests.dll

86 + 8 = the 94 Stryker discovers. services/acl/stryker-config.json already says "test-projects": ["Acl.Tests/Acl.Tests.csproj"], but "solution": "Acl.slnx" includes Acl.IntegrationTests, which references both mutated projects — so Stryker discovers and runs it anyway. Stryker.NET 4.15 has no --test-case-filter, so the xUnit trait that keeps these out of make unit (Category=Integration) has no effect here.

Why it matters: a test that fails in the initial run is unusable for killing mutants, so the ACL score is depressed by however many mutants those 8 tests would have caught. ACL currently reports 90.45 % against a break: 90 threshold — a 0.45 pp margin on a number that is measured wrong. The other three services (95.45 / 91.32 / 100) have no integration project and do not log the warning.

Candidate fixes, in rough order of preference — pick one in review:

  1. Keep the integration project out of what Stryker reads: a --project-scoped run, or a solution file that lists only the mutated projects plus Acl.Tests.
  2. Make the 8 tests skip rather than fail when OpenZaak is unreachable, which also fixes dotnet test services/acl/Acl.slnx for anyone without a stack.
  3. Drop Acl.IntegrationTests from Acl.slnx and give make integration its own solution file.

Re-baseline the ACL threshold after the fix — the score should only go up, and the ratchet (CLAUDE.md §5) should be set from a number measured with a green initial test run.

Every `mutation` run logs, twice: ``` [WRN] 8 tests are failing. Stryker will continue but outcome will be impacted. [INF] Number of tests found: 94 for project …/Acl.Application/Acl.Application.csproj ``` Those 8 are `Acl.IntegrationTests` — they talk to a real OpenZaak (ADR-0006) and cannot pass in the `mutation` job, which brings up no stack. Confirmed locally: ``` $ dotnet test services/acl/Acl.Tests/Acl.Tests.csproj # 86 passed $ dotnet test services/acl/Acl.slnx Passed! - Failed: 0, Passed: 86 … Acl.Tests.dll Failed! - Failed: 8, Passed: 0 … Acl.IntegrationTests.dll ``` 86 + 8 = the 94 Stryker discovers. `services/acl/stryker-config.json` already says `"test-projects": ["Acl.Tests/Acl.Tests.csproj"]`, but `"solution": "Acl.slnx"` includes `Acl.IntegrationTests`, which references both mutated projects — so Stryker discovers and runs it anyway. Stryker.NET 4.15 has no `--test-case-filter`, so the xUnit trait that keeps these out of `make unit` (`Category=Integration`) has no effect here. **Why it matters:** a test that fails in the initial run is unusable for killing mutants, so the ACL score is depressed by however many mutants those 8 tests would have caught. ACL currently reports **90.45 %** against a `break: 90` threshold — a 0.45 pp margin on a number that is measured wrong. The other three services (95.45 / 91.32 / 100) have no integration project and do not log the warning. Candidate fixes, in rough order of preference — pick one in review: 1. Keep the integration project out of what Stryker reads: a `--project`-scoped run, or a solution file that lists only the mutated projects plus `Acl.Tests`. 2. Make the 8 tests **skip** rather than fail when OpenZaak is unreachable, which also fixes `dotnet test services/acl/Acl.slnx` for anyone without a stack. 3. Drop `Acl.IntegrationTests` from `Acl.slnx` and give `make integration` its own solution file. Re-baseline the ACL threshold after the fix — the score should only go up, and the ratchet (CLAUDE.md §5) should be set from a number measured with a green initial test run.
not added this to the Iteration 6 — Production Posture milestone 2026-09-10 10:08:23 +00:00
not added the type:bugarea:acl labels 2026-09-10 10:08:23 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area:acl
area:bff
area:docs
area:domain
area:event-subscriber
area:infra
area:portal-behandel
area:portal-beheer
area:portal-openbaar
area:portal-self-service
area:projection
area:workflow
type:adr-proposal
type:bug
type:chore
type:slice
No labels area:acl type:bug
Milestone
No items
No Milestone Iteration 6 — Production Posture
Projects
Clear projects
No projects
Assignees
Clear assignees
eho (Edwin van den Houdt)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: eho/register-referentie#174