## What & why

The Helm chart landed in #167 with two gaps written into ADR-0033: `make k8s-lint` existed
but no CI job ran it, and *"a second deployment description to keep in step with compose —
nothing enforces that today; a drift check belongs in CI (follow-up)"*. Both are closed here.

**`make k8s-drift`** (`infra/helm/check-drift.py`, stdlib only) compares what each stack
actually deploys rather than diffing two files that differ by design: workload names and
resolved container images, taken from `docker compose config --format json` and a rendered
chart. The six differences that exist today are declared in `DEVIATIONS` with the reason
each was forced — the four `*-init` Django services folded into their web pods, and the two
bootstrap Jobs compose runs from the host — so only a *new* difference fails.

**A `k8s` CI job** runs `k8s-lint` then `k8s-drift` on every push and PR. No cluster, no
marketplace action: helm is fetched as the pinned static binary the Talos runbook already
gives developers.

Closes #168

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — the red commit reports all six
      real differences; the green commit declares them.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issue (`refs #168`).
- [x] CI green — awaiting the run on this PR (`make k8s-lint` and `make k8s-drift` pass locally).
- [x] `docker compose up` unaffected — no service, image or compose file is touched.
- [x] Docs updated — `docs/runbooks/ci.md` (job table + the one place local and CI now
      differ), `docs/runbooks/kubernetes-talos.md` §7/§"not ported", and ADR-0033's cost note.
- [x] No ADR needed: no new dependency (python stdlib, and helm/docker were already
      prerequisites of the `k8s-*` targets), no boundary moved, no §8 rule bent.
- [x] Not user-visible, so no demo note.

## Notes for reviewers

Verified by hand that both drift classes fail the check, not just that it passes today:

- bumping `OPENZAAK_TAG` in compose alone → reports `openzaak` and `oz-celery` with both
  image strings;
- adding a workload to `values.yaml` alone → reports it by name.

Deliberate limits (there is a `ponytail:` note in the script):

- **Names and images only**, as sets — no per-workload env, ports or volumes. Those differ
  by design in four documented places, so comparing them would mean re-encoding every
  deviation field by field for very little more signal.
- **The three observability workloads are rendered with `enabled=true`** by the check, even
  though both stacks default them off, so their images can't drift unwatched.
- **`k8s-lint`/`k8s-drift` are not in `make ci`**, to avoid making `helm` a hard
  prerequisite for everyone. That is now the only local/CI difference; it's called out in
  `docs/runbooks/ci.md`.

Follow-ups filed while reviewing the chart, not addressed here: #169 (the published docs
omit every ADR after 0010 and all runbooks but `ci.md`) and #170 (the production-posture
ADR #25 asked for — secrets are still plain text in `values.yaml`).Reviewed-on: #171

2026-09-18 13:25:24 +00:00

8.0 KiB

Raw Blame History

CI runbook — Gitea Actions

Status: active. The workflow .gitea/workflows/ci.yaml runs on Gitea's hosted ubuntu-latest runner — no self-hosted runner required. make ci is still the local gate — it runs the same checks via the same make targets, with one exception: the k8s job's targets are not in make ci, because helm is optional for everyone not deploying to Kubernetes. Run make k8s-lint k8s-drift by hand after touching the chart or the compose file.

The pipeline

.gitea/workflows/ci.yaml runs on every push and pull request to main. Each job calls a make target — the single source of truth for the checks, so local and CI cannot drift:

Job	Target	Needs
`lint`	`make lint` → `dotnet format … --verify-no-changes`	.NET 10 SDK
`build`	`make build` → `dotnet build … -c Release`	.NET 10 SDK
`unit`	`make unit` → `dotnet test … -c Release --filter "Category!=Integration"`	.NET 10 SDK
`frontend`	`make frontend` → Nx lint/test/build for the four portals	pnpm + Node
`k8s`	`make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`)	pinned `helm` binary + `docker compose`
`mutation`	`make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL); uploads the HTML report as an artifact	.NET 10 SDK
`verify-stack`	the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down`	container engine + egress (base images, nuget, `selectielijst.openzaak.nl`)

Why one verify-stack job, not three. The single self-hosted runner runs jobs sequentially, so booting OpenZaak once (instead of once per check) is the cheapest layout (issue #58). It subsumes the old integration, notifications, and compose-smoke jobs — the bring-up step is the "compose up reaches green health" gate. No setup-dotnet: the ACL test runs in a built image and every check reaches services by container IP (the runner can't reach published ports — see gitea-actions-gotchas.md §5/§6).

All uses: references are absolute, tag-pinned URLs (https://github.com/actions/checkout@v4, https://github.com/actions/setup-dotnet@v4) per CLAUDE.md §8.7 and §15 — Gitea Actions resolves them from GitHub.

verify-stack runs on a containerized runner. Workspace bind mounts do not reach the sibling containers Compose starts, so config/assets are streamed into external named volumes via docker cp (infra/seed-config.sh), and the upstream images are used verbatim (no build). If you add a service that needs a repo file at runtime, seed it the same way — don't bind-mount it. Note: bare docker compose up no longer self-seeds; use make up. See gitea-actions-gotchas.md.

Mutation testing (the ratchet)

The mutation job enforces test strength, not just coverage (CLAUDE.md §5). Stryker.NET is pinned as a local dotnet tool (.config/dotnet-tools.json), so it runs identically locally and in CI:

make mutation     # dotnet tool restore + dotnet stryker on the ACL

Config lives in services/acl/stryker-config.json. It runs in solution mode against Acl.slnx, mutating the two projects under test (Acl.Application, Acl.Infrastructure); Acl.Api has no tests and is skipped.

Baseline (the ratchet): the ACL is the first service with branching logic, so it sets the repo-wide baseline. Observed score 95%; enforced break threshold 90% (one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the job — when the score drops below break. Per §5 the baseline only moves up, and only as a slice's stated outcome; never lower it. New services add their own mutation run as they gain logic.

The HTML report is written to services/acl/StrykerOutput/<timestamp>/reports/ (git-ignored); open it to see survived vs. killed mutants.

In CI the mutation job publishes that report as the acl-mutation-report artifact (download it from the run's summary page). The upload step uses if: always(), so the report is available even when the ratchet fails — which is exactly when you want to inspect the survivors. It is the repo's first use of actions/upload-artifact, pinned to @v3: @v4 refuses to run on Gitea (its @actions/artifact v2 library blocks any non-github.com server as "GHES"), while @v3 speaks the artifact protocol Gitea implements. See gitea-actions-gotchas.md §4 (§15).

Running the stack locally without `make` (Windows / Docker Desktop)

make and the bash helpers assume a Unix shell. To bring the whole stack up on a machine without them (e.g. Windows + Docker Desktop), use the local compose file, which bind-mounts the config instead of seeding volumes — so it needs no make, no seed step, and no bash:

docker compose -f infra/docker-compose.local.yml up -d --build          # any engine
docker compose -f infra/docker-compose.local.yml up -d --build --wait    # Docker Desktop (Compose v2)
docker compose -f infra/docker-compose.local.yml down --volumes

On Linux/macOS the same thing is wrapped as make local / make local-down.

infra/docker-compose.local.yml mirrors the canonical infra/docker-compose.yml but swaps the external config volumes for bind mounts — valid locally because a local daemon can see the working directory (the seed/volume dance only exists for the containerized CI runner). Keep the two files in sync.

Running CI locally (`make ci`)

make ci runs the exact same checks as the pipeline — handy to run before pushing:

make ci          # lint + build + unit + mutation + verify — mirrors the pipeline
make lint        # or a single stage
make mutation    # Stryker.NET ratchet on the ACL
make verify      # the live-stack stage: full stack up once → ACL + NRC checks → down

make verify mirrors the CI verify-stack job: it boots the full stack once and runs both the ACL ↔ OpenZaak and OpenZaak → NRC checks against it. For fast, single-concern local iteration use a lighter throwaway stack instead:
make integration          # ACL ↔ OpenZaak only (no NRC)
make verify-notifications # OpenZaak → NRC delivery only

Prerequisites: .NET 10 SDK, a container engine with Compose v2, and curl.

On a rootless Podman box (the default dev setup here), the smoke target needs the Podman API socket and a Compose provider:

systemctl --user enable --now podman.socket            # start the API socket
ln -sf "$(command -v podman)" ~/.local/bin/docker      # docker -> podman shim
# install Docker Compose v2 into ~/.local/bin as `docker-compose` (the provider)

The Makefile auto-points DOCKER_HOST at /run/user/$(id -u)/podman/podman.sock when that socket exists and DOCKER_HOST is unset, so make smoke "just works" locally while leaving real Docker hosts / CI runners untouched.

Runner: `ubuntu-latest`

All jobs run on Gitea's hosted ubuntu-latest runner — no self-hosted runner setup is required. The hosted runner ships with Docker and Docker Compose v2, so make smoke (docker compose … up --wait) works without extra configuration.

If Gitea's hosted runners are unavailable and a self-hosted fallback is needed, register an act_runner with the ubuntu-latest label:

VER=0.2.11
curl -fsSL -o /usr/local/bin/act_runner \
  "https://dl.gitea.com/act_runner/${VER}/act_runner-${VER}-linux-amd64"
chmod +x /usr/local/bin/act_runner

act_runner register --no-interactive \
  --instance https://git.labs.respellion.tech \
  --token <REGISTRATION_TOKEN> \
  --name respellion-ci-1 \
  --labels "ubuntu-latest:docker://node:20-bookworm"

act_runner daemon

8.0 KiB Raw Blame History