CI / k8s (push) Successful in 6s
CI / unit (push) Canceled after 0s
CI / frontend (push) Canceled after 0s
CI / mutation (push) Canceled after 0s
CI / verify-stack (push) Canceled after 0s
CI / build (push) Canceled after 9s
CI / lint (push) Canceled after 24s
Deploy to Talos / deploy (push) Successful in 5m1s
## What & why The chart has been deployable by hand since #25 and linted in CI since #168. This makes a merged PR actually ship it to the Talos VM on the lab server. `.gitea/workflows/deploy.yaml` runs on a push to `main` (a squash-merged PR) and on manual dispatch: 1. **Tunnel** — neither the Kubernetes API nor the in-cluster registry is publicly reachable, so 6443, 30500 and 30141 are forwarded over the same SSH hop into the Fedora host that the Gitea-runner pipeline uses (`ssh -p 6667 user@labs.respellion.tech`). 2. **Images** — `make k8s-images K8S_REGISTRY=localhost:30500`, pushed *through* the tunnel. 3. **Deploy** — `make k8s-reseed TALOS_HOST=… K8S_REGISTRY=<vm-ip>:30500`, pulled by the node from its own NodePort. 4. **Roll** — `rollout restart` + `rollout status` on the nine repo deployments. 5. **Smoke** — `GET /openbaar/register` through the openbaar portal. Three decisions worth the review: - **One registry, two names.** The push target (`localhost:30500`, the tunnel) and the pull target (`<vm-ip>:30500`, the node's own NodePort) address the same store. The pull name has to be the one in the node's registry-mirror patch, which is what makes plain HTTP acceptable. - **`k8s-reseed`, not `k8s-up`.** A Job's pod template is immutable, so a chart change to any bootstrap Job would otherwise fail the upgrade with `cannot patch … with kind Job`. The Jobs are idempotent by design, so re-running them every deploy is safe and removes that whole class of failure. Cost: a few minutes per deploy, and `seed-zaaktype` needs egress from the VM. - **No re-run of the checks.** PR CI is the merge gate, so `main` is green by construction. Deploys **queue** (`cancel-in-progress: false`) — a `helm upgrade` killed half-way leaves the release in `pending-upgrade` and has to be unwedged by hand. Settings on the repo (already added): secrets `TALOS_SSH_KEY` and `TALOS_KUBECONFIG` (base64, and its `server:` must be `https://127.0.0.1:6443` — Talos puts `127.0.0.1` in the apiserver cert SANs, so TLS still verifies through the tunnel); variables `TALOS_VM_IP` (default `192.168.122.173`) and `TALOS_HOST` (default `localhost`). Closes #175 ## Definition of Done - [x] Linked Gitea issue (above). - [ ] Failing test committed before the implementation — **n/a**: this is a deployment workflow with no unit under test. Its check is the run itself: `rollout status` and the public-register smoke both have to pass or the job fails. `make k8s-lint` / `make k8s-drift` (#168) already gate the chart it deploys. - [x] Implementation — one workflow file, no production code touched. - [x] Conventional Commits referencing the issue (`refs #175`). - [ ] CI green — awaiting the run on this PR. - [x] `docker compose up` unaffected — no service, image or compose file is touched. - [x] Docs updated — `docs/runbooks/kubernetes-talos.md` §9 (the tunnel, the two registry names, the secrets table, the smoke) and a pointer from `docs/runbooks/ci.md`. - [x] No ADR needed: no new dependency (kubectl/helm/crane are already prerequisites of the `k8s-*` targets), no service boundary moved, no CLAUDE.md §8 rule bent. - [ ] Demo note — not user-visible. ## Notes for reviewers - **The first deploy is the real test.** It cannot be dry-run: the tunnel, the secrets and the registry only exist on the lab server. Merging is how we find out; `Pods on failure` dumps `get pods,jobs` if it doesn't. - **Known gap — the portals still aren't browsable.** PKCE needs a secure context, so a NodePort on an IP can't serve them (runbook §5); they need `make k8s-portals` or an SSH forward. Giving the server a hostname + TLS is the follow-up, and is where `TALOS_HOST` stops defaulting to `localhost`. - **Databases are `emptyDir`.** Any change to a database pod's template wipes it; the `k8s-reseed` in the deploy re-runs the bootstrap, so the stack recovers, but submitted registrations do not. Persistence is runbook §6. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #176
160 lines
8.2 KiB
Markdown
160 lines
8.2 KiB
Markdown
# CI runbook — Gitea Actions
|
|
|
|
> **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's
|
|
> hosted `ubuntu-latest` runner — no self-hosted runner required.
|
|
> **`make ci` is still the local gate** — it runs the same checks via the same
|
|
> `make` targets, with one exception: the `k8s` job's targets are not in `make ci`,
|
|
> because `helm` is optional for everyone not deploying to Kubernetes. Run
|
|
> `make k8s-lint k8s-drift` by hand after touching the chart or the compose file.
|
|
|
|
## The pipeline
|
|
|
|
`.gitea/workflows/ci.yaml` runs on every push and pull request to `main`. Each job
|
|
calls a `make` target — the **single source of truth** for the checks, so local
|
|
and CI cannot drift:
|
|
|
|
| Job | Target | Needs |
|
|
|---|---|---|
|
|
| `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK |
|
|
| `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK |
|
|
| `unit` | `make unit` → `dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK |
|
|
| `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
|
|
| `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
|
|
| `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
|
|
| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
|
|
|
|
> **Why one `verify-stack` job, not three.** The single self-hosted runner runs jobs
|
|
> **sequentially**, so booting OpenZaak once (instead of once per check) is the
|
|
> cheapest layout (issue #58). It subsumes the old `integration`, `notifications`, and
|
|
> `compose-smoke` jobs — the bring-up step *is* the "compose up reaches green health"
|
|
> gate. No `setup-dotnet`: the ACL test runs in a built image and every check reaches
|
|
> services by **container IP** (the runner can't reach published ports — see
|
|
> [gitea-actions-gotchas.md §5/§6](gitea-actions-gotchas.md)).
|
|
|
|
A second workflow, `.gitea/workflows/deploy.yaml`, deploys the stack to the Talos
|
|
cluster on the lab server when a PR is merged to `main` — see
|
|
[kubernetes-talos.md §9](kubernetes-talos.md) for its secrets and the SSH tunnel it needs.
|
|
|
|
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
|
|
`https://github.com/actions/setup-dotnet@v4`) per CLAUDE.md §8.7 and §15 — Gitea
|
|
Actions resolves them from GitHub.
|
|
|
|
> **`verify-stack` runs on a containerized runner.** Workspace bind mounts do
|
|
> **not** reach the sibling containers Compose starts, so config/assets are
|
|
> streamed into external named volumes via `docker cp` (`infra/seed-config.sh`),
|
|
> and the upstream images are used verbatim (no build). If you add a service that
|
|
> needs a repo file at runtime, seed it the same way — don't bind-mount it. Note:
|
|
> bare `docker compose up` no longer self-seeds; use `make up`. See
|
|
> [gitea-actions-gotchas.md](gitea-actions-gotchas.md).
|
|
|
|
## Mutation testing (the ratchet)
|
|
|
|
The `mutation` job enforces test *strength*, not just coverage (CLAUDE.md §5).
|
|
[Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) is pinned as a local
|
|
dotnet tool (`.config/dotnet-tools.json`), so it runs identically locally and in CI:
|
|
|
|
```bash
|
|
make mutation # dotnet tool restore + dotnet stryker on the ACL
|
|
```
|
|
|
|
Config lives in [`services/acl/stryker-config.json`](../../services/acl/stryker-config.json).
|
|
It runs in **solution mode** against `Acl.slnx`, mutating the two projects under test
|
|
(`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` has no tests and is skipped.
|
|
|
|
**Baseline (the ratchet):** the ACL is the first service with branching logic, so it
|
|
sets the repo-wide baseline. Observed score **95%**; enforced `break` threshold **90%**
|
|
(one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the
|
|
job — when the score drops below `break`. Per §5 the baseline only moves **up**, and only
|
|
as a slice's stated outcome; never lower it. New services add their own mutation run as
|
|
they gain logic.
|
|
|
|
The HTML report is written to `services/acl/StrykerOutput/<timestamp>/reports/` (git-ignored);
|
|
open it to see survived vs. killed mutants.
|
|
|
|
In CI the `mutation` job publishes that report as the **`acl-mutation-report`** artifact
|
|
(download it from the run's summary page). The upload step uses `if: always()`, so the
|
|
report is available even when the ratchet *fails* — which is exactly when you want to inspect
|
|
the survivors. It is the repo's first use of `actions/upload-artifact`, pinned to **`@v3`**:
|
|
`@v4` refuses to run on Gitea (its `@actions/artifact` v2 library blocks any non-github.com
|
|
server as "GHES"), while `@v3` speaks the artifact protocol Gitea implements. See
|
|
[gitea-actions-gotchas.md §4](gitea-actions-gotchas.md) (§15).
|
|
|
|
## Running the stack locally without `make` (Windows / Docker Desktop)
|
|
|
|
`make` and the bash helpers assume a Unix shell. To bring the whole stack up on a
|
|
machine without them (e.g. Windows + Docker Desktop), use the **local compose
|
|
file**, which bind-mounts the config instead of seeding volumes — so it needs no
|
|
`make`, no seed step, and no bash:
|
|
|
|
```bash
|
|
docker compose -f infra/docker-compose.local.yml up -d --build # any engine
|
|
docker compose -f infra/docker-compose.local.yml up -d --build --wait # Docker Desktop (Compose v2)
|
|
docker compose -f infra/docker-compose.local.yml down --volumes
|
|
```
|
|
|
|
On Linux/macOS the same thing is wrapped as `make local` / `make local-down`.
|
|
|
|
`infra/docker-compose.local.yml` mirrors the canonical `infra/docker-compose.yml`
|
|
but swaps the external config volumes for bind mounts — valid locally because a
|
|
local daemon can see the working directory (the seed/volume dance only exists for
|
|
the containerized CI runner). Keep the two files in sync.
|
|
|
|
## Running CI locally (`make ci`)
|
|
|
|
`make ci` runs the exact same checks as the pipeline — handy to run before pushing:
|
|
|
|
```bash
|
|
make ci # lint + build + unit + mutation + verify — mirrors the pipeline
|
|
make lint # or a single stage
|
|
make mutation # Stryker.NET ratchet on the ACL
|
|
make verify # the live-stack stage: full stack up once → ACL + NRC checks → down
|
|
```
|
|
|
|
> **`make verify`** mirrors the CI `verify-stack` job: it boots the full stack once and
|
|
> runs both the ACL ↔ OpenZaak and OpenZaak → NRC checks against it. For fast,
|
|
> single-concern local iteration use a lighter throwaway stack instead:
|
|
>
|
|
> ```bash
|
|
> make integration # ACL ↔ OpenZaak only (no NRC)
|
|
> make verify-notifications # OpenZaak → NRC delivery only
|
|
> ```
|
|
|
|
**Prerequisites:** .NET 10 SDK, a container engine with Compose v2, and `curl`.
|
|
|
|
On a **rootless Podman** box (the default dev setup here), the `smoke` target needs
|
|
the Podman API socket and a Compose provider:
|
|
|
|
```bash
|
|
systemctl --user enable --now podman.socket # start the API socket
|
|
ln -sf "$(command -v podman)" ~/.local/bin/docker # docker -> podman shim
|
|
# install Docker Compose v2 into ~/.local/bin as `docker-compose` (the provider)
|
|
```
|
|
|
|
The Makefile auto-points `DOCKER_HOST` at `/run/user/$(id -u)/podman/podman.sock`
|
|
when that socket exists and `DOCKER_HOST` is unset, so `make smoke` "just works"
|
|
locally while leaving real Docker hosts / CI runners untouched.
|
|
|
|
## Runner: `ubuntu-latest`
|
|
|
|
All jobs run on Gitea's hosted **`ubuntu-latest`** runner — no self-hosted runner
|
|
setup is required. The hosted runner ships with Docker and Docker Compose v2, so
|
|
`make smoke` (`docker compose … up --wait`) works without extra configuration.
|
|
|
|
If Gitea's hosted runners are unavailable and a self-hosted fallback is needed,
|
|
register an `act_runner` with the `ubuntu-latest` label:
|
|
|
|
```bash
|
|
VER=0.2.11
|
|
curl -fsSL -o /usr/local/bin/act_runner \
|
|
"https://dl.gitea.com/act_runner/${VER}/act_runner-${VER}-linux-amd64"
|
|
chmod +x /usr/local/bin/act_runner
|
|
|
|
act_runner register --no-interactive \
|
|
--instance https://git.labs.respellion.tech \
|
|
--token <REGISTRATION_TOKEN> \
|
|
--name respellion-ci-1 \
|
|
--labels "ubuntu-latest:docker://node:20-bookworm"
|
|
|
|
act_runner daemon
|
|
```
|