Compare commits

..
Author SHA1 Message Date
not 70de3d0a4d ci(k8s): gate the Helm chart on every PR (refs #168)
CI / lint (pull_request) Successful in 1m20s
CI / k8s (pull_request) Successful in 8s
CI / build (pull_request) Successful in 1m5s
CI / unit (pull_request) Successful in 1m33s
CI / frontend (pull_request) Successful in 3m9s
CI / mutation (pull_request) Successful in 6m53s
CI / verify-stack (pull_request) Successful in 9m32s
`make k8s-lint` existed since the chart landed but nothing ran it, so the chart had
no automated coverage at all. A `k8s` job now runs it plus `make k8s-drift` on every
push and PR: no cluster, ~20s, and it catches the two failure modes the chart is
actually exposed to — a values typo that renders invalid YAML, and a change made to
one stack but not the other.

helm is installed as its pinned static binary (the URL the Talos runbook already
gives developers) rather than via a marketplace action: nothing extra to vet.

The `k8s` targets stay out of `make ci` on purpose — helm is optional for anyone not
deploying to Kubernetes — which is the one place local and CI now differ, noted in
docs/runbooks/ci.md.
2026-09-10 11:01:29 +02:00
not fddf14e5f9 feat(k8s): declare the six platform deviations so only new drift fails (refs #168)
Six differences exist by design and are now recorded in DEVIATIONS with the reason
each one was forced, so the check passes on today's tree and fails on tomorrow's
accident: the four `*-init` Django services folded into their web pods (one migrator
per database), and the two bootstrap Jobs compose runs from the host instead.

Verified against both drift classes by hand: bumping OPENZAAK_TAG in compose alone
reports openzaak + oz-celery, and adding a workload to values.yaml alone reports it
by name.
2026-09-10 11:00:32 +02:00
not 5000b749b9 test(k8s): fail when compose and the Helm chart describe different stacks (refs #168)
ADR-0033 shipped the chart with this cost written down: "a second deployment
description to keep in step with compose. Nothing enforces that today; a drift
check belongs in CI (follow-up)." An upstream image bump or a new service applied
to only one of the two files lands unnoticed.

`make k8s-drift` compares what each stack actually deploys — workload names and
resolved container images, from `docker compose config` and a rendered chart —
rather than diffing the two files, which differ by design.

Red: it reports the six differences that exist today, all of them the platform
deviations ADR-0033 forced. Declaring those as intended is the green step.
2026-09-10 10:59:53 +02:00
9 changed files with 160 additions and 74 deletions
+21
View File
@@ -41,6 +41,27 @@ jobs:
nuget-${{ runner.os }}-
- run: make lint
# The Helm chart's only automated gate: it renders and schema-checks the whole
# stack, and checks it still describes the same stack as the compose file
# (ADR-0033). No cluster involved — see docs/runbooks/kubernetes-talos.md.
k8s:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
# helm as its pinned static binary rather than a marketplace action: one URL,
# the same one the Talos runbook §0 gives a developer, and no third-party
# action to vet (CLAUDE.md §13). The drift check also needs `docker compose`,
# which the runner already has (see docs/runbooks/ci.md).
- name: Install helm
run: |
mkdir -p "$HOME/.local/bin"
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz \
| tar xz -O linux-amd64/helm > "$HOME/.local/bin/helm"
chmod +x "$HOME/.local/bin/helm"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- run: make k8s-lint
- run: make k8s-drift
build:
runs-on: ubuntu-latest
steps:
+8 -4
View File
@@ -43,7 +43,7 @@ export DOCKER_HOST := unix://$(PODMAN_SOCK)
endif
endif
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-drift k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions)
## `verify` is the live-stack stage (full stack up once → ACL + notification checks).
@@ -64,9 +64,6 @@ frontend:
## lint: verify formatting (no changes)
lint:
dotnet format $(SLN) --verify-no-changes
# Only pages in mkdocs.yml's nav are published, and mkdocs keeps a build green
# when one is missing — so the nav is checked here rather than not at all.
python3 infra/check-docs-nav.py
## build: release build
build:
@@ -353,6 +350,13 @@ k8s-lint:
helm lint $(K8S_CHART)
helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null
## k8s-drift: fail if compose and the Helm chart describe different stacks
# Compose is CI-canonical (ADR-0033) and the chart is a transcription of it; this
# compares what each one deploys — workload names and resolved images. Needs
# `docker compose` and `helm`, no cluster.
k8s-drift:
python3 infra/helm/check-drift.py
## k8s-registry: deploy the in-cluster image registry (NodePort 30500)
k8s-registry:
kubectl apply -f infra/helm/registry.yaml
@@ -3,8 +3,8 @@
- **Status:** Accepted
- **Date:** 2026-09-04
- **Deciders:** Respellion engineering
- **Slice:** #25 (S-24) — raised directly as a deployment-target request and matched to
that issue afterwards; see the "Process note" at the end
- **Slice:** _(none yet — raised directly as a deployment-target request; see
"Process note" at the end)_
## Context
@@ -126,8 +126,9 @@ Consequences of that shape, each chosen deliberately:
**Negative / costs**
- A second deployment description to keep in step with compose. Nothing enforces that
today; a drift check belongs in CI (follow-up).
- A second deployment description to keep in step with compose. `make k8s-drift` (#168)
now enforces the part that bites — the workload set and the resolved images, with the
four deviations below declared — but not per-workload env, ports or volumes.
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
one surfaces as `ContainerCreating`, not as a clear error.
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
-2
View File
@@ -14,8 +14,6 @@ should teach.
In Dutch; the strategic framing lives in `Respellion/innovation-lab`.
- **[Working in Gitea](gitea-workflow.md)** — issues, milestones, branches, PRs.
- **[CI runbook](runbooks/ci.md)** — the pipeline and the `make ci` local gate.
- **[Kubernetes on Talos](runbooks/kubernetes-talos.md)** — the second deployment target:
one Helm chart, a single-node cluster, and the parts that bite (ADR-0033).
## Quickstart
+6 -2
View File
@@ -2,8 +2,10 @@
> **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's
> hosted `ubuntu-latest` runner — no self-hosted runner required.
> **`make ci` is still the local gate** — it runs the exact same checks
> (the workflow calls the same `make` targets).
> **`make ci` is still the local gate** — it runs the same checks via the same
> `make` targets, with one exception: the `k8s` job's targets are not in `make ci`,
> because `helm` is optional for everyone not deploying to Kubernetes. Run
> `make k8s-lint k8s-drift` by hand after touching the chart or the compose file.
## The pipeline
@@ -16,6 +18,8 @@ and CI cannot drift:
| `lint` | `make lint``dotnet format … --verify-no-changes` | .NET 10 SDK |
| `build` | `make build``dotnet build … -c Release` | .NET 10 SDK |
| `unit` | `make unit``dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK |
| `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
| `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
| `mutation` | `make mutation``dotnet tool restore``dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
+4 -1
View File
@@ -322,6 +322,7 @@ The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the da
```bash
make k8s-lint # render + schema-check the chart, no cluster needed
make k8s-drift # fail if compose and the chart describe different stacks
make k8s-portals # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=... # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
@@ -366,5 +367,7 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
`K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
chart. The Kubernetes path is verified with §5's smoke test.
chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
image bump that skipped the chart fails there, but nothing deploys the chart in CI.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/env python3
"""Fail when a page under docs/ is missing from mkdocs.yml's nav.
docs/ is the source of truth (CLAUDE.md §12), but only the pages listed in the nav
are published — and mkdocs' own `omitted_files: warn` keeps a build green while
silently dropping them, which is how every ADR after 0010 and every runbook but
ci.md fell off the site.
ponytail: a substring test, not a YAML parse — a page's path either appears in
mkdocs.yml or it doesn't, and that needs no dependency.
"""
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
nav = (ROOT / "mkdocs.yml").read_text()
missing = sorted(
str(page.relative_to(ROOT / "docs"))
for page in (ROOT / "docs").rglob("*.md")
if str(page.relative_to(ROOT / "docs")) not in nav
)
if missing:
print(f"{len(missing)} page(s) under docs/ are not in mkdocs.yml's nav:")
print("\n".join(f" {m}" for m in missing))
sys.exit(1)
print("docs nav complete: every page under docs/ is published")
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Fail when the compose stack and the Helm chart stop describing the same stack.
`infra/docker-compose.yml` is CI-canonical; `infra/helm/big-reference` is a
transcription of it (ADR-0033), and until now nothing kept the two in step — an
upstream image bump or a new service applied to only one of them landed
unnoticed. This compares what each side actually *deploys*, not the two files:
the rendered chart against `docker compose config`. Both tools are already
prerequisites of the `k8s-*` make targets.
Run it with `make k8s-drift`. No cluster needed.
ponytail: names and images only, as sets — no per-workload env/ports/volumes.
Those differ by design in four documented places (ADR-0033), so comparing them
would mean re-encoding every deviation field by field; a tag bump and a missing
service are the drift that actually bites.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
COMPOSE = ROOT / "infra/docker-compose.yml"
CHART = ROOT / "infra/helm/big-reference"
# The busybox init container that every `waitFor` workload gets exists only in
# the chart (compose has `depends_on`). Rendering it under a sentinel makes it
# filterable without teaching the check what busybox is.
BUSYBOX = "drift-check-ignored-init-image"
# Differences that Kubernetes forces, not drift (ADR-0033). A name listed here is
# expected to be on exactly one side; anything else fails.
DEVIATIONS = {
# The four Django services apply their own setup_configuration in the web pod
# (`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]`) rather than in a
# separate init Job. Both that script and /start.sh run `manage.py migrate`, and
# Kubernetes has no `depends_on: service_completed_successfully` to serialise them,
# so the Job and its web pod migrated the same database concurrently.
"oz-init": "folded into the openzaak pod",
"nrc-init": "folded into the nrc-web pod",
"objecttypen-init": "folded into the objecttypen pod",
"objecten-init": "folded into the objecten pod",
# Compose seeds these from the host — the verify scripts `docker cp` the two
# scripts into a running container, and docker-compose.local.yml carries
# `local-seed` + `nrc-subscribe` for `make local`. A cluster has no host to seed
# from, so both became Jobs in the chart.
"seed-zaaktype": "compose seeds the catalogus from the host (infra/openzaak/seed_catalogus.py)",
"nrc-subscribe": "compose registers the abonnement from the host (infra/local/register-abonnement.py)",
}
# Workloads the observability backplane adds. Off by default in both stacks'
# defaults, so they are rendered on purpose here — otherwise their images drift
# unwatched.
OBSERVABILITY = ["tempo", "prometheus", "grafana"]
def compose_services() -> dict[str, str]:
"""Service name -> image, with ${TAG:-default} interpolation already applied."""
out = run(["docker", "compose", "-f", str(COMPOSE), "config", "--format", "json"])
return {name: svc.get("image", "") for name, svc in json.loads(out)["services"].items()}
def chart_workloads() -> dict[str, str]:
"""Workload name -> image, read back out of the rendered manifests."""
out = run(
["helm", "template", "big", str(CHART), "-n", "big", "--set", f"images.busybox={BUSYBOX}"]
+ [f"--set=workloads.{w}.enabled=true" for w in OBSERVABILITY]
)
workloads = {}
for doc in out.split("\n---"):
if not re.search(r"^kind: (Deployment|Job)$", doc, re.M):
continue
name = re.search(r"^ name: (\S+)$", doc, re.M)[1]
images = [i for i in re.findall(r"^\s+image: (\S+)$", doc, re.M) if i != BUSYBOX]
workloads[name] = images[0]
return workloads
def run(argv: list[str]) -> str:
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
sys.exit(f"{argv[0]} failed:\n{proc.stderr}")
return proc.stdout
def main() -> int:
compose, chart = compose_services(), chart_workloads()
problems = []
for name in sorted(set(compose) - set(chart) - set(DEVIATIONS)):
problems.append(f" {name}: in docker-compose.yml, not in the chart")
for name in sorted(set(chart) - set(compose) - set(DEVIATIONS)):
problems.append(f" {name}: in the chart, not in docker-compose.yml")
for name in sorted(set(compose) & set(chart)):
if compose[name] != chart[name]:
problems.append(f" {name}: compose runs {compose[name]}, the chart runs {chart[name]}")
if problems:
print("compose and the Helm chart describe different stacks:\n" + "\n".join(problems))
print(
"\nPort the change to the other stack, or — if the difference is forced by\n"
"Kubernetes — declare it in DEVIATIONS in this file, with the reason."
)
return 1
print(f"no drift: {len(chart)} workloads, images identical on both stacks")
for name, why in sorted(DEVIATIONS.items()):
print(f" deviation (declared): {name}{why}")
return 0
if __name__ == "__main__":
sys.exit(main())
-31
View File
@@ -32,30 +32,6 @@ nav:
- "ADR-0008: Read projection store": architecture/adr-0008-read-projection-store.md
- "ADR-0009: External-task job worker": architecture/adr-0009-external-task-job-worker.md
- "ADR-0010: BFF OIDC validation": architecture/adr-0010-bff-oidc.md
- "ADR-0011: Approval status flow": architecture/adr-0011-approval-status-flow.md
- "ADR-0012: Citizen reference correlation": architecture/adr-0012-citizen-reference-correlation.md
- "ADR-0013: Behandel-portal wiring": architecture/adr-0013-behandel-portal-wiring.md
- "ADR-0014: Withdrawal cancels the process": architecture/adr-0014-withdrawal-cancels-the-process.md
- "ADR-0015: Beoordeling escalation": architecture/adr-0015-beoordeling-escalation.md
- "ADR-0016: Diploma eligibility DMN": architecture/adr-0016-diploma-eligibility-dmn.md
- "ADR-0017: Document-wait timeout": architecture/adr-0017-document-wait-timeout-cancellation.md
- "ADR-0018: Diploma upload via the ACL": architecture/adr-0018-diploma-upload-via-acl-documenten.md
- "ADR-0019: Zaak cancellation on timeout": architecture/adr-0019-zaak-cancellation-on-timeout.md
- "ADR-0020: Local stack self-seeds": architecture/adr-0020-local-stack-self-seeds.md
- "ADR-0021: Zaaktype by identificatie": architecture/adr-0021-acl-resolves-zaaktype-by-identificatie.md
- "ADR-0022: Quartz scheduler": architecture/adr-0022-quartz-scheduler.md
- "ADR-0023: Observability stack": architecture/adr-0023-observability-stack.md
- "ADR-0024: Prometheus AspNetCore exporter": architecture/adr-0024-prometheus-aspnetcore-exporter.md
- "ADR-0025: BFF reads catalogus via the ACL": architecture/adr-0025-bff-reads-catalogus-via-acl.md
- "ADR-0026: Mutable default-fill store": architecture/adr-0026-mutable-default-fill-store.md
- "ADR-0027: RegisterRecord objecttype": architecture/adr-0027-registerrecord-objecttype-schema.md
- "ADR-0028: Objecten holds the register": architecture/adr-0028-objecten-holds-the-register.md
- "ADR-0029: Objecten publishes to NRC": architecture/adr-0029-objecten-publishes-to-nrc.md
- "ADR-0030: Projection sourced from the register": architecture/adr-0030-projection-sourced-from-the-register.md
- "ADR-0031: MFA on the medewerker realm": architecture/adr-0031-mfa-on-the-medewerker-realm.md
- "ADR-0032: Werkbak live refresh": architecture/adr-0032-werkbak-live-refresh.md
- "ADR-0033: Kubernetes via one Helm chart": architecture/adr-0033-kubernetes-via-one-helm-chart.md
- "ADR-0034: Caddy serves the portals": architecture/adr-0034-caddy-serves-the-portals.md
- FDS-architectuur:
- Overzicht: architecture/fds/README.md
- Componentview (L3): architecture/fds/c4-component-view.md
@@ -70,15 +46,8 @@ nav:
- Working in Gitea: gitea-workflow.md
- Frontend decisions: frontend-decisions.md
- Demo script: demo-script.md
- Synthetic data: synthetic-data.md
- Runbooks:
- CI: runbooks/ci.md
- OpenZaak: runbooks/openzaak.md
- Open Notificaties (NRC): runbooks/opennotificaties.md
- Keycloak: runbooks/keycloak.md
- Flowable: runbooks/flowable.md
- Kubernetes on Talos: runbooks/kubernetes-talos.md
- Gitea Actions gotchas: runbooks/gitea-actions-gotchas.md
markdown_extensions:
- admonition