not
17f1f2f809
## What & why `docs/` is the source of truth (CLAUDE.md §12), but only pages listed in `mkdocs.yml`'s nav are published — and mkdocs' own `validation.nav.omitted_files: warn` keeps the build green while dropping the rest. So the site had quietly stopped at **ADR-0010** and **`runbooks/ci.md`**: 31 pages, including every ADR from 0011 to 0034, six of the seven runbooks, and `synthetic-data.md`, existed in the repo and nowhere else. - `infra/check-docs-nav.py` fails when a page under `docs/` is not in the nav. It runs in `make lint`, so the existing CI job gates it — python3 only, no new tooling, and no mkdocs install needed to check it. - The nav now lists all 34 ADRs, all 7 runbooks and `synthetic-data.md`. - ADR-0033's `Slice:` header said "none yet"; #25 closed it. - The landing page gained a pointer to the Talos runbook. Closes #169 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing test committed before the fix — the red commit lists all 31 missing pages. - [x] Implementation makes the test pass. - [x] Conventional Commits referencing the issue (`refs #169`). - [ ] CI green — awaiting the run on this PR (`python3 infra/check-docs-nav.py` passes locally; `make lint` also needs the .NET SDK, which CI has). - [x] `docker compose up` unaffected — docs and `mkdocs.yml` only, plus one `make lint` line. - [x] Docs updated — that is the change. - [x] No ADR needed: no dependency, no boundary, no §8 rule touched. - [x] Not user-visible, so no demo note. ## Notes for reviewers - The check is a **substring test**, not a YAML parse (there's a `ponytail:` note in the script): a page's path either appears in `mkdocs.yml` or it doesn't. That keeps it dependency-free — `mkdocs.yml` can't be read by `yaml.safe_load` anyway, it carries a `!!python/name:` tag for the mermaid fence. It does not check that an entry *points at a file that exists*; mkdocs' `not_found: warn` covers that direction. - ADR labels in the nav are shortened by hand (`"ADR-0013: Behandel-portal wiring"`), since several H1s are a full sentence. **Known gap, not fixed here:** CLAUDE.md §12 says the site is "published via a Gitea Actions workflow to Gitea Pages", and no such workflow exists — `mkdocs build` is never run, by CI or by any make target. Gitea has no built-in Pages, so publishing needs a decision (a `gitea-pages` server, an artifact, or a static host) rather than a patch. Worth its own issue if the published site is actually wanted; until then this PR makes the nav correct for whoever runs `mkdocs serve`.Reviewed-on: #172
162 lines
10 KiB
Markdown
162 lines
10 KiB
Markdown
# ADR-0033: Kubernetes deployment is one values-driven Helm chart, not a chart per service
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-09-04
|
|
- **Deciders:** Respellion engineering
|
|
- **Slice:** #25 (S-24) — raised directly as a deployment-target request and matched to
|
|
that issue afterwards; see the "Process note" at the end
|
|
|
|
## Context
|
|
|
|
The stack is defined once, in `infra/docker-compose.yml`: 30-odd containers made of six
|
|
upstream Common Ground modules (OpenZaak, Open Notificaties, Objecten, Objecttypen,
|
|
Keycloak, Flowable), their databases and workers, five .NET services, four portals, six
|
|
one-shot bootstrap containers, and an observability backplane (off by default here). Compose is the
|
|
CI-canonical stack: `make verify` and every `verify-*` script drive it.
|
|
|
|
We now also want the stack on Kubernetes — first target a **single-node Talos VM on a
|
|
laptop**. Four properties of this particular stack shape the answer:
|
|
|
|
- **The upstream images are used verbatim** and read their configuration from a mounted
|
|
directory (`setup_configuration/data.yaml`, Keycloak realm exports, BPMN/DMN). Compose
|
|
streams those files into external volumes (`infra/seed-config.sh`) because bind mounts
|
|
don't reach sibling containers on the CI runner. Kubernetes needs the same files as
|
|
ConfigMaps — from *somewhere*.
|
|
- **Django's `URLValidator` rejects single-label hosts.** Compose works around it by
|
|
handing the ACL and the seeds a container *IP* (ADR-0009, ADR-0020, ADR-0029, and the
|
|
`objecten.local` network alias). In Kubernetes a Service FQDN is already multi-label, so
|
|
the workaround has a natural replacement — but the hosts have to line up exactly, since
|
|
Objecten reflects the request Host into the URLs it publishes to NRC.
|
|
- **The OIDC issuer must be one string** for both the browser and the BFF (ADR-0010).
|
|
`infra/host-browser.yml` already solved this for a host browser: pin `KC_HOSTNAME`, keep
|
|
backchannel discovery in-cluster, and mount a `config.json` per portal.
|
|
- **Nothing here is highly available.** One replica of everything, on one node.
|
|
|
|
## Decision
|
|
|
|
**One chart — `infra/helm/big-reference` — whose `values.yaml` is a near-literal
|
|
transcription of the compose file, rendered by three generic templates (Deployment, Job,
|
|
Service) over a `workloads` map.** Adding a service is a values edit.
|
|
|
|
Consequences of that shape, each chosen deliberately:
|
|
|
|
- **Config files are not copied into the chart.** `infra/helm/seed-configmaps.sh` creates
|
|
the ConfigMaps from the files that already live in the repo — the Kubernetes sibling of
|
|
`infra/seed-config.sh`. The chart therefore needs `make k8s-seed` before `helm install`,
|
|
which is the same two-step dance compose already has.
|
|
- **Bootstrap one-shots become Jobs, with no ordering mechanism.** Every one is idempotent
|
|
(ADR-0020); each waits for the TCP ports it needs via a busybox init container and
|
|
Kubernetes retries the rest. `make k8s-reseed` re-runs them.
|
|
- **The four Django services apply their own `setup_configuration`** —
|
|
`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]` — instead of getting a
|
|
separate `*-init` Job like compose. Both of those image scripts run
|
|
`manage.py migrate`, and compose serialises them with
|
|
`depends_on: service_completed_successfully`; Kubernetes has no such edge, so a Job and
|
|
its web pod migrate the same database concurrently and Django dies with
|
|
*"relation zgw_consumers_service already exists"*. Running the two steps in order inside
|
|
the one container leaves exactly one migrator per database, and deletes four workloads.
|
|
- **`args`, never `command`.** Compose's `command:` replaces the image's CMD; Kubernetes'
|
|
`command:` replaces its ENTRYPOINT. Transcribing one to the other silently broke every
|
|
upstream image that relies on its entrypoint — postgres ran as root and refused to
|
|
start, Keycloak tried to exec `start-dev` as a binary. The chart now `fail`s at render
|
|
time if a workload sets `command`, because the symptom (a crashloop three layers down)
|
|
is nothing like the cause.
|
|
- **Published ports are NodePorts.** No ingress controller, no LoadBalancer, no TLS. The
|
|
four portals are the exception in *use*, not in wiring: PKCE needs `crypto.subtle`, which
|
|
browsers expose only in a secure context, so a portal has to be reached over `localhost`
|
|
(`make k8s-portals` forwards them) or eventually over HTTPS. `.Values.host` is therefore
|
|
"the address the browser uses", not "the node's address" — it pins Keycloak's issuer and
|
|
each portal's `config.json`, and both must agree with the URL bar (ADR-0010).
|
|
- **Databases are `emptyDir` by default**, so the stack comes up on a cluster with no CSI
|
|
driver; setting `persistence.storageClass` switches every database to a PVC.
|
|
- **Only two hosts become FQDNs** — OpenZaak (for the ACL and the zaaktype seed) and
|
|
Objecten (for the ACL's register writes), the two that Django validates as URLs.
|
|
Everything else keeps the short compose service name, because the upstream
|
|
`setup_configuration` files name those and Objecten matches an objecttype URL against the
|
|
one it was configured with. The portals used to be a third case — nginx's `resolver` never
|
|
appends search domains, so the bare `bff` upstream could not resolve on Kubernetes — which
|
|
ADR-0034 removed by serving them with Caddy, whose resolver honours `/etc/resolv.conf`.
|
|
- **Compose stays CI-canonical.** The chart is a second deployment target, not a
|
|
replacement; the acceptance, verify and e2e lanes are unchanged.
|
|
|
|
### Alternatives considered
|
|
|
|
- **A chart per service, or an umbrella of 30 subcharts.** The conventional layout, and
|
|
roughly 1,500 lines of near-identical YAML for a stack where 28 of 30 workloads are
|
|
"one pod, one image, some env". It buys independent versioning we don't want (the stack
|
|
is demoed as a whole) and costs the eye-diffability against the compose file that keeps
|
|
the two stacks honest.
|
|
- **`kompose convert`.** One-shot generation, no ongoing artefact to maintain — but it
|
|
drops exactly the parts that carry the design (init ordering, the config volumes, the
|
|
issuer pinning) and produces output nobody owns.
|
|
- **Bitnami PostgreSQL/Redis subcharts.** Six more dependencies (CLAUDE.md §13) and a
|
|
second way of expressing the same three-line database.
|
|
- **ingress-nginx with hostname routing.** Needs a controller, `/etc/hosts` entries and a
|
|
matching issuer host; NodePorts need none of it and reuse the mechanism
|
|
`infra/host-browser.yml` already proves.
|
|
- **A registry on the laptop** (the obvious home for images built there). Talos cannot
|
|
side-load an image, so a registry is required either way — but reaching one on the host
|
|
means opening an inbound port on firewalld's `libvirt` zone, which needs root, and
|
|
pushing to it over plain HTTP means an `insecure-registries` entry in the Docker daemon,
|
|
which needs root again. `infra/helm/registry.yaml` runs the registry *in* the cluster on
|
|
a NodePort instead: pushing laptop → node is outbound and unfiltered, the node pulls from
|
|
its own NodePort, and `docker save | crane push --insecure` needs no daemon
|
|
configuration. Cost: one more (throwaway, `emptyDir`) workload, and a re-push if its pod
|
|
is replaced.
|
|
- **Helm hooks (`pre-install`/`post-install`) for bootstrap ordering.** Hooks run after
|
|
`--wait`, which would deadlock: OpenZaak's readiness needs the migrations that the hook
|
|
is supposed to run. Idempotent Jobs plus retries need no such sequencing.
|
|
|
|
- ponytail ceiling: single-node assumptions are baked in — one replica per workload,
|
|
`Recreate` rollouts, ReadWriteOnce volumes, no PodDisruptionBudgets, no resource
|
|
requests or limits (a laptop VM schedules everything or nothing), plain HTTP.
|
|
Upgrade path for a real cluster: add requests/limits per workload (the field is already
|
|
passed through), swap NodePorts for an Ingress with TLS, and give the databases a real
|
|
StorageClass — none of which changes the workload graph.
|
|
|
|
## Consequences
|
|
|
|
**Positive**
|
|
|
|
- One file to read to see what the cluster runs, and it lines up with the compose file
|
|
line for line.
|
|
- The compose IP workarounds disappear: cluster DNS supplies multi-label hosts.
|
|
- `make k8s-lint` renders and schema-checks the whole stack without a cluster.
|
|
- The config inputs have exactly one home (the repo) for both stacks — no fork to drift.
|
|
|
|
**Negative / costs**
|
|
|
|
- A second deployment description to keep in step with compose. Nothing enforces that
|
|
today; a drift check belongs in CI (follow-up).
|
|
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
|
|
one surfaces as `ContainerCreating`, not as a clear error.
|
|
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
|
|
schema errors, not intent.
|
|
- The verify/e2e lanes do not run against the chart, so the Kubernetes path is verified by
|
|
hand (docs/runbooks/kubernetes-talos.md §5) rather than by CI.
|
|
- The chart deviates from compose in four places now (args, self-configuring Django pods,
|
|
FQDN hosts, NodePorts). Each is forced by the platform and commented where it appears,
|
|
but it is four more things that can drift.
|
|
|
|
## Coupling rules touched (CLAUDE.md §8)
|
|
|
|
None. The chart deploys the same graph: portals reach only the BFF (§8.3), only the ACL
|
|
holds ZGW credentials (§8.1), only the Workflow Client talks to Flowable (§8.2), each
|
|
service keeps its own database (§8.5). No workload gained a peer it didn't have in compose.
|
|
|
|
## Verified
|
|
|
|
Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
|
|
under virt-manager: 29 pods ready and four bootstrap Jobs complete in under three minutes,
|
|
with zero restarts, using ~4.4 GB of the VM's 10 GB. The smoke test in the runbook's §5
|
|
walks the whole path — portal proxy → BFF → domain → Flowable → ACL → OpenZaak + Objecten →
|
|
NRC → event-subscriber → projection → public register — plus a werkbak read with an
|
|
MFA'd medewerker token. The browser flow itself was driven with Playwright against
|
|
`http://localhost:30140`: secure context, PKCE, Keycloak form, login, no console errors.
|
|
|
|
## Process note
|
|
|
|
CLAUDE.md §14 wants the ADR proposal issue opened before the code, and §7 wants a slice
|
|
issue behind the work. This landed the other way round — chart first, on request. The
|
|
issue and the CI drift check are the outstanding follow-ups.
|