not
70de3d0a4d
CI / lint (pull_request) Successful in 1m20s
CI / k8s (pull_request) Successful in 8s
CI / build (pull_request) Successful in 1m5s
CI / unit (pull_request) Successful in 1m33s
CI / frontend (pull_request) Successful in 3m9s
CI / mutation (pull_request) Successful in 6m53s
CI / verify-stack (pull_request) Successful in 9m32s
`make k8s-lint` existed since the chart landed but nothing ran it, so the chart had no automated coverage at all. A `k8s` job now runs it plus `make k8s-drift` on every push and PR: no cluster, ~20s, and it catches the two failure modes the chart is actually exposed to — a values typo that renders invalid YAML, and a change made to one stack but not the other. helm is installed as its pinned static binary (the URL the Talos runbook already gives developers) rather than via a marketplace action: nothing extra to vet. The `k8s` targets stay out of `make ci` on purpose — helm is optional for anyone not deploying to Kubernetes — which is the one place local and CI now differ, noted in docs/runbooks/ci.md.
163 lines
10 KiB
Markdown
163 lines
10 KiB
Markdown
# ADR-0033: Kubernetes deployment is one values-driven Helm chart, not a chart per service
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-09-04
|
|
- **Deciders:** Respellion engineering
|
|
- **Slice:** _(none yet — raised directly as a deployment-target request; see
|
|
"Process note" at the end)_
|
|
|
|
## Context
|
|
|
|
The stack is defined once, in `infra/docker-compose.yml`: 30-odd containers made of six
|
|
upstream Common Ground modules (OpenZaak, Open Notificaties, Objecten, Objecttypen,
|
|
Keycloak, Flowable), their databases and workers, five .NET services, four portals, six
|
|
one-shot bootstrap containers, and an observability backplane (off by default here). Compose is the
|
|
CI-canonical stack: `make verify` and every `verify-*` script drive it.
|
|
|
|
We now also want the stack on Kubernetes — first target a **single-node Talos VM on a
|
|
laptop**. Four properties of this particular stack shape the answer:
|
|
|
|
- **The upstream images are used verbatim** and read their configuration from a mounted
|
|
directory (`setup_configuration/data.yaml`, Keycloak realm exports, BPMN/DMN). Compose
|
|
streams those files into external volumes (`infra/seed-config.sh`) because bind mounts
|
|
don't reach sibling containers on the CI runner. Kubernetes needs the same files as
|
|
ConfigMaps — from *somewhere*.
|
|
- **Django's `URLValidator` rejects single-label hosts.** Compose works around it by
|
|
handing the ACL and the seeds a container *IP* (ADR-0009, ADR-0020, ADR-0029, and the
|
|
`objecten.local` network alias). In Kubernetes a Service FQDN is already multi-label, so
|
|
the workaround has a natural replacement — but the hosts have to line up exactly, since
|
|
Objecten reflects the request Host into the URLs it publishes to NRC.
|
|
- **The OIDC issuer must be one string** for both the browser and the BFF (ADR-0010).
|
|
`infra/host-browser.yml` already solved this for a host browser: pin `KC_HOSTNAME`, keep
|
|
backchannel discovery in-cluster, and mount a `config.json` per portal.
|
|
- **Nothing here is highly available.** One replica of everything, on one node.
|
|
|
|
## Decision
|
|
|
|
**One chart — `infra/helm/big-reference` — whose `values.yaml` is a near-literal
|
|
transcription of the compose file, rendered by three generic templates (Deployment, Job,
|
|
Service) over a `workloads` map.** Adding a service is a values edit.
|
|
|
|
Consequences of that shape, each chosen deliberately:
|
|
|
|
- **Config files are not copied into the chart.** `infra/helm/seed-configmaps.sh` creates
|
|
the ConfigMaps from the files that already live in the repo — the Kubernetes sibling of
|
|
`infra/seed-config.sh`. The chart therefore needs `make k8s-seed` before `helm install`,
|
|
which is the same two-step dance compose already has.
|
|
- **Bootstrap one-shots become Jobs, with no ordering mechanism.** Every one is idempotent
|
|
(ADR-0020); each waits for the TCP ports it needs via a busybox init container and
|
|
Kubernetes retries the rest. `make k8s-reseed` re-runs them.
|
|
- **The four Django services apply their own `setup_configuration`** —
|
|
`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]` — instead of getting a
|
|
separate `*-init` Job like compose. Both of those image scripts run
|
|
`manage.py migrate`, and compose serialises them with
|
|
`depends_on: service_completed_successfully`; Kubernetes has no such edge, so a Job and
|
|
its web pod migrate the same database concurrently and Django dies with
|
|
*"relation zgw_consumers_service already exists"*. Running the two steps in order inside
|
|
the one container leaves exactly one migrator per database, and deletes four workloads.
|
|
- **`args`, never `command`.** Compose's `command:` replaces the image's CMD; Kubernetes'
|
|
`command:` replaces its ENTRYPOINT. Transcribing one to the other silently broke every
|
|
upstream image that relies on its entrypoint — postgres ran as root and refused to
|
|
start, Keycloak tried to exec `start-dev` as a binary. The chart now `fail`s at render
|
|
time if a workload sets `command`, because the symptom (a crashloop three layers down)
|
|
is nothing like the cause.
|
|
- **Published ports are NodePorts.** No ingress controller, no LoadBalancer, no TLS. The
|
|
four portals are the exception in *use*, not in wiring: PKCE needs `crypto.subtle`, which
|
|
browsers expose only in a secure context, so a portal has to be reached over `localhost`
|
|
(`make k8s-portals` forwards them) or eventually over HTTPS. `.Values.host` is therefore
|
|
"the address the browser uses", not "the node's address" — it pins Keycloak's issuer and
|
|
each portal's `config.json`, and both must agree with the URL bar (ADR-0010).
|
|
- **Databases are `emptyDir` by default**, so the stack comes up on a cluster with no CSI
|
|
driver; setting `persistence.storageClass` switches every database to a PVC.
|
|
- **Only two hosts become FQDNs** — OpenZaak (for the ACL and the zaaktype seed) and
|
|
Objecten (for the ACL's register writes), the two that Django validates as URLs.
|
|
Everything else keeps the short compose service name, because the upstream
|
|
`setup_configuration` files name those and Objecten matches an objecttype URL against the
|
|
one it was configured with. The portals used to be a third case — nginx's `resolver` never
|
|
appends search domains, so the bare `bff` upstream could not resolve on Kubernetes — which
|
|
ADR-0034 removed by serving them with Caddy, whose resolver honours `/etc/resolv.conf`.
|
|
- **Compose stays CI-canonical.** The chart is a second deployment target, not a
|
|
replacement; the acceptance, verify and e2e lanes are unchanged.
|
|
|
|
### Alternatives considered
|
|
|
|
- **A chart per service, or an umbrella of 30 subcharts.** The conventional layout, and
|
|
roughly 1,500 lines of near-identical YAML for a stack where 28 of 30 workloads are
|
|
"one pod, one image, some env". It buys independent versioning we don't want (the stack
|
|
is demoed as a whole) and costs the eye-diffability against the compose file that keeps
|
|
the two stacks honest.
|
|
- **`kompose convert`.** One-shot generation, no ongoing artefact to maintain — but it
|
|
drops exactly the parts that carry the design (init ordering, the config volumes, the
|
|
issuer pinning) and produces output nobody owns.
|
|
- **Bitnami PostgreSQL/Redis subcharts.** Six more dependencies (CLAUDE.md §13) and a
|
|
second way of expressing the same three-line database.
|
|
- **ingress-nginx with hostname routing.** Needs a controller, `/etc/hosts` entries and a
|
|
matching issuer host; NodePorts need none of it and reuse the mechanism
|
|
`infra/host-browser.yml` already proves.
|
|
- **A registry on the laptop** (the obvious home for images built there). Talos cannot
|
|
side-load an image, so a registry is required either way — but reaching one on the host
|
|
means opening an inbound port on firewalld's `libvirt` zone, which needs root, and
|
|
pushing to it over plain HTTP means an `insecure-registries` entry in the Docker daemon,
|
|
which needs root again. `infra/helm/registry.yaml` runs the registry *in* the cluster on
|
|
a NodePort instead: pushing laptop → node is outbound and unfiltered, the node pulls from
|
|
its own NodePort, and `docker save | crane push --insecure` needs no daemon
|
|
configuration. Cost: one more (throwaway, `emptyDir`) workload, and a re-push if its pod
|
|
is replaced.
|
|
- **Helm hooks (`pre-install`/`post-install`) for bootstrap ordering.** Hooks run after
|
|
`--wait`, which would deadlock: OpenZaak's readiness needs the migrations that the hook
|
|
is supposed to run. Idempotent Jobs plus retries need no such sequencing.
|
|
|
|
- ponytail ceiling: single-node assumptions are baked in — one replica per workload,
|
|
`Recreate` rollouts, ReadWriteOnce volumes, no PodDisruptionBudgets, no resource
|
|
requests or limits (a laptop VM schedules everything or nothing), plain HTTP.
|
|
Upgrade path for a real cluster: add requests/limits per workload (the field is already
|
|
passed through), swap NodePorts for an Ingress with TLS, and give the databases a real
|
|
StorageClass — none of which changes the workload graph.
|
|
|
|
## Consequences
|
|
|
|
**Positive**
|
|
|
|
- One file to read to see what the cluster runs, and it lines up with the compose file
|
|
line for line.
|
|
- The compose IP workarounds disappear: cluster DNS supplies multi-label hosts.
|
|
- `make k8s-lint` renders and schema-checks the whole stack without a cluster.
|
|
- The config inputs have exactly one home (the repo) for both stacks — no fork to drift.
|
|
|
|
**Negative / costs**
|
|
|
|
- A second deployment description to keep in step with compose. `make k8s-drift` (#168)
|
|
now enforces the part that bites — the workload set and the resolved images, with the
|
|
four deviations below declared — but not per-workload env, ports or volumes.
|
|
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
|
|
one surfaces as `ContainerCreating`, not as a clear error.
|
|
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
|
|
schema errors, not intent.
|
|
- The verify/e2e lanes do not run against the chart, so the Kubernetes path is verified by
|
|
hand (docs/runbooks/kubernetes-talos.md §5) rather than by CI.
|
|
- The chart deviates from compose in four places now (args, self-configuring Django pods,
|
|
FQDN hosts, NodePorts). Each is forced by the platform and commented where it appears,
|
|
but it is four more things that can drift.
|
|
|
|
## Coupling rules touched (CLAUDE.md §8)
|
|
|
|
None. The chart deploys the same graph: portals reach only the BFF (§8.3), only the ACL
|
|
holds ZGW credentials (§8.1), only the Workflow Client talks to Flowable (§8.2), each
|
|
service keeps its own database (§8.5). No workload gained a peer it didn't have in compose.
|
|
|
|
## Verified
|
|
|
|
Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
|
|
under virt-manager: 29 pods ready and four bootstrap Jobs complete in under three minutes,
|
|
with zero restarts, using ~4.4 GB of the VM's 10 GB. The smoke test in the runbook's §5
|
|
walks the whole path — portal proxy → BFF → domain → Flowable → ACL → OpenZaak + Objecten →
|
|
NRC → event-subscriber → projection → public register — plus a werkbak read with an
|
|
MFA'd medewerker token. The browser flow itself was driven with Playwright against
|
|
`http://localhost:30140`: secure context, PKCE, Keycloak form, login, no console errors.
|
|
|
|
## Process note
|
|
|
|
CLAUDE.md §14 wants the ADR proposal issue opened before the code, and §7 wants a slice
|
|
issue behind the work. This landed the other way round — chart first, on request. The
|
|
issue and the CI drift check are the outstanding follow-ups.
|