eho/register-referentie
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

S-24/#25 · Helm chart + Kubernetes deployment, and Caddy for the portals (#166) (#167)
CI / lint (push) Successful in 1m17s
Details
CI / build (push) Successful in 1m12s
Details
CI / unit (push) Successful in 1m26s
Details
CI / frontend (push) Successful in 2m58s
Details
CI / mutation (push) Successful in 9m1s
Details
CI / verify-stack (push) Successful in 8m53s
Details

Browse Source 
## What & why

Two changes, made and verified together on a real cluster.

**S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`,
whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered
by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a
service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without
a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see
below), which is more than it asked for — the part it asked for that is **not** here is the
production-posture write-up (HA, secrets, backup), see Known gaps.

**#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass`
upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search
domains. That had cost two workarounds in one script: rewriting the resolver address for
rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on
Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy
bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's
`BFF_HOST` env are deleted.

Closes #25
Closes #166

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — twice: the Caddyfile contract test
      before the Caddyfiles, `make k8s-lint` before the chart.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`).
- [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new
      unit self-check pass locally; the compose e2e and mutation lanes are CI's).
- [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes —
      the portal images were rebuilt and verified standalone, but a full `make up` run has
      not been done on this branch. Please confirm in review or let CI's smoke test speak.
- [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`,
      `demo-script.md`, and the docs that named nginx.
- [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy).
- [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a
      user-visible slice, and the Caddy swap is invisible to the demo script beyond the
      wording fix included here.

## How it was verified

Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three
minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB.

- Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak +
  Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference
  matching the submitted registration).
- Werkbak read with an MFA'd medewerker token → 200.
- The browser flow driven with Playwright against `http://localhost:30140`: secure context,
  `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**.
- Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies
  its own groups, and a portal does *not* proxy a neighbour's group.

## Notes for reviewers

Three bugs this shook out, each fixed at the cause rather than the symptom:

1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes'
   replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that
   relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec
   `start-dev`. The chart now `fail`s at render time on `command`.
2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run
   `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such
   edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already
   exists`). The four Django services now do both steps in order in the web pod — which
   also deletes four workloads.
3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now
   also restarts `event-subscriber` and `projection-api`, which create the projection
   schema on start and otherwise keep writing to a schema-less database.

Known gaps / follow-ups:

- **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the
  ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine
  for a laptop demo, and exactly what #25's "production posture" ADR should address — I
  suggest a follow-up issue rather than stretching this PR.
- **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into
  `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay
  in step. Worth a small follow-up.
- **This is two slices in one PR.** They were built and verified together and the diff is
  entangled (the chart was written against Caddy from the start), so splitting now would
  mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather.
- **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear.
  One conflict, in the `unit:` target where both branches add a self-check line — resolved
  by keeping both. #161's `infra/host-browser.yml` arrived with
  `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the
  `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a
  path the images no longer have.Reviewed-on: #167
This commit was merged in pull request #167.
This commit is contained in:
not
2026-09-10 08:53:58 +00:00
parent d6b3f9764f
commit 1dd8bd4e1b
42 changed files with 1991 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/adr-0032-werkbak-live-refresh.md
+2 -2
View File
@@ -18,7 +18,7 @@ Three forces shape the mechanism:
- **Nothing notifies anyone.** The trigger lives in Flowable. The domain does not publish
task events, and there is no bus between the domain and the BFF.
- **The BFF is stateless** and sits behind each portal's nginx.
- **The BFF is stateless** and sits behind each portal's reverse proxy.
- **This is the repo's first live-updating view**, so the choice sets a precedent.
## Decision
@@ -40,7 +40,7 @@ werkbak is readable at all.
Neither buys freshness here, because **nothing notifies the BFF either**:
- **SSE** (`text/event-stream`) would mean a new streaming endpoint whose handler polls the
domain and forwards diffs — the same latency, plus connection lifecycle, nginx
domain and forwards diffs — the same latency, plus connection lifecycle, proxy
buffering, and auth on a long-lived connection.
- **WebSocket/SignalR** adds a dependency (CLAUDE.md §13) and makes the BFF stateful and
sticky-session-bound. A genuine push path would *also* need the domain to publish task
docs/architecture/adr-0033-kubernetes-via-one-helm-chart.md
+161
View File
@@ -0,0 +1,161 @@
# ADR-0033: Kubernetes deployment is one values-driven Helm chart, not a chart per service
- **Status:** Accepted
- **Date:** 2026-09-04
- **Deciders:** Respellion engineering
- **Slice:** _(none yet — raised directly as a deployment-target request; see
"Process note" at the end)_
## Context
The stack is defined once, in `infra/docker-compose.yml`: 30-odd containers made of six
upstream Common Ground modules (OpenZaak, Open Notificaties, Objecten, Objecttypen,
Keycloak, Flowable), their databases and workers, five .NET services, four portals, six
one-shot bootstrap containers, and an observability backplane (off by default here). Compose is the
CI-canonical stack: `make verify` and every `verify-*` script drive it.
We now also want the stack on Kubernetes — first target a **single-node Talos VM on a
laptop**. Four properties of this particular stack shape the answer:
- **The upstream images are used verbatim** and read their configuration from a mounted
directory (`setup_configuration/data.yaml`, Keycloak realm exports, BPMN/DMN). Compose
streams those files into external volumes (`infra/seed-config.sh`) because bind mounts
don't reach sibling containers on the CI runner. Kubernetes needs the same files as
ConfigMaps — from *somewhere*.
- **Django's `URLValidator` rejects single-label hosts.** Compose works around it by
handing the ACL and the seeds a container *IP* (ADR-0009, ADR-0020, ADR-0029, and the
`objecten.local` network alias). In Kubernetes a Service FQDN is already multi-label, so
the workaround has a natural replacement — but the hosts have to line up exactly, since
Objecten reflects the request Host into the URLs it publishes to NRC.
- **The OIDC issuer must be one string** for both the browser and the BFF (ADR-0010).
`infra/host-browser.yml` already solved this for a host browser: pin `KC_HOSTNAME`, keep
backchannel discovery in-cluster, and mount a `config.json` per portal.
- **Nothing here is highly available.** One replica of everything, on one node.
## Decision
**One chart — `infra/helm/big-reference` — whose `values.yaml` is a near-literal
transcription of the compose file, rendered by three generic templates (Deployment, Job,
Service) over a `workloads` map.** Adding a service is a values edit.
Consequences of that shape, each chosen deliberately:
- **Config files are not copied into the chart.** `infra/helm/seed-configmaps.sh` creates
the ConfigMaps from the files that already live in the repo — the Kubernetes sibling of
`infra/seed-config.sh`. The chart therefore needs `make k8s-seed` before `helm install`,
which is the same two-step dance compose already has.
- **Bootstrap one-shots become Jobs, with no ordering mechanism.** Every one is idempotent
(ADR-0020); each waits for the TCP ports it needs via a busybox init container and
Kubernetes retries the rest. `make k8s-reseed` re-runs them.
- **The four Django services apply their own `setup_configuration`** —
`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]` — instead of getting a
separate `*-init` Job like compose. Both of those image scripts run
`manage.py migrate`, and compose serialises them with
`depends_on: service_completed_successfully`; Kubernetes has no such edge, so a Job and
its web pod migrate the same database concurrently and Django dies with
*"relation zgw_consumers_service already exists"*. Running the two steps in order inside
the one container leaves exactly one migrator per database, and deletes four workloads.
- **`args`, never `command`.** Compose's `command:` replaces the image's CMD; Kubernetes'
`command:` replaces its ENTRYPOINT. Transcribing one to the other silently broke every
upstream image that relies on its entrypoint — postgres ran as root and refused to
start, Keycloak tried to exec `start-dev` as a binary. The chart now `fail`s at render
time if a workload sets `command`, because the symptom (a crashloop three layers down)
is nothing like the cause.
- **Published ports are NodePorts.** No ingress controller, no LoadBalancer, no TLS. The
four portals are the exception in *use*, not in wiring: PKCE needs `crypto.subtle`, which
browsers expose only in a secure context, so a portal has to be reached over `localhost`
(`make k8s-portals` forwards them) or eventually over HTTPS. `.Values.host` is therefore
"the address the browser uses", not "the node's address" — it pins Keycloak's issuer and
each portal's `config.json`, and both must agree with the URL bar (ADR-0010).
- **Databases are `emptyDir` by default**, so the stack comes up on a cluster with no CSI
driver; setting `persistence.storageClass` switches every database to a PVC.
- **Only two hosts become FQDNs** — OpenZaak (for the ACL and the zaaktype seed) and
Objecten (for the ACL's register writes), the two that Django validates as URLs.
Everything else keeps the short compose service name, because the upstream
`setup_configuration` files name those and Objecten matches an objecttype URL against the
one it was configured with. The portals used to be a third case — nginx's `resolver` never
appends search domains, so the bare `bff` upstream could not resolve on Kubernetes — which
ADR-0034 removed by serving them with Caddy, whose resolver honours `/etc/resolv.conf`.
- **Compose stays CI-canonical.** The chart is a second deployment target, not a
replacement; the acceptance, verify and e2e lanes are unchanged.
### Alternatives considered
- **A chart per service, or an umbrella of 30 subcharts.** The conventional layout, and
roughly 1,500 lines of near-identical YAML for a stack where 28 of 30 workloads are
"one pod, one image, some env". It buys independent versioning we don't want (the stack
is demoed as a whole) and costs the eye-diffability against the compose file that keeps
the two stacks honest.
- **`kompose convert`.** One-shot generation, no ongoing artefact to maintain — but it
drops exactly the parts that carry the design (init ordering, the config volumes, the
issuer pinning) and produces output nobody owns.
- **Bitnami PostgreSQL/Redis subcharts.** Six more dependencies (CLAUDE.md §13) and a
second way of expressing the same three-line database.
- **ingress-nginx with hostname routing.** Needs a controller, `/etc/hosts` entries and a
matching issuer host; NodePorts need none of it and reuse the mechanism
`infra/host-browser.yml` already proves.
- **A registry on the laptop** (the obvious home for images built there). Talos cannot
side-load an image, so a registry is required either way — but reaching one on the host
means opening an inbound port on firewalld's `libvirt` zone, which needs root, and
pushing to it over plain HTTP means an `insecure-registries` entry in the Docker daemon,
which needs root again. `infra/helm/registry.yaml` runs the registry *in* the cluster on
a NodePort instead: pushing laptop → node is outbound and unfiltered, the node pulls from
its own NodePort, and `docker save | crane push --insecure` needs no daemon
configuration. Cost: one more (throwaway, `emptyDir`) workload, and a re-push if its pod
is replaced.
- **Helm hooks (`pre-install`/`post-install`) for bootstrap ordering.** Hooks run after
`--wait`, which would deadlock: OpenZaak's readiness needs the migrations that the hook
is supposed to run. Idempotent Jobs plus retries need no such sequencing.
- ponytail ceiling: single-node assumptions are baked in — one replica per workload,
`Recreate` rollouts, ReadWriteOnce volumes, no PodDisruptionBudgets, no resource
requests or limits (a laptop VM schedules everything or nothing), plain HTTP.
Upgrade path for a real cluster: add requests/limits per workload (the field is already
passed through), swap NodePorts for an Ingress with TLS, and give the databases a real
StorageClass — none of which changes the workload graph.
## Consequences
**Positive**
- One file to read to see what the cluster runs, and it lines up with the compose file
line for line.
- The compose IP workarounds disappear: cluster DNS supplies multi-label hosts.
- `make k8s-lint` renders and schema-checks the whole stack without a cluster.
- The config inputs have exactly one home (the repo) for both stacks — no fork to drift.
**Negative / costs**
- A second deployment description to keep in step with compose. Nothing enforces that
today; a drift check belongs in CI (follow-up).
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
one surfaces as `ContainerCreating`, not as a clear error.
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
schema errors, not intent.
- The verify/e2e lanes do not run against the chart, so the Kubernetes path is verified by
hand (docs/runbooks/kubernetes-talos.md §5) rather than by CI.
- The chart deviates from compose in four places now (args, self-configuring Django pods,
FQDN hosts, NodePorts). Each is forced by the platform and commented where it appears,
but it is four more things that can drift.
## Coupling rules touched (CLAUDE.md §8)
None. The chart deploys the same graph: portals reach only the BFF (§8.3), only the ACL
holds ZGW credentials (§8.1), only the Workflow Client talks to Flowable (§8.2), each
service keeps its own database (§8.5). No workload gained a peer it didn't have in compose.
## Verified
Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: 29 pods ready and four bootstrap Jobs complete in under three minutes,
with zero restarts, using ~4.4 GB of the VM's 10 GB. The smoke test in the runbook's §5
walks the whole path — portal proxy → BFF → domain → Flowable → ACL → OpenZaak + Objecten →
NRC → event-subscriber → projection → public register — plus a werkbak read with an
MFA'd medewerker token. The browser flow itself was driven with Playwright against
`http://localhost:30140`: secure context, PKCE, Keycloak form, login, no console errors.
## Process note
CLAUDE.md §14 wants the ADR proposal issue opened before the code, and §7 wants a slice
issue behind the work. This landed the other way round — chart first, on request. The
issue and the CI drift check are the outstanding follow-ups.
docs/architecture/adr-0034-caddy-serves-the-portals.md
+103
View File
@@ -0,0 +1,103 @@
# ADR-0034: The portals are served by Caddy, not nginx
- **Status:** Accepted
- **Date:** 2026-09-04
- **Deciders:** Respellion engineering
- **Slice:** _(none yet — raised directly alongside the Kubernetes deployment, ADR-0033)_
## Context
Each portal ships as one image that does two jobs: serve the built Angular app, and
reverse-proxy *its own* BFF endpoint group so the browser calls a single origin (no CORS,
and the DigiD/medewerker token rides along — ADR-0010, ADR-0013). Until now that was nginx
with a hand-written `nginx.conf` per app.
Two workarounds had accumulated around nginx's resolver, both for the same root cause:
**nginx resolves a variable `proxy_pass` upstream itself**, using only the `resolver`
directive, and never the search domains in `/etc/resolv.conf`.
1. `resolver 127.0.0.11` (Docker's embedded DNS) is wrong on rootless podman, which uses a
network-specific aardvark address — so `apps/portal-nginx-resolver.sh` rewrote the
directive at container start by reading the pod's actual nameserver.
2. On Kubernetes the bare `bff` name cannot resolve at all without the `svc.cluster.local`
search domain, so the same script gained a `BFF_HOST` override that the Helm chart set
per portal (ADR-0033).
Both existed only to tell the proxy how to resolve one hostname.
## Decision
**Serve the portals with `caddy:2-alpine` and a small `Caddyfile` per app, replacing the
nginx runtime stage, the four `nginx.conf` files, and the resolver workaround.**
Caddy dials its upstream per request through Go's resolver, which reads
`/etc/resolv.conf` — nameserver *and* search domains. So `reverse_proxy bff:8080` resolves
correctly under Docker, rootless podman and Kubernetes with no per-engine configuration,
and it still starts before the BFF exists and picks up its restarts (the property the
variable `proxy_pass` was there to buy). `apps/portal-nginx-resolver.sh`, its unit test and
the chart's `BFF_HOST` env are deleted.
The Caddyfile uses `handle` blocks rather than a bare `try_files`:
```
handle /behandel/* { reverse_proxy bff:8080 }
handle { root * /usr/share/caddy; try_files {path} /index.html; file_server }
```
`handle` blocks are mutually exclusive and matched most-specific-first. This matters:
Caddy's default directive order puts rewrites (`try_files`) *before* `reverse_proxy`, so a
top-level `try_files {path} /index.html` would rewrite every API path to `/index.html`
before the proxy ever saw it — the SPA fallback would silently eat the API. The `handle`
form makes the routing explicit instead of relying on directive-order trivia.
`infra/test_portal_caddyfiles.py` (in `make unit`) asserts each portal proxies exactly its
own endpoint groups and keeps the SPA fallback. The four files are near-identical, so a
copy-paste slip is cheap to make and expensive to find: proxying another portal's group
hands a browser an endpoint its token isn't for, and the failure surfaces as a 401 three
services away.
### Alternatives considered
- **Keep nginx.** Zero migration, and it works — but the resolver workaround stays, and it
had already grown a second head for Kubernetes. Both heads are nginx-specific.
- **Keep nginx, hard-code the FQDN.** Would need a different config per deployment target
(compose vs Kubernetes), which is exactly the fork the chart was written to avoid.
- **Drop the proxy and use CORS.** Turns the same-origin design (ADR-0010) inside out:
CORS preflights, an explicit origin allowlist in the BFF, and a token attached
cross-origin. Not a serving decision — an architectural regression.
- **Kubernetes Ingress in front of the portals.** Solves nothing about compose, adds a
controller, and the portals would still need something to serve static files.
- ponytail ceiling: plain HTTP on `:80`, no compression, no cache headers beyond Caddy's
defaults, and Caddy's automatic HTTPS deliberately unused (there is no hostname to get a
certificate for). Upgrade path: `encode zstd gzip` and a cache policy for immutable
Angular bundles; a real hostname makes TLS a one-line `Caddyfile` change, which is the
main reason this is worth having in place.
## Consequences
**Positive**
- One resolver behaviour across compose, podman and Kubernetes; a script, a unit test and a
chart env var are deleted rather than maintained.
- The images gain `curl` for free (the alpine nginx image had only busybox `wget`), which
the compose healthchecks can use.
- Routing intent is readable: one `handle` block per endpoint group, one for the app.
- TLS later is a one-line change instead of a new component.
**Negative / costs**
- A new runtime dependency in four images (CLAUDE.md §13): Caddy replaces nginx rather than
joining it, so the count is unchanged, but it is a less familiar config language for
anyone who has only read nginx configs.
- The images grew: 90.6 MB against nginx's 75.7 MB, because `caddy:2-alpine` carries a
bigger static binary than nginx's. Measured, not estimated.
- Caddy's directive-order rule is a genuine footgun (see above); the `handle` form and the
Caddyfile comments exist to keep the next person out of it.
- Any operational note that says "the portal's nginx" is now wrong; the ones in `docs/` were
updated with this ADR.
## Coupling rules touched (CLAUDE.md §8)
None. §8.3 is unchanged and unchanged in kind: the portals still talk only to the BFF, and
the proxy is still the thing that makes that same-origin.
docs/demo-script.md
+1 -1
View File
@@ -389,7 +389,7 @@ make verify-e2e # → login as jan-burger → submit → "ontvangen" co
open http://localhost:8140
```
> The portal is served same-origin with the BFF (nginx proxies `/self-service` + `/openbaar`), so no
> The portal is served same-origin with the BFF (Caddy proxies `/self-service` + `/openbaar`), so no
> CORS; the OIDC authority comes from `/config.json` at runtime. See `docs/frontend-decisions.md`.
---
docs/frontend-decisions.md
+8 -6
View File
@@ -77,11 +77,13 @@ with the submit form (S-08c, #67); any deviation from NL DS will be recorded her
## Serving + e2e (S-08d, #68)
- **Served by nginx, same-origin as the BFF.** The compose `self-service` image serves the built app
- **Served by Caddy, same-origin as the BFF.** The compose `self-service` image serves the built app
and **reverse-proxies** `/self-service/*` + `/openbaar/*` to the `bff` service. Because the
api-client uses **relative URLs**, the browser calls the app's own origin → nginx forwards to the
BFF: **no CORS**, and the DigiD token (same-origin) is attached by the interceptor. nginx resolves
the BFF at request time (a `resolver` + variable `proxy_pass`) so it starts before the BFF is up.
api-client uses **relative URLs**, the browser calls the app's own origin → Caddy forwards to the
BFF: **no CORS**, and the DigiD token (same-origin) is attached by the interceptor. Caddy dials
the BFF per request through the system resolver, so it starts before the BFF is up, picks up its
restarts, and resolves the bare `bff` name on every engine — compose, podman and Kubernetes
(ADR-0034; the `Caddyfile` sits next to each app's `Dockerfile`).
- **Runtime config.** The app fetches `/config.json` before bootstrap (`main.ts`); `appConfig` is a
factory. The dev default (`public/config.json`) points at `localhost:8180`; the Docker image bakes
the compose value (`keycloak:8080`). One build, per-environment OIDC authority.
@@ -110,7 +112,7 @@ with the submit form (S-08c, #67); any deviation from NL DS will be recorded her
`angular-auth-oidc-client`, no interceptor, and no `config.json` — `main.ts` bootstraps `appConfig`
directly with just `provideHttpClient` + `provideRouter`. This is the deliberate contrast to
self-service and keeps the app trivially cacheable/CDN-able.
- **Same-origin via nginx, like self-service.** The compose `openbaar` image serves the built app and
- **Same-origin via Caddy, like self-service.** The compose `openbaar` image serves the built app and
reverse-proxies `/openbaar` to the BFF; the api-client's relative calls stay same-origin (no CORS).
Served on `:8141`, health-checked over IPv4 (`127.0.0.1`), no Keycloak dependency.
- **Public-safe by construction.** The portal only ever sees the BFF's `OpenbaarProjection.PublicView`
@@ -138,7 +140,7 @@ frontend work is the medewerker realm auth and the werkbak/decide page. Wiring r
**BFF remains the security boundary** (`behandelaar` policy, 401/403 on `/behandel/*`, ADR-0013);
the frontend role signal is for display/UX, and the werkbak page surfaces a load failure (e.g. a
403 for a non-behandelaar) rather than swallowing it.
- **Same-origin via nginx, like the other portals.** The compose `behandel` image serves the built
- **Same-origin via Caddy, like the other portals.** The compose `behandel` image serves the built
app and reverse-proxies `/behandel` to the BFF (relative calls, no CORS). Served on `:8142`,
health-checked over IPv4 (`127.0.0.1`), depends on Keycloak for the medewerker realm.
- **Werkbak = decide-and-refresh.** `WerkbakPage` loads `GET /behandel/werkbak` on open and renders a
docs/runbooks/kubernetes-talos.md
+370
View File
@@ -0,0 +1,370 @@
# Deploying the stack to a single-node Talos cluster
The Helm chart in `infra/helm/big-reference` is a port of `infra/docker-compose.yml`
(ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up
on a Talos VM under virt-manager on a laptop, including the parts that bite.
Compose remains the CI-canonical stack — `make verify`, the acceptance lane and the
Playwright e2e all still drive it. Kubernetes is a second deployment target.
## 0. What you need
On the laptop, four static binaries, all installable to `~/.local/bin` without root:
```bash
curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64
curl -sSLo ~/.local/bin/kubectl https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane
chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane}
```
Match `talosctl` to the Talos ISO you booted (`talosctl version --insecure -n <ip>` reports
the server's tag). `crane` is what pushes images to a plain-HTTP registry without a
root-level Docker daemon change — see §2.
**VM sizing.** 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare
(measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The
chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the
only budget there is. Resize a stopped VM with:
```bash
virsh -c qemu:///system destroy talos # it's in maintenance mode; nothing is lost
virsh -c qemu:///system setmaxmem talos 10G --config
virsh -c qemu:///system setmem talos 10G --config
virsh -c qemu:///system setvcpus talos 6 --config --maximum
virsh -c qemu:///system setvcpus talos 6 --config
```
Two addresses matter throughout:
| Name | Meaning | Example |
|---|---|---|
| `TALOS_HOST` | the VM's IP — used by the browser, `talosctl` and `kubectl` | `192.168.122.33` |
| `K8S_REGISTRY` | `TALOS_HOST:30500` — the in-cluster registry (§2) | `192.168.122.33:30500` |
Find the VM's address with `virsh -c qemu:///system net-dhcp-leases default`.
## 1. Install Talos onto the VM
### The virt-manager trap
virt-manager treats the install ISO as one-shot: on the VM's **first shutdown** it ejects
the CD and rewrites the boot order to `hd`. A Talos VM booted from `metal-amd64.iso` runs
entirely in RAM, so the disk is still empty — the next start lands on
`Boot failed: not a bootable disk`. Put the ISO back before installing:
```bash
virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert
virt-xml -c qemu:///system talos --edit --boot cdrom,hd
virsh -c qemu:///system start talos
```
Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio
it is `/dev/vda`, and Talos's default selector expects `/dev/sda`:
```bash
talosctl get disks --insecure -n <TALOS_HOST> -e <TALOS_HOST>
```
### Generate the machine config
Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular
`machine.install` is now `UnattendedInstallConfig`, and patching the old field is rejected
with *"UnattendedInstallConfig config is incompatible with v1alpha1 config"*. Write
`patch.yaml` as a multi-document patch:
```yaml
machine:
certSANs:
- 192.168.122.33
registries:
mirrors:
# The in-cluster registry (§2) speaks plain HTTP.
"192.168.122.33:30500":
endpoints:
- http://192.168.122.33:30500
---
apiVersion: v1alpha1
kind: UnattendedInstallConfig
provisioning:
diskSelector:
match: disk.dev_path == "/dev/vda"
```
```bash
talosctl gen config big https://<TALOS_HOST>:6443 --output-dir ~/.talos/big --config-patch @patch.yaml
talosctl apply-config --insecure -n <TALOS_HOST> -e <TALOS_HOST> --file ~/.talos/big/controlplane.yaml
```
Talos installs to the disk and **kexecs straight into the installed system**, so the CD
boot order doesn't get in the way here. Then point the client at the node and bootstrap:
```bash
talosctl config merge ~/.talos/big/talosconfig
talosctl config endpoint <TALOS_HOST>
talosctl config node <TALOS_HOST>
talosctl bootstrap # wait for `talosctl version` to answer first
talosctl kubeconfig -f ~/.kube/config
```
A single-node cluster must run workloads on the control plane, or CoreDNS never schedules:
```bash
kubectl taint node --all node-role.kubernetes.io/control-plane-
```
Finally, make a VM restart boot the installed system rather than the ISO (takes effect at
the next full power cycle):
```bash
virsh -c qemu:///system change-media talos sda --eject --config
virt-xml -c qemu:///system talos --edit --boot hd
```
## 2. A registry the node can pull from
Talos has no Docker daemon and no way to side-load an image, so this repo's images have to
come from a registry. The registry runs **inside the cluster**, published on NodePort
30500 (`infra/helm/registry.yaml`):
```bash
make k8s-registry
```
Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port
opened on firewalld's `libvirt` zone (`sudo firewall-cmd --zone=libvirt --add-port=5000/tcp`),
which needs root. Pushing from the laptop *to* the node is outbound and always allowed, and
the node pulls from its own NodePort. If you do open that port, put a registry on the
laptop instead and point `K8S_REGISTRY` at `<laptop-ip>:5000` — the mirror patch in §1 has
an entry ready for it.
Its storage is `emptyDir`, so if the registry pod is ever replaced, re-run `make k8s-images`.
## 3. Build and push the images
```bash
make k8s-images K8S_REGISTRY=<TALOS_HOST>:30500
```
This builds the nine images with `docker compose build` — same contexts and Dockerfiles as
compose, no second build definition — then `docker save | crane push --insecure` each one.
`docker push` is not used: the registry speaks plain HTTP, which the Docker daemon refuses
without a root-level `insecure-registries` entry, while crane just takes `--insecure`.
## 4. Deploy
```bash
make k8s-up TALOS_HOST=<TALOS_HOST> K8S_REGISTRY=<TALOS_HOST>:30500
```
That does two things:
1. `make k8s-seed` — creates the ConfigMaps the chart mounts, from the config files that
already live in this repo (`infra/helm/seed-configmaps.sh`): the four
`setup_configuration/data.yaml` files, the Keycloak realm exports, the BPMN + DMN, and
the two bootstrap scripts. Re-run it after editing any of them.
2. `helm upgrade --install` of the chart into namespace `big`.
First bring-up takes a few minutes: the four Django services migrate their databases and
apply their `setup_configuration`, Flowable creates its schema, and the bootstrap Jobs
deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement.
```bash
kubectl -n big get pods -w
kubectl -n big get jobs # all four must reach COMPLETIONS 1/1
```
The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere
specific:
| Job | What breaks without it |
|---|---|
| `flowable-init` | no `registratie` process, no diploma DMN |
| `registerrecord-init` | the register has no RegisterRecord objecttype, so writes are refused |
| `seed-zaaktype` | the ACL can't resolve `BIG-REGISTRATIE`, so no zaak is created |
| `nrc-subscribe` | register writes never reach the projection — the public register stays empty |
## 5. Use it
### The portals must be reached over `localhost`
The portals' OIDC flow uses PKCE, which needs `crypto.subtle` — and browsers only expose
that in a **secure context**: HTTPS, or an origin on `localhost`/`127.0.0.1`. A NodePort on
the VM's IP is neither, so `http://<TALOS_HOST>:30140` fails before it can even build the
authorize URL:
```
ERROR TypeError: Cannot read properties of undefined (reading 'digest')
at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize
```
So deploy with `TALOS_HOST=localhost` — which pins Keycloak's issuer and the portals'
`config.json` authority to `http://localhost:30180` — and forward the browser-facing
services to those same ports:
```bash
make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500
make k8s-portals # stays in the foreground; Ctrl-C stops all five forwards
```
| URL (needs `make k8s-portals`) | What |
|---|---|
| `http://localhost:30140` | self-service portal (DigiD) |
| `http://localhost:30141` | openbaar register (anonymous) |
| `http://localhost:30142` | behandel portal (medewerker) |
| `http://localhost:30143` | beheer portal (medewerker) |
| `http://localhost:30180` | Keycloak (admin/admin) |
The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed
string, so the port the browser uses has to match the one baked into `config.json`.
This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
`localhost:8180`); only the addresses differ.
### The admin UIs work straight off the NodePorts
These are server-rendered and need no secure context, so they are reachable at the VM's
address with no forwarding:
| URL | What |
|---|---|
| `http://<TALOS_HOST>:30000` | OpenZaak admin (admin/admin) |
| `http://<TALOS_HOST>:30001` | Open Notificaties admin (admin/admin) |
| `http://<TALOS_HOST>:30020` / `:30021` | Objecttypen / Objecten admin |
| `http://<TALOS_HOST>:30080` | BFF (`/health`) |
| `http://<TALOS_HOST>:30090` | Flowable REST (rest-admin/test) |
### Credentials
Log in with the test users from `docs/synthetic-data.md` (all password `test123`, e.g.
`jan-burger` for self-service, `merel-behandelaar` for behandel). The `medewerker` realm
enforces MFA (ADR-0031) — print a current code with
`python3 infra/keycloak/check_realms.py otp`. Walk the flow in `docs/demo-script.md`.
`TALOS_HOST` is not cosmetic: it pins Keycloak's issuer (`KC_HOSTNAME`) and the portals'
OIDC authority to the same string, which is what makes a browser token pass the BFF's
validation (ADR-0010). Change it and you must re-run `make k8s-up` — the chart rolls the
portals for you, because their `config.json` is a subPath mount and would otherwise keep
serving the old authority.
### Smoke-test the whole chain without a browser
With the forwards running:
```bash
TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \
-d grant_type=password -d client_id=big-portal \
-d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token)
# through the portal's Caddy, so this also proves the BFF reverse proxy
curl -s -X POST http://localhost:30140/self-service/registrations \
-H "Authorization: Bearer $TOK" -H 'Content-Length: 0'
# → {"registrationId":"…","status":"Ingediend"}
curl -s http://localhost:30141/openbaar/register
# → [{"id":"…","status":"INGEDIEND","reference":"<the registrationId>"}]
```
The second call proves the whole Common Ground path: portal → BFF → domain → Flowable →
ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register.
## 6. Keeping the databases (recommended if you iterate on the chart)
By default every database is an `emptyDir`: no CSI driver needed, and the data lives as
long as the pod. Note what that means in practice — **any** change to a database pod's
template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack
then needs its bootstrap re-run:
```bash
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...
```
which re-runs the four Jobs *and* restarts `event-subscriber` + `projection-api`, because
those two create the projection schema on start and otherwise keep writing to a
schema-less database (`relation "processed_notifications" does not exist`). For persistence, install Rancher's local-path-provisioner — on Talos it
must write under `/var` and its namespace needs the privileged Pod Security label:
```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31
patches:
- patch: |-
kind: ConfigMap
apiVersion: v1
metadata:
name: local-path-config
namespace: local-path-storage
data:
config.json: |-
{ "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] }
- patch: |-
apiVersion: v1
kind: Namespace
metadata:
name: local-path-storage
labels:
pod-security.kubernetes.io/enforce: privileged
```
```bash
kubectl apply -k .
make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path'
```
The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the data behind;
`make k8s-purge` drops the namespace and with it the volumes.
## 7. Day-to-day
```bash
make k8s-lint # render + schema-check the chart, no cluster needed
make k8s-portals # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=... # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
make k8s-seed # after editing a data.yaml, a realm export, or the BPMN
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=... # re-run the bootstrap Jobs + reset the projection schema
make k8s-down # uninstall, keep the database PVCs
make k8s-purge # uninstall and drop the namespace
```
This repo's images are pulled with `imagePullPolicy: Always` (the `dev` tag is mutable), so
`kubectl -n big rollout restart deploy/<name>` after `make k8s-images` picks up a rebuild.
Upstream images stay `IfNotPresent`: their tags are pinned, and keeping them out of the pod
template avoids needless churn — a changed template makes a Job unpatchable.
`k8s-reseed` is also the path for *changing* a Job in the chart: a Job's pod template is
immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`.
## 8. When it doesn't work
| Symptom | Cause |
|---|---|
| `Boot failed: not a bootable disk` | virt-manager ejected the install ISO on first shutdown — see §1 |
| The VM comes back in maintenance mode after a restart | the ISO is still attached and boots first; eject it and set `--boot hd` (§1) |
| `apply-config` rejects the patch with *"incompatible with v1alpha1"* | Talos ≥1.14 owns that field in its own config document — patch the document, not `machine.*` (§1) |
| CoreDNS `Pending` forever | the control-plane taint is still on the only node (§1) |
| `ImagePullBackOff` … `pull QPS exceeded` | transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry |
| `ImagePullBackOff` on a `register-referentie/*` image | the registry mirror patch is missing: `talosctl get registriesconfig` |
| Pod stuck in `ContainerCreating`, event names a ConfigMap | `make k8s-seed` |
| `seed-zaaktype` retrying | publishing a zaaktype validates the resultaattype against `selectielijst.openzaak.nl`, so this one Job needs outbound internet from the VM (ADR-0006) |
| `TypeError: Cannot read properties of undefined (reading 'digest')` on a portal | not a secure context: `crypto.subtle` is absent on `http://<ip>`. Use `localhost` + `make k8s-portals` (§5) |
| Login redirects but the portal stays logged out, or the BFF answers 401 | `TALOS_HOST` doesn't match the address in the browser's URL bar — issuer mismatch. Re-run `make k8s-up` with the right value |
| A portal returns 502 on `/self-service/…` | the BFF is unreachable from the portal pod: check `kubectl -n big get svc bff` and the BFF's own readiness |
| Public register empty after a submit | usually a wiped `emptyDir` database (§6): `make k8s-reseed`. Confirm with `kubectl -n big logs deploy/event-subscriber \| grep 42P01` |
| `helm upgrade` fails with `cannot patch … with kind Job` | see §7 — use `make k8s-reseed` |
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |
## What is not ported
- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
images too, so switching them on means pushing them as well:
`K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
chart. The Kubernetes path is verified with §5's smoke test.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.