S-24/#25 · Helm chart + Kubernetes deployment, and Caddy for the portals (#166) #167

Merged

not merged 6 commits from feat/25-helm-kubernetes-caddy into main 2026-09-10 08:53:59 +00:00

Conversation 0 Commits 6 Files Changed 42

+1991 -183

not commented 2026-09-04 15:27:06 +00:00

Contributor

Copy Link

Copy Source

What & why

Two changes, made and verified together on a real cluster.

S-24 / #25 — a Helm chart for the platform. One chart, infra/helm/big-reference,
whose values.yaml is a near-literal transcription of infra/docker-compose.yml, rendered
by three generic templates (Deployment, Job, Service) over a workloads map. Adding a
service is a values edit. make k8s-lint renders and schema-checks the whole stack without
a cluster. The issue asked for a sketch; this is deployed and verified end to end (see
below), which is more than it asked for — the part it asked for that is not here is the
production-posture write-up (HA, secrets, backup), see Known gaps.

#166 — Caddy replaces nginx in the portals. nginx resolves a variable proxy_pass
upstream itself, using only the resolver directive and never /etc/resolv.conf's search
domains. That had cost two workarounds in one script: rewriting the resolver address for
rootless podman, and injecting a full FQDN so the bare bff name could resolve on
Kubernetes. Caddy dials per request through the system resolver, so reverse_proxy bff:8080 works on every engine unchanged; apps/portal-nginx-resolver.sh and the chart's
BFF_HOST env are deleted.

Closes #25
Closes #166

Definition of Done

• Linked Gitea issue (above).
• Failing test committed before the implementation — twice: the Caddyfile contract test
  before the Caddyfiles, make k8s-lint before the chart.
• Implementation makes the test pass.
• Conventional Commits referencing the issues (refs #25 / refs #166).
• CI green — awaiting the run on this PR (make k8s-lint, dotnet format and the new
  unit self-check pass locally; the compose e2e and mutation lanes are CI's).
• docker compose up from a fresh clone reaches green health checks within 3 minutes —
  the portal images were rebuilt and verified standalone, but a full make up run has
  not been done on this branch. Please confirm in review or let CI's smoke test speak.
• Docs updated — docs/runbooks/kubernetes-talos.md (new), frontend-decisions.md,
  demo-script.md, and the docs that named nginx.
• ADR added — ADR-0033 (chart) and ADR-0034 (Caddy).
• Demo note in docs/demo-script.md — not added: the deployment target is not a
  user-visible slice, and the Caddy swap is invisible to the demo script beyond the
  wording fix included here.

How it was verified

Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: 29 pods ready and four bootstrap Jobs complete in under three
minutes, zero restarts, using ~4.4 GB of the VM's 10 GB.

• Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak +
  Objecten → NRC → event-subscriber → projection → public register (INGEDIEND, reference
  matching the submitted registration).
• Werkbak read with an MFA'd medewerker token → 200.
• The browser flow driven with Playwright against http://localhost:30140: secure context,
  crypto.subtle present, Keycloak form reached, login completed, no console errors.
• Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies
  its own groups, and a portal does not proxy a neighbour's group.

Notes for reviewers

Three bugs this shook out, each fixed at the cause rather than the symptom:

1. command vs args. Compose's command: replaces the image CMD; Kubernetes'
   replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that
   relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec
   start-dev. The chart now fails at render time on command.
2. Concurrent migrations. Both /setup_configuration.sh and /start.sh run
   manage.py migrate; compose serialises them with depends_on, Kubernetes has no such
   edge, so the init Job and its web pod raced (relation "zgw_consumers_service" already exists). The four Django services now do both steps in order in the web pod — which
   also deletes four workloads.
3. emptyDir databases are wiped by any pod-template change. make k8s-reseed now
   also restarts event-subscriber and projection-api, which create the projection
   schema on start and otherwise keep writing to a schema-less database.

Known gaps / follow-ups:

• Secrets. values.yaml carries the dev credentials in plain text (admin/admin, the
  ZGW client secret, the two Objecten tokens) and the chart has no Secret objects. Fine
  for a laptop demo, and exactly what #25's "production posture" ADR should address — I
  suggest a follow-up issue rather than stretching this PR.
• No CI gate for the chart yet. make k8s-lint exists but is not wired into
  .gitea/workflows/ci.yaml, and nothing enforces that the chart and the compose file stay
  in step. Worth a small follow-up.
• This is two slices in one PR. They were built and verified together and the diff is
  entangled (the chart was written against Caddy from the start), so splitting now would
  mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather.
• Rebased onto #161 (merged as #165) rather than merged, to keep the history linear.
  One conflict, in the unit: target where both branches add a self-check line — resolved
  by keeping both. #161's infra/host-browser.yml arrived with
  /usr/share/nginx/html/config.json and is fixed to /usr/share/caddy/ inside the
  feat(portals) commit, so no commit on this branch leaves that overlay pointing at a
  path the images no longer have.

## What & why Two changes, made and verified together on a real cluster. **S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`, whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see below), which is more than it asked for — the part it asked for that is **not** here is the production-posture write-up (HA, secrets, backup), see Known gaps. **#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass` upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search domains. That had cost two workarounds in one script: rewriting the resolver address for rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's `BFF_HOST` env are deleted. Closes #25 Closes #166 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing test committed before the implementation — twice: the Caddyfile contract test before the Caddyfiles, `make k8s-lint` before the chart. - [x] Implementation makes the test pass. - [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`). - [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new unit self-check pass locally; the compose e2e and mutation lanes are CI's). - [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes — the portal images were rebuilt and verified standalone, but a full `make up` run has not been done on this branch. Please confirm in review or let CI's smoke test speak. - [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`, `demo-script.md`, and the docs that named nginx. - [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy). - [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a user-visible slice, and the Caddy swap is invisible to the demo script beyond the wording fix included here. ## How it was verified Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk) under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB. - Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference matching the submitted registration). - Werkbak read with an MFA'd medewerker token → 200. - The browser flow driven with Playwright against `http://localhost:30140`: secure context, `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**. - Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies its own groups, and a portal does *not* proxy a neighbour's group. ## Notes for reviewers Three bugs this shook out, each fixed at the cause rather than the symptom: 1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes' replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec `start-dev`. The chart now `fail`s at render time on `command`. 2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already exists`). The four Django services now do both steps in order in the web pod — which also deletes four workloads. 3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now also restarts `event-subscriber` and `projection-api`, which create the projection schema on start and otherwise keep writing to a schema-less database. Known gaps / follow-ups: - **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine for a laptop demo, and exactly what #25's "production posture" ADR should address — I suggest a follow-up issue rather than stretching this PR. - **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay in step. Worth a small follow-up. - **This is two slices in one PR.** They were built and verified together and the diff is entangled (the chart was written against Caddy from the start), so splitting now would mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather. - **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear. One conflict, in the `unit:` target where both branches add a self-check line — resolved by keeping both. #161's `infra/host-browser.yml` arrived with `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a path the images no longer have.

not added the area:portal-self-servicearea:portal-behandelarea:infra labels 2026-09-04 15:27:23 +00:00

not added this to the Iteration 6 — Production Posture milestone 2026-09-04 15:27:23 +00:00

not added the type:slice label 2026-09-04 15:27:43 +00:00

not added 6 commits 2026-09-04 15:51:46 +00:00

test(portals): assert each portal proxies only its own endpoint group (refs #166) ... 51d99855d1

The four portal proxy configs are near-identical, so a copy-paste slip is cheap to
introduce and expensive to find: proxying another portal's endpoint group hands a
browser an endpoint its token is not for, and the failure surfaces as a 401 three
services away. Asserts each portal proxies exactly its own groups to the BFF and
keeps the SPA fallback for Angular's client-side routes.

Red: the Caddyfiles it reads do not exist yet.

feat(portals): serve each portal with Caddy instead of nginx (refs #166) ... 4edcf00267

nginx resolves a variable `proxy_pass` upstream itself, using only the `resolver`
directive and never the search domains in /etc/resolv.conf. That cost two
workarounds in one script: rewriting the resolver address for rootless podman
(Docker's 127.0.0.11 is wrong there), and injecting a full FQDN so the bare `bff`
name could resolve on Kubernetes at all.

Caddy dials its upstream per request through the system resolver, which reads
nameserver *and* search domains, so `reverse_proxy bff:8080` resolves on every
engine with no per-engine configuration — and it still starts before the BFF
exists and picks up its restarts. Both workarounds are deleted with the script.

Routing uses mutually-exclusive `handle` blocks, not a bare `try_files`: Caddy
sorts rewrites *before* reverse_proxy, so a top-level SPA fallback would rewrite
every API path to /index.html before the proxy saw it.

docs(portals): ADR-0034 — Caddy serves the portals (refs #166) ... 2d40c84e2c

Records the decision, the directive-order footgun that shapes the Caddyfiles, and
the measured cost (the images grew 75.7 MB → 90.6 MB). Also updates the three
frontend-decisions entries and the two other docs that named nginx.

test(k8s): gate the Helm chart with a render + schema check (refs #25) ... 916d671d49

`make k8s-lint` runs `helm lint` plus a full `helm template`, so a values typo or a
malformed resource is caught without a cluster — the only automated check the chart
can have while CI has no Kubernetes to deploy into.

Red: there is no chart to lint yet.

feat(k8s): Helm chart for the whole stack on a single-node cluster (refs #25) ... 7a5840149c

One chart whose values.yaml is a near-literal transcription of
infra/docker-compose.yml, rendered by three generic templates (Deployment, Job,
Service) over a `workloads` map — so the two stacks can be diffed by eye instead
of by archaeology, and adding a service is a values edit.

Platform-forced deviations, each commented where it appears:
- `args`, never `command`: compose replaces the image CMD, Kubernetes replaces the
  ENTRYPOINT. The chart fails to render on `command`, because the symptom (postgres
  refusing to run as root, Keycloak exec-ing `start-dev`) is nothing like the cause.
- The four Django services apply their own setup_configuration in the web pod
  rather than in a separate init Job: both scripts migrate, and without compose's
  depends_on they race the same database.
- OpenZaak and Objecten are addressed by service FQDN, because Django rejects a
  single-label host in a URL — the reason compose passes container IPs around.
- NodePorts, no ingress; databases are emptyDir until persistence.storageClass is
  set, so the stack comes up on a cluster with no CSI driver.

The upstream config inputs stay in the repo and become ConfigMaps via
infra/helm/seed-configmaps.sh — the Kubernetes sibling of infra/seed-config.sh —
so the compose stack and the chart cannot fork. infra/helm/registry.yaml runs an
in-cluster registry because Talos cannot side-load an image and a laptop-side one
needs a root-level firewall change.

docs(k8s): ADR-0033 + the Talos deployment runbook (refs #25) ... e6aaed7c8c

ADR-0033 records why one values-driven chart rather than 30 subcharts, the four
platform-forced deviations from compose, and the alternatives (kompose, bitnami
subcharts, ingress-nginx, Helm hooks for ordering, a laptop-side registry).

The runbook is the walkthrough as actually performed on a single-node Talos v1.14
VM under virt-manager, including the parts that bite: virt-manager ejecting the
install ISO on first shutdown, Talos 1.14 moving the install disk into its own
config document, the control-plane taint, and why the portals must be reached
over localhost (crypto.subtle needs a secure context for PKCE).

not force-pushed feat/25-helm-kubernetes-caddy from 6e00deb6ee to e6aaed7c8c 2026-09-04 15:51:46 +00:00 Compare

not merged commit 1dd8bd4e1b into main 2026-09-10 08:53:59 +00:00

not referenced this pull request 2026-09-10 08:59:06 +00:00

ADR: production posture for the Kubernetes deployment — secrets, HA, backup #170

not referenced this pull request 2026-09-10 09:01:57 +00:00

ci(k8s): gate the Helm chart in CI + a compose↔chart drift check (closes #168) #171

not referenced this issue from a commit 2026-09-18 13:25:26 +00:00

ci(k8s): gate the Helm chart in CI + a compose↔chart drift check (closes #168) (#171)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area:acl

area:bff

area:docs

area:domain

area:event-subscriber

area:infra

area:portal-behandel

area:portal-beheer

area:portal-openbaar

area:portal-self-service

area:projection

area:workflow

type:adr-proposal

type:bug

type:chore

type:slice

No labels area:infra area:portal-behandel area:portal-self-service type:slice

Milestone

No items

No Milestone Iteration 6 — Production Posture

Projects

Clear projects

No projects

Assignees

Clear assignees

eho (Edwin van den Houdt)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: eho/register-referentie#167