register-referentie/docs/runbooks/kubernetes-talos.md

Deploying the stack to a single-node Talos cluster

The Helm chart in infra/helm/big-reference is a port of infra/docker-compose.yml (ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up on a Talos VM under virt-manager on a laptop, including the parts that bite.

Compose remains the CI-canonical stack — make verify, the acceptance lane and the Playwright e2e all still drive it. Kubernetes is a second deployment target.

0. What you need

On the laptop, four static binaries, all installable to ~/.local/bin without root:

curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64
curl -sSLo ~/.local/bin/kubectl   https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane
chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane}

Match talosctl to the Talos ISO you booted (talosctl version --insecure -n <ip> reports the server's tag). crane is what pushes images to a plain-HTTP registry without a root-level Docker daemon change — see §2.

VM sizing. 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare (measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the only budget there is. Resize a stopped VM with:

virsh -c qemu:///system destroy talos      # it's in maintenance mode; nothing is lost
virsh -c qemu:///system setmaxmem talos 10G --config
virsh -c qemu:///system setmem    talos 10G --config
virsh -c qemu:///system setvcpus  talos 6 --config --maximum
virsh -c qemu:///system setvcpus  talos 6 --config

Two addresses matter throughout:

Name	Meaning	Example
TALOS_HOST	the VM's IP — used by the browser, talosctl and kubectl	192.168.122.33
K8S_REGISTRY	TALOS_HOST:30500 — the in-cluster registry (§2)	192.168.122.33:30500

Find the VM's address with virsh -c qemu:///system net-dhcp-leases default.

1. Install Talos onto the VM

The virt-manager trap

virt-manager treats the install ISO as one-shot: on the VM's first shutdown it ejects the CD and rewrites the boot order to hd. A Talos VM booted from metal-amd64.iso runs entirely in RAM, so the disk is still empty — the next start lands on Boot failed: not a bootable disk. Put the ISO back before installing:

virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert
virt-xml -c qemu:///system talos --edit --boot cdrom,hd
virsh -c qemu:///system start talos

Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio it is /dev/vda, and Talos's default selector expects /dev/sda:

talosctl get disks --insecure -n <TALOS_HOST> -e <TALOS_HOST>

Generate the machine config

Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular machine.install is now UnattendedInstallConfig, and patching the old field is rejected with "UnattendedInstallConfig config is incompatible with v1alpha1 config". Write patch.yaml as a multi-document patch:

machine:
  certSANs:
    - 192.168.122.33
  registries:
    mirrors:
      # The in-cluster registry (§2) speaks plain HTTP.
      "192.168.122.33:30500":
        endpoints:
          - http://192.168.122.33:30500
---
apiVersion: v1alpha1
kind: UnattendedInstallConfig
provisioning:
  diskSelector:
    match: disk.dev_path == "/dev/vda"

talosctl gen config big https://<TALOS_HOST>:6443 --output-dir ~/.talos/big --config-patch @patch.yaml
talosctl apply-config --insecure -n <TALOS_HOST> -e <TALOS_HOST> --file ~/.talos/big/controlplane.yaml

Talos installs to the disk and kexecs straight into the installed system, so the CD boot order doesn't get in the way here. Then point the client at the node and bootstrap:

talosctl config merge ~/.talos/big/talosconfig
talosctl config endpoint <TALOS_HOST>
talosctl config node <TALOS_HOST>
talosctl bootstrap                      # wait for `talosctl version` to answer first
talosctl kubeconfig -f ~/.kube/config

A single-node cluster must run workloads on the control plane, or CoreDNS never schedules:

kubectl taint node --all node-role.kubernetes.io/control-plane-

Finally, make a VM restart boot the installed system rather than the ISO (takes effect at the next full power cycle):

virsh -c qemu:///system change-media talos sda --eject --config
virt-xml -c qemu:///system talos --edit --boot hd

2. A registry the node can pull from

Talos has no Docker daemon and no way to side-load an image, so this repo's images have to come from a registry. The registry runs inside the cluster, published on NodePort 30500 (infra/helm/registry.yaml):

make k8s-registry

Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port opened on firewalld's libvirt zone (sudo firewall-cmd --zone=libvirt --add-port=5000/tcp), which needs root. Pushing from the laptop to the node is outbound and always allowed, and the node pulls from its own NodePort. If you do open that port, put a registry on the laptop instead and point K8S_REGISTRY at <laptop-ip>:5000 — the mirror patch in §1 has an entry ready for it.

Its storage is emptyDir, so if the registry pod is ever replaced, re-run make k8s-images.

3. Build and push the images

make k8s-images K8S_REGISTRY=<TALOS_HOST>:30500

This builds the nine images with docker compose build — same contexts and Dockerfiles as compose, no second build definition — then docker save | crane push --insecure each one. docker push is not used: the registry speaks plain HTTP, which the Docker daemon refuses without a root-level insecure-registries entry, while crane just takes --insecure.

4. Deploy

make k8s-up TALOS_HOST=<TALOS_HOST> K8S_REGISTRY=<TALOS_HOST>:30500

That does two things:

1. make k8s-seed — creates the ConfigMaps the chart mounts, from the config files that already live in this repo (infra/helm/seed-configmaps.sh): the four setup_configuration/data.yaml files, the Keycloak realm exports, the BPMN + DMN, and the two bootstrap scripts. Re-run it after editing any of them.
2. helm upgrade --install of the chart into namespace big.

First bring-up takes a few minutes: the four Django services migrate their databases and apply their setup_configuration, Flowable creates its schema, and the bootstrap Jobs deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement.

kubectl -n big get pods -w
kubectl -n big get jobs        # all four must reach COMPLETIONS 1/1

The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere specific:

Job	What breaks without it
flowable-init	no registratie process, no diploma DMN
registerrecord-init	the register has no RegisterRecord objecttype, so writes are refused
seed-zaaktype	the ACL can't resolve BIG-REGISTRATIE, so no zaak is created
nrc-subscribe	register writes never reach the projection — the public register stays empty

5. Use it

The portals must be reached over localhost

The portals' OIDC flow uses PKCE, which needs crypto.subtle — and browsers only expose that in a secure context: HTTPS, or an origin on localhost/127.0.0.1. A NodePort on the VM's IP is neither, so http://<TALOS_HOST>:30140 fails before it can even build the authorize URL:

ERROR TypeError: Cannot read properties of undefined (reading 'digest')
    at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize

So deploy with TALOS_HOST=localhost — which pins Keycloak's issuer and the portals' config.json authority to http://localhost:30180 — and forward the browser-facing services to those same ports:

make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500
make k8s-portals          # stays in the foreground; Ctrl-C stops all five forwards

URL (needs make k8s-portals)	What
http://localhost:30140	self-service portal (DigiD)
http://localhost:30141	openbaar register (anonymous)
http://localhost:30142	behandel portal (medewerker)
http://localhost:30143	beheer portal (medewerker)
http://localhost:30180	Keycloak (admin/admin)

The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed string, so the port the browser uses has to match the one baked into config.json.

This is the same mechanism infra/host-browser.yml uses for the compose stack (which pins localhost:8180); only the addresses differ.

All of this is what §10 removes: with a public domain the portals have real certificates, so the browser gets its secure context and no forwarding is involved.

The admin UIs work straight off the NodePorts

These are server-rendered and need no secure context, so they are reachable at the VM's address with no forwarding:

URL	What
http://<TALOS_HOST>:30000	OpenZaak admin (admin/admin)
http://<TALOS_HOST>:30001	Open Notificaties admin (admin/admin)
http://<TALOS_HOST>:30020 / :30021	Objecttypen / Objecten admin
http://<TALOS_HOST>:30080	BFF (/health)
http://<TALOS_HOST>:30090	Flowable REST (rest-admin/test)

Credentials

Log in with the test users from docs/synthetic-data.md (all password test123, e.g. jan-burger for self-service, merel-behandelaar for behandel). The medewerker realm enforces MFA (ADR-0031) — print a current code with python3 infra/keycloak/check_realms.py otp. Walk the flow in docs/demo-script.md.

TALOS_HOST is not cosmetic: it pins Keycloak's issuer (KC_HOSTNAME) and the portals' OIDC authority to the same string, which is what makes a browser token pass the BFF's validation (ADR-0010). Change it and you must re-run make k8s-up — the chart rolls the portals for you, because their config.json is a subPath mount and would otherwise keep serving the old authority.

Smoke-test the whole chain without a browser

With the forwards running:

TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \
  -d grant_type=password -d client_id=big-portal \
  -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token)

# through the portal's Caddy, so this also proves the BFF reverse proxy
curl -s -X POST http://localhost:30140/self-service/registrations \
  -H "Authorization: Bearer $TOK" -H 'Content-Length: 0'
# → {"registrationId":"…","status":"Ingediend"}

curl -s http://localhost:30141/openbaar/register
# → [{"id":"…","status":"INGEDIEND","reference":"<the registrationId>"}]

The second call proves the whole Common Ground path: portal → BFF → domain → Flowable → ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register.

6. Keeping the databases (recommended if you iterate on the chart)

By default every database is an emptyDir: no CSI driver needed, and the data lives as long as the pod. Note what that means in practice — any change to a database pod's template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack then needs its bootstrap re-run:

make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...

which re-runs the four Jobs and restarts event-subscriber + projection-api, because those two create the projection schema on start and otherwise keep writing to a schema-less database (relation "processed_notifications" does not exist). For persistence, install Rancher's local-path-provisioner — on Talos it must write under /var and its namespace needs the privileged Pod Security label:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31
patches:
  - patch: |-
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: local-path-config
        namespace: local-path-storage
      data:
        config.json: |-
          { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] }
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: local-path-storage
        labels:
          pod-security.kubernetes.io/enforce: privileged

kubectl apply -k .
make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path'

The PVCs carry helm.sh/resource-policy: keep, so make k8s-down leaves the data behind; make k8s-purge drops the namespace and with it the volumes.

7. Day-to-day

make k8s-lint                      # render + schema-check the chart, no cluster needed
make k8s-drift                     # fail if compose and the chart describe different stacks
make k8s-portals                   # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=...   # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
make k8s-seed                      # after editing a data.yaml, a realm export, or the BPMN
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...   # re-run the bootstrap Jobs + reset the projection schema
make k8s-down                      # uninstall, keep the database PVCs
make k8s-purge                     # uninstall and drop the namespace

This repo's images are pulled with imagePullPolicy: Always (the dev tag is mutable), so kubectl -n big rollout restart deploy/<name> after make k8s-images picks up a rebuild. Upstream images stay IfNotPresent: their tags are pinned, and keeping them out of the pod template avoids needless churn — a changed template makes a Job unpatchable.

k8s-reseed is also the path for changing a Job in the chart: a Job's pod template is immutable, so helm upgrade is rejected with cannot patch "…" with kind Job.

8. When it doesn't work

Symptom	Cause
Boot failed: not a bootable disk	virt-manager ejected the install ISO on first shutdown — see §1
The VM comes back in maintenance mode after a restart	the ISO is still attached and boots first; eject it and set --boot hd (§1)
apply-config rejects the patch with "incompatible with v1alpha1"	Talos ≥1.14 owns that field in its own config document — patch the document, not machine.* (§1)
CoreDNS Pending forever	the control-plane taint is still on the only node (§1)
ImagePullBackOff … pull QPS exceeded	transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry
ImagePullBackOff on a register-referentie/* image	the registry mirror patch is missing: talosctl get registriesconfig
Pod stuck in ContainerCreating, event names a ConfigMap	make k8s-seed
seed-zaaktype retrying	publishing a zaaktype validates the resultaattype against selectielijst.openzaak.nl, so this one Job needs outbound internet from the VM (ADR-0006)
TypeError: Cannot read properties of undefined (reading 'digest') on a portal	not a secure context: crypto.subtle is absent on http://<ip>. Use localhost + make k8s-portals (§5)
Login redirects but the portal stays logged out, or the BFF answers 401	TALOS_HOST doesn't match the address in the browser's URL bar — issuer mismatch. Re-run make k8s-up with the right value
A portal returns 502 on /self-service/…	the BFF is unreachable from the portal pod: check kubectl -n big get svc bff and the BFF's own readiness
Public register empty after a submit	usually a wiped emptyDir database (§6): make k8s-reseed. Confirm with kubectl -n big logs deploy/event-subscriber | grep 42P01
helm upgrade fails with cannot patch … with kind Job	see §7 — use make k8s-reseed
Pods Evicted / OOMKilled	the VM is too small (§0)
A Job shows BackoffLimitExceeded	read it: kubectl -n big logs job/<name>

10. Publishing it on a public domain

By default the stack has no hostname: it is reached on NodePorts, and §5's secure-context problem forces TALOS_HOST=localhost plus five SSH forwards. Setting public.domain puts a Caddy deployment in front of it that terminates TLS for real hostnames (ADR-0035), and the forwards go away.

Once, outside the cluster

DNS — five A records to the host's public address (the cluster is behind it):

register.<domain>  mijn.<domain>  behandel.<domain>  beheer.<domain>  auth.<domain>  →  46.224.220.37

The host's forward — the public IP is on the Fedora host, so it has to hand 80/443 to the node. This is the only host-side configuration, and it is dumb layer 4:

sudo firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32080
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=443:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32443
sudo firewall-cmd --permanent --zone=public --add-masquerade
sudo firewall-cmd --reload

--add-masquerade is what makes the return path work: without it the node answers the client's address directly and the reply never goes back through the host.

A StorageClass. Caddy's certificates live in /data, which is an emptyDir unless persistence.storageClass is set (§6). Let's Encrypt allows five duplicate certificates per week, so on an emptyDir a handful of pod restarts leaves the edge serving an untrusted certificate until the limit resets. Install local-path first (§6).

Deploy

make k8s-up TALOS_HOST=<domain-facing name> K8S_REGISTRY=<TALOS_VM_IP>:30500 \
  K8S_SET='--set public.domain=<domain> --set public.email=<ops address> --set persistence.storageClass=local-path'

public.domain is the only switch: with it empty nothing in templates/edge.yaml renders and the stack behaves exactly as §4 describes. With it set, KC_HOSTNAME and the portals' config.json both become https://auth.<domain> — one helper builds both, so the issuer and the authority cannot drift (ADR-0010).

Watch the first certificate being issued:

kubectl -n big logs deploy/caddy-edge -f        # "certificate obtained successfully"
curl -sSI https://register.<domain>/openbaar/register | head -1

When it doesn't work

Symptom	Cause
ACME fails with connection refused or a timeout on the HTTP-01 challenge	the host's 80 → 32080 forward is missing, or --add-masquerade is
ACME fails with NXDOMAIN / no such host	the A record isn't there yet. Caddy retries with backoff; fix DNS and it recovers
An untrusted certificate after several restarts	the Let's Encrypt duplicate limit, from certificates on an emptyDir — see above
The portal loads but login bounces back logged out	public.domain changed without the portals rolling. The chart hashes the issuer into their pod template, so helm upgrade should do it — check kubectl -n big describe deploy/self-service
404 from the edge on a name that should work	the name isn't in public.routes; Caddy answers 404 for a Host it has no site block for

What is not ported

• Observability (Tempo, Prometheus, Grafana) is defined but disabled — those are built images too, so switching them on means pushing them as well: K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'. The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
• The verify/e2e lanes. make verify* and the Playwright e2e drive compose, not the chart. The Kubernetes path is verified with §5's smoke test. CI's k8s job runs the two clusterless checks (k8s-lint, k8s-drift) on every PR — a values typo or a compose image bump that skipped the chart fails there, but nothing deploys the chart in CI.
• Ingress, TLS, and resource requests. See the ponytail ceiling in ADR-0033.