register-referentie/docs/runbooks/kubernetes-talos.md

Deploying the stack to a single-node Talos cluster

The Helm chart in infra/helm/big-reference is a port of infra/docker-compose.yml (ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up on a Talos VM under virt-manager on a laptop, including the parts that bite.

Compose remains the CI-canonical stack — make verify, the acceptance lane and the Playwright e2e all still drive it. Kubernetes is a second deployment target.

0. What you need

On the laptop, four static binaries, all installable to ~/.local/bin without root:

curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64
curl -sSLo ~/.local/bin/kubectl   https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane
chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane}

Match talosctl to the Talos ISO you booted (talosctl version --insecure -n <ip> reports the server's tag). crane is what pushes images to a plain-HTTP registry without a root-level Docker daemon change — see §2.

VM sizing. 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare (measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the only budget there is. Resize a stopped VM with:

virsh -c qemu:///system destroy talos      # it's in maintenance mode; nothing is lost
virsh -c qemu:///system setmaxmem talos 10G --config
virsh -c qemu:///system setmem    talos 10G --config
virsh -c qemu:///system setvcpus  talos 6 --config --maximum
virsh -c qemu:///system setvcpus  talos 6 --config

Two addresses matter throughout:

Name	Meaning	Example
TALOS_HOST	the VM's IP — used by the browser, talosctl and kubectl	192.168.122.33
K8S_REGISTRY	TALOS_HOST:30500 — the in-cluster registry (§2)	192.168.122.33:30500

Find the VM's address with virsh -c qemu:///system net-dhcp-leases default.

1. Install Talos onto the VM

The virt-manager trap

virt-manager treats the install ISO as one-shot: on the VM's first shutdown it ejects the CD and rewrites the boot order to hd. A Talos VM booted from metal-amd64.iso runs entirely in RAM, so the disk is still empty — the next start lands on Boot failed: not a bootable disk. Put the ISO back before installing:

virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert
virt-xml -c qemu:///system talos --edit --boot cdrom,hd
virsh -c qemu:///system start talos

Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio it is /dev/vda, and Talos's default selector expects /dev/sda:

talosctl get disks --insecure -n <TALOS_HOST> -e <TALOS_HOST>

Generate the machine config

Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular machine.install is now UnattendedInstallConfig, and patching the old field is rejected with "UnattendedInstallConfig config is incompatible with v1alpha1 config". Write patch.yaml as a multi-document patch:

machine:
  certSANs:
    - 192.168.122.33
  registries:
    mirrors:
      # The in-cluster registry (§2) speaks plain HTTP.
      "192.168.122.33:30500":
        endpoints:
          - http://192.168.122.33:30500
---
apiVersion: v1alpha1
kind: UnattendedInstallConfig
provisioning:
  diskSelector:
    match: disk.dev_path == "/dev/vda"

talosctl gen config big https://<TALOS_HOST>:6443 --output-dir ~/.talos/big --config-patch @patch.yaml
talosctl apply-config --insecure -n <TALOS_HOST> -e <TALOS_HOST> --file ~/.talos/big/controlplane.yaml

Talos installs to the disk and kexecs straight into the installed system, so the CD boot order doesn't get in the way here. Then point the client at the node and bootstrap:

talosctl config merge ~/.talos/big/talosconfig
talosctl config endpoint <TALOS_HOST>
talosctl config node <TALOS_HOST>
talosctl bootstrap                      # wait for `talosctl version` to answer first
talosctl kubeconfig -f ~/.kube/config

A single-node cluster must run workloads on the control plane, or CoreDNS never schedules:

kubectl taint node --all node-role.kubernetes.io/control-plane-

Finally, make a VM restart boot the installed system rather than the ISO (takes effect at the next full power cycle):

virsh -c qemu:///system change-media talos sda --eject --config
virt-xml -c qemu:///system talos --edit --boot hd

2. A registry the node can pull from

Talos has no Docker daemon and no way to side-load an image, so this repo's images have to come from a registry. The registry runs inside the cluster, published on NodePort 30500 (infra/helm/registry.yaml):

make k8s-registry

Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port opened on firewalld's libvirt zone (sudo firewall-cmd --zone=libvirt --add-port=5000/tcp), which needs root. Pushing from the laptop to the node is outbound and always allowed, and the node pulls from its own NodePort. If you do open that port, put a registry on the laptop instead and point K8S_REGISTRY at <laptop-ip>:5000 — the mirror patch in §1 has an entry ready for it.

Its storage is emptyDir, so if the registry pod is ever replaced, re-run make k8s-images.

3. Build and push the images

make k8s-images K8S_REGISTRY=<TALOS_HOST>:30500

This builds the nine images with docker compose build — same contexts and Dockerfiles as compose, no second build definition — then docker save | crane push --insecure each one. docker push is not used: the registry speaks plain HTTP, which the Docker daemon refuses without a root-level insecure-registries entry, while crane just takes --insecure.

4. Deploy

make k8s-up TALOS_HOST=<TALOS_HOST> K8S_REGISTRY=<TALOS_HOST>:30500

That does two things:

1. make k8s-seed — creates the ConfigMaps the chart mounts, from the config files that already live in this repo (infra/helm/seed-configmaps.sh): the four setup_configuration/data.yaml files, the Keycloak realm exports, the BPMN + DMN, and the two bootstrap scripts. Re-run it after editing any of them.
2. helm upgrade --install of the chart into namespace big.

First bring-up takes a few minutes: the four Django services migrate their databases and apply their setup_configuration, Flowable creates its schema, and the bootstrap Jobs deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement.

kubectl -n big get pods -w
kubectl -n big get jobs        # all four must reach COMPLETIONS 1/1

The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere specific:

Job	What breaks without it
flowable-init	no registratie process, no diploma DMN
registerrecord-init	the register has no RegisterRecord objecttype, so writes are refused
seed-zaaktype	the ACL can't resolve BIG-REGISTRATIE, so no zaak is created
nrc-subscribe	register writes never reach the projection — the public register stays empty

5. Use it

The portals must be reached over localhost

The portals' OIDC flow uses PKCE, which needs crypto.subtle — and browsers only expose that in a secure context: HTTPS, or an origin on localhost/127.0.0.1. A NodePort on the VM's IP is neither, so http://<TALOS_HOST>:30140 fails before it can even build the authorize URL:

ERROR TypeError: Cannot read properties of undefined (reading 'digest')
    at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize

So deploy with TALOS_HOST=localhost — which pins Keycloak's issuer and the portals' config.json authority to http://localhost:30180 — and forward the browser-facing services to those same ports:

make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500
make k8s-portals          # stays in the foreground; Ctrl-C stops all five forwards

URL (needs make k8s-portals)	What
http://localhost:30140	self-service portal (DigiD)
http://localhost:30141	openbaar register (anonymous)
http://localhost:30142	behandel portal (medewerker)
http://localhost:30143	beheer portal (medewerker)
http://localhost:30180	Keycloak (admin/admin)

The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed string, so the port the browser uses has to match the one baked into config.json.

This is the same mechanism infra/host-browser.yml uses for the compose stack (which pins localhost:8180); only the addresses differ.

The admin UIs work straight off the NodePorts

These are server-rendered and need no secure context, so they are reachable at the VM's address with no forwarding:

URL	What
http://<TALOS_HOST>:30000	OpenZaak admin (admin/admin)
http://<TALOS_HOST>:30001	Open Notificaties admin (admin/admin)
http://<TALOS_HOST>:30020 / :30021	Objecttypen / Objecten admin
http://<TALOS_HOST>:30080	BFF (/health)
http://<TALOS_HOST>:30090	Flowable REST (rest-admin/test)

Credentials

Log in with the test users from docs/synthetic-data.md (all password test123, e.g. jan-burger for self-service, merel-behandelaar for behandel). The medewerker realm enforces MFA (ADR-0031) — print a current code with python3 infra/keycloak/check_realms.py otp. Walk the flow in docs/demo-script.md.

TALOS_HOST is not cosmetic: it pins Keycloak's issuer (KC_HOSTNAME) and the portals' OIDC authority to the same string, which is what makes a browser token pass the BFF's validation (ADR-0010). Change it and you must re-run make k8s-up — the chart rolls the portals for you, because their config.json is a subPath mount and would otherwise keep serving the old authority.

Smoke-test the whole chain without a browser

With the forwards running:

TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \
  -d grant_type=password -d client_id=big-portal \
  -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token)

# through the portal's Caddy, so this also proves the BFF reverse proxy
curl -s -X POST http://localhost:30140/self-service/registrations \
  -H "Authorization: Bearer $TOK" -H 'Content-Length: 0'
# → {"registrationId":"…","status":"Ingediend"}

curl -s http://localhost:30141/openbaar/register
# → [{"id":"…","status":"INGEDIEND","reference":"<the registrationId>"}]

The second call proves the whole Common Ground path: portal → BFF → domain → Flowable → ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register.

6. Keeping the databases (recommended if you iterate on the chart)

By default every database is an emptyDir: no CSI driver needed, and the data lives as long as the pod. Note what that means in practice — any change to a database pod's template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack then needs its bootstrap re-run:

make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...

which re-runs the four Jobs and restarts event-subscriber + projection-api, because those two create the projection schema on start and otherwise keep writing to a schema-less database (relation "processed_notifications" does not exist). For persistence, install Rancher's local-path-provisioner — on Talos it must write under /var and its namespace needs the privileged Pod Security label:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31
patches:
  - patch: |-
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: local-path-config
        namespace: local-path-storage
      data:
        config.json: |-
          { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] }
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: local-path-storage
        labels:
          pod-security.kubernetes.io/enforce: privileged

kubectl apply -k .
make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path'

The PVCs carry helm.sh/resource-policy: keep, so make k8s-down leaves the data behind; make k8s-purge drops the namespace and with it the volumes.

7. Day-to-day

make k8s-lint                      # render + schema-check the chart, no cluster needed
make k8s-drift                     # fail if compose and the chart describe different stacks
make k8s-portals                   # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=...   # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
make k8s-seed                      # after editing a data.yaml, a realm export, or the BPMN
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...   # re-run the bootstrap Jobs + reset the projection schema
make k8s-down                      # uninstall, keep the database PVCs
make k8s-purge                     # uninstall and drop the namespace

This repo's images are pulled with imagePullPolicy: Always (the dev tag is mutable), so kubectl -n big rollout restart deploy/<name> after make k8s-images picks up a rebuild. Upstream images stay IfNotPresent: their tags are pinned, and keeping them out of the pod template avoids needless churn — a changed template makes a Job unpatchable.

k8s-reseed is also the path for changing a Job in the chart: a Job's pod template is immutable, so helm upgrade is rejected with cannot patch "…" with kind Job.

8. When it doesn't work

Symptom	Cause
Boot failed: not a bootable disk	virt-manager ejected the install ISO on first shutdown — see §1
The VM comes back in maintenance mode after a restart	the ISO is still attached and boots first; eject it and set --boot hd (§1)
apply-config rejects the patch with "incompatible with v1alpha1"	Talos ≥1.14 owns that field in its own config document — patch the document, not machine.* (§1)
CoreDNS Pending forever	the control-plane taint is still on the only node (§1)
ImagePullBackOff … pull QPS exceeded	transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry
ImagePullBackOff on a register-referentie/* image	the registry mirror patch is missing: talosctl get registriesconfig
Pod stuck in ContainerCreating, event names a ConfigMap	make k8s-seed
seed-zaaktype retrying	publishing a zaaktype validates the resultaattype against selectielijst.openzaak.nl, so this one Job needs outbound internet from the VM (ADR-0006)
TypeError: Cannot read properties of undefined (reading 'digest') on a portal	not a secure context: crypto.subtle is absent on http://<ip>. Use localhost + make k8s-portals (§5)
Login redirects but the portal stays logged out, or the BFF answers 401	TALOS_HOST doesn't match the address in the browser's URL bar — issuer mismatch. Re-run make k8s-up with the right value
A portal returns 502 on /self-service/…	the BFF is unreachable from the portal pod: check kubectl -n big get svc bff and the BFF's own readiness
Public register empty after a submit	usually a wiped emptyDir database (§6): make k8s-reseed. Confirm with kubectl -n big logs deploy/event-subscriber | grep 42P01
helm upgrade fails with cannot patch … with kind Job	see §7 — use make k8s-reseed
Pods Evicted / OOMKilled	the VM is too small (§0)
A Job shows BackoffLimitExceeded	read it: kubectl -n big logs job/<name>

9. Deploying on merge to main

.gitea/workflows/deploy.yaml runs the §3–§4 steps against the lab server's Talos VM every time a PR is squash-merged to main (and on demand via Run workflow). PR CI is the merge gate, so the workflow deploys without re-running the checks.

The cluster's API and registry are not exposed publicly, so the job forwards them over the same SSH hop the Gitea-runner pipeline uses:

ssh -p 6667 user@labs.respellion.tech  -L 6443 -L 30500 -L 30141  →  <TALOS_VM_IP>

Consequences worth knowing:

• Images are pushed to localhost:30500 (the tunnel) and pulled by the node from <TALOS_VM_IP>:30500 (its own NodePort, the address in the Talos registry-mirror patch). Same registry, two names — hence the two K8S_REGISTRY values in the workflow.
• It calls make k8s-reseed, not make k8s-up: the bootstrap Jobs are idempotent, and deleting them first is what stops a changed Job template from wedging helm upgrade (§7).
• dev is a mutable tag, so a rollout restart of the nine repo deployments is what actually puts the new images in the pods.
• Deploys queue (cancel-in-progress: false): a helm upgrade killed half-way leaves the release in pending-upgrade, which has to be unwedged by hand.

Settings, all on the repository in Gitea:

Kind	Name	What
Secret	TALOS_SSH_KEY	private key for user@labs.respellion.tech (the Fedora host)
Secret	TALOS_KUBECONFIG	base64 of the kubeconfig, server: https://127.0.0.1:6443 — Talos puts 127.0.0.1 in the apiserver cert SANs, so TLS still verifies through the tunnel
Variable	TALOS_VM_IP	the VM's libvirt address (default 192.168.122.173)
Variable	TALOS_HOST	the browser-facing host baked into Keycloak's issuer (default localhost, see §5)

The last step smokes GET /openbaar/register through the openbaar portal, which exercises portal → Caddy → BFF → projection. An empty register passes; a 502 does not.

Not covered: the portals still need make k8s-portals (or an SSH forward) to be usable in a browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the upgrade path.

Publishing through the labs Caddy

The portals can be reached on real hostnames through the Caddy that already fronts *.labs.respellion.tech (repo Infra, infra/development/). The chain:

browser → Caddy (labs server, TLS) → openssh-server:3014x/30180
        → reverse SSH tunnel → Fedora host → <TALOS_VM_IP>:3014x/30180 (NodePorts)

URL	NodePort
https://big-register.labs.respellion.tech	30141 openbaar
https://big-mijn.labs.respellion.tech	30140 self-service
https://big-behandel.labs.respellion.tech	30142 behandel
https://big-beheer.labs.respellion.tech	30143 beheer
https://big-auth.labs.respellion.tech	30180 Keycloak (/admin blocked)

HTTPS makes the portals a secure context, so PKCE works without port-forwards — but Keycloak's issuer must be the public origin. Deploy with it:

make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500 \
  K8S_SET="--set keycloakUrl=https://big-auth.labs.respellion.tech"

For deploy-on-merge, set the repository variable KEYCLOAK_URL to the same value. With it set, the localhost port-forwards (§5) no longer log in: the issuer is one string.

Staff logins still hit the enforced OTP step. For a demo, set the repository variable OTP_AUTOFILL=true (chart value demo.otpAutofill): Keycloak then uses the big-demo theme, which fills in and submits the code from the fixture secret, so the step is visible but needs no authenticator. Keycloak restarts when the value flips. Demo only — the secret is committed.

The theme lives in infra/keycloak/themes/big-demo/ and is seeded as the rr-kc-theme ConfigMap by infra/helm/seed-configmaps.sh on every deploy. Keycloak runs start-dev, which doesn't cache themes, so an edit shows up about a minute after the ConfigMap changes. A new theme file also needs a key in the seed script and a path in the keycloak files in values.yaml.

One-time setup:

1. Fedora host: install infra/development/big-portals-tunnel.service from the Infra repo (instructions in the file).
2. Labs server: deploy the Infra Caddyfile + compose.yml (Caddy joins the openssh_default network to reach the tunnel ends).

What is not ported

• Observability (Tempo, Prometheus, Grafana) is defined but disabled — those are built images too, so switching them on means pushing them as well: K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'. The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
• The verify/e2e lanes. make verify* and the Playwright e2e drive compose, not the chart. The Kubernetes path is verified with §5's smoke test. CI's k8s job runs the two clusterless checks (k8s-lint, k8s-drift) on every PR — a values typo or a compose image bump that skipped the chart fails there, but nothing deploys the chart in CI.
• Ingress, TLS, and resource requests. See the ponytail ceiling in ADR-0033.