# Deploying the stack to a single-node Talos cluster The Helm chart in `infra/helm/big-reference` is a port of `infra/docker-compose.yml` (ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up on a Talos VM under virt-manager on a laptop, including the parts that bite. Compose remains the CI-canonical stack — `make verify`, the acceptance lane and the Playwright e2e all still drive it. Kubernetes is a second deployment target. ## 0. What you need On the laptop, four static binaries, all installable to `~/.local/bin` without root: ```bash curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64 curl -sSLo ~/.local/bin/kubectl https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane} ``` Match `talosctl` to the Talos ISO you booted (`talosctl version --insecure -n ` reports the server's tag). `crane` is what pushes images to a plain-HTTP registry without a root-level Docker daemon change — see §2. **VM sizing.** 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare (measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the only budget there is. Resize a stopped VM with: ```bash virsh -c qemu:///system destroy talos # it's in maintenance mode; nothing is lost virsh -c qemu:///system setmaxmem talos 10G --config virsh -c qemu:///system setmem talos 10G --config virsh -c qemu:///system setvcpus talos 6 --config --maximum virsh -c qemu:///system setvcpus talos 6 --config ``` Two addresses matter throughout: | Name | Meaning | Example | |---|---|---| | `TALOS_HOST` | the VM's IP — used by the browser, `talosctl` and `kubectl` | `192.168.122.33` | | `K8S_REGISTRY` | `TALOS_HOST:30500` — the in-cluster registry (§2) | `192.168.122.33:30500` | Find the VM's address with `virsh -c qemu:///system net-dhcp-leases default`. ## 1. Install Talos onto the VM ### The virt-manager trap virt-manager treats the install ISO as one-shot: on the VM's **first shutdown** it ejects the CD and rewrites the boot order to `hd`. A Talos VM booted from `metal-amd64.iso` runs entirely in RAM, so the disk is still empty — the next start lands on `Boot failed: not a bootable disk`. Put the ISO back before installing: ```bash virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert virt-xml -c qemu:///system talos --edit --boot cdrom,hd virsh -c qemu:///system start talos ``` Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio it is `/dev/vda`, and Talos's default selector expects `/dev/sda`: ```bash talosctl get disks --insecure -n -e ``` ### Generate the machine config Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular `machine.install` is now `UnattendedInstallConfig`, and patching the old field is rejected with *"UnattendedInstallConfig config is incompatible with v1alpha1 config"*. Write `patch.yaml` as a multi-document patch: ```yaml machine: certSANs: - 192.168.122.33 registries: mirrors: # The in-cluster registry (§2) speaks plain HTTP. "192.168.122.33:30500": endpoints: - http://192.168.122.33:30500 --- apiVersion: v1alpha1 kind: UnattendedInstallConfig provisioning: diskSelector: match: disk.dev_path == "/dev/vda" ``` ```bash talosctl gen config big https://:6443 --output-dir ~/.talos/big --config-patch @patch.yaml talosctl apply-config --insecure -n -e --file ~/.talos/big/controlplane.yaml ``` Talos installs to the disk and **kexecs straight into the installed system**, so the CD boot order doesn't get in the way here. Then point the client at the node and bootstrap: ```bash talosctl config merge ~/.talos/big/talosconfig talosctl config endpoint talosctl config node talosctl bootstrap # wait for `talosctl version` to answer first talosctl kubeconfig -f ~/.kube/config ``` A single-node cluster must run workloads on the control plane, or CoreDNS never schedules: ```bash kubectl taint node --all node-role.kubernetes.io/control-plane- ``` Finally, make a VM restart boot the installed system rather than the ISO (takes effect at the next full power cycle): ```bash virsh -c qemu:///system change-media talos sda --eject --config virt-xml -c qemu:///system talos --edit --boot hd ``` ## 2. A registry the node can pull from Talos has no Docker daemon and no way to side-load an image, so this repo's images have to come from a registry. The registry runs **inside the cluster**, published on NodePort 30500 (`infra/helm/registry.yaml`): ```bash make k8s-registry ``` Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port opened on firewalld's `libvirt` zone (`sudo firewall-cmd --zone=libvirt --add-port=5000/tcp`), which needs root. Pushing from the laptop *to* the node is outbound and always allowed, and the node pulls from its own NodePort. If you do open that port, put a registry on the laptop instead and point `K8S_REGISTRY` at `:5000` — the mirror patch in §1 has an entry ready for it. Its storage is `emptyDir`, so if the registry pod is ever replaced, re-run `make k8s-images`. ## 3. Build and push the images ```bash make k8s-images K8S_REGISTRY=:30500 ``` This builds the nine images with `docker compose build` — same contexts and Dockerfiles as compose, no second build definition — then `docker save | crane push --insecure` each one. `docker push` is not used: the registry speaks plain HTTP, which the Docker daemon refuses without a root-level `insecure-registries` entry, while crane just takes `--insecure`. ## 4. Deploy ```bash make k8s-up TALOS_HOST= K8S_REGISTRY=:30500 ``` That does two things: 1. `make k8s-seed` — creates the ConfigMaps the chart mounts, from the config files that already live in this repo (`infra/helm/seed-configmaps.sh`): the four `setup_configuration/data.yaml` files, the Keycloak realm exports, the BPMN + DMN, and the two bootstrap scripts. Re-run it after editing any of them. 2. `helm upgrade --install` of the chart into namespace `big`. First bring-up takes a few minutes: the four Django services migrate their databases and apply their `setup_configuration`, Flowable creates its schema, and the bootstrap Jobs deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement. ```bash kubectl -n big get pods -w kubectl -n big get jobs # all four must reach COMPLETIONS 1/1 ``` The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere specific: | Job | What breaks without it | |---|---| | `flowable-init` | no `registratie` process, no diploma DMN | | `registerrecord-init` | the register has no RegisterRecord objecttype, so writes are refused | | `seed-zaaktype` | the ACL can't resolve `BIG-REGISTRATIE`, so no zaak is created | | `nrc-subscribe` | register writes never reach the projection — the public register stays empty | ## 5. Use it ### The portals must be reached over `localhost` The portals' OIDC flow uses PKCE, which needs `crypto.subtle` — and browsers only expose that in a **secure context**: HTTPS, or an origin on `localhost`/`127.0.0.1`. A NodePort on the VM's IP is neither, so `http://:30140` fails before it can even build the authorize URL: ``` ERROR TypeError: Cannot read properties of undefined (reading 'digest') at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize ``` So deploy with `TALOS_HOST=localhost` — which pins Keycloak's issuer and the portals' `config.json` authority to `http://localhost:30180` — and forward the browser-facing services to those same ports: ```bash make k8s-up TALOS_HOST=localhost K8S_REGISTRY=:30500 make k8s-portals # stays in the foreground; Ctrl-C stops all five forwards ``` | URL (needs `make k8s-portals`) | What | |---|---| | `http://localhost:30140` | self-service portal (DigiD) | | `http://localhost:30141` | openbaar register (anonymous) | | `http://localhost:30142` | behandel portal (medewerker) | | `http://localhost:30143` | beheer portal (medewerker) | | `http://localhost:30180` | Keycloak (admin/admin) | The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed string, so the port the browser uses has to match the one baked into `config.json`. This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins `localhost:8180`); only the addresses differ. ### The admin UIs work straight off the NodePorts These are server-rendered and need no secure context, so they are reachable at the VM's address with no forwarding: | URL | What | |---|---| | `http://:30000` | OpenZaak admin (admin/admin) | | `http://:30001` | Open Notificaties admin (admin/admin) | | `http://:30020` / `:30021` | Objecttypen / Objecten admin | | `http://:30080` | BFF (`/health`) | | `http://:30090` | Flowable REST (rest-admin/test) | ### Credentials Log in with the test users from `docs/synthetic-data.md` (all password `test123`, e.g. `jan-burger` for self-service, `merel-behandelaar` for behandel). The `medewerker` realm enforces MFA (ADR-0031) — print a current code with `python3 infra/keycloak/check_realms.py otp`. Walk the flow in `docs/demo-script.md`. `TALOS_HOST` is not cosmetic: it pins Keycloak's issuer (`KC_HOSTNAME`) and the portals' OIDC authority to the same string, which is what makes a browser token pass the BFF's validation (ADR-0010). Change it and you must re-run `make k8s-up` — the chart rolls the portals for you, because their `config.json` is a subPath mount and would otherwise keep serving the old authority. ### Smoke-test the whole chain without a browser With the forwards running: ```bash TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \ -d grant_type=password -d client_id=big-portal \ -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token) # through the portal's Caddy, so this also proves the BFF reverse proxy curl -s -X POST http://localhost:30140/self-service/registrations \ -H "Authorization: Bearer $TOK" -H 'Content-Length: 0' # → {"registrationId":"…","status":"Ingediend"} curl -s http://localhost:30141/openbaar/register # → [{"id":"…","status":"INGEDIEND","reference":""}] ``` The second call proves the whole Common Ground path: portal → BFF → domain → Flowable → ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register. ## 6. Keeping the databases (recommended if you iterate on the chart) By default every database is an `emptyDir`: no CSI driver needed, and the data lives as long as the pod. Note what that means in practice — **any** change to a database pod's template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack then needs its bootstrap re-run: ```bash make k8s-reseed TALOS_HOST=... K8S_REGISTRY=... ``` which re-runs the four Jobs *and* restarts `event-subscriber` + `projection-api`, because those two create the projection schema on start and otherwise keep writing to a schema-less database (`relation "processed_notifications" does not exist`). For persistence, install Rancher's local-path-provisioner — on Talos it must write under `/var` and its namespace needs the privileged Pod Security label: ```yaml # kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31 patches: - patch: |- kind: ConfigMap apiVersion: v1 metadata: name: local-path-config namespace: local-path-storage data: config.json: |- { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] } - patch: |- apiVersion: v1 kind: Namespace metadata: name: local-path-storage labels: pod-security.kubernetes.io/enforce: privileged ``` ```bash kubectl apply -k . make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path' ``` The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the data behind; `make k8s-purge` drops the namespace and with it the volumes. ## 7. Day-to-day ```bash make k8s-lint # render + schema-check the chart, no cluster needed make k8s-portals # forward the portals + Keycloak to localhost (browser access) make k8s-images K8S_REGISTRY=... # after changing a service or a portal make k8s-up TALOS_HOST=... K8S_REGISTRY=... make k8s-seed # after editing a data.yaml, a realm export, or the BPMN make k8s-reseed TALOS_HOST=... K8S_REGISTRY=... # re-run the bootstrap Jobs + reset the projection schema make k8s-down # uninstall, keep the database PVCs make k8s-purge # uninstall and drop the namespace ``` This repo's images are pulled with `imagePullPolicy: Always` (the `dev` tag is mutable), so `kubectl -n big rollout restart deploy/` after `make k8s-images` picks up a rebuild. Upstream images stay `IfNotPresent`: their tags are pinned, and keeping them out of the pod template avoids needless churn — a changed template makes a Job unpatchable. `k8s-reseed` is also the path for *changing* a Job in the chart: a Job's pod template is immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`. ## 8. When it doesn't work | Symptom | Cause | |---|---| | `Boot failed: not a bootable disk` | virt-manager ejected the install ISO on first shutdown — see §1 | | The VM comes back in maintenance mode after a restart | the ISO is still attached and boots first; eject it and set `--boot hd` (§1) | | `apply-config` rejects the patch with *"incompatible with v1alpha1"* | Talos ≥1.14 owns that field in its own config document — patch the document, not `machine.*` (§1) | | CoreDNS `Pending` forever | the control-plane taint is still on the only node (§1) | | `ImagePullBackOff` … `pull QPS exceeded` | transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry | | `ImagePullBackOff` on a `register-referentie/*` image | the registry mirror patch is missing: `talosctl get registriesconfig` | | Pod stuck in `ContainerCreating`, event names a ConfigMap | `make k8s-seed` | | `seed-zaaktype` retrying | publishing a zaaktype validates the resultaattype against `selectielijst.openzaak.nl`, so this one Job needs outbound internet from the VM (ADR-0006) | | `TypeError: Cannot read properties of undefined (reading 'digest')` on a portal | not a secure context: `crypto.subtle` is absent on `http://`. Use `localhost` + `make k8s-portals` (§5) | | Login redirects but the portal stays logged out, or the BFF answers 401 | `TALOS_HOST` doesn't match the address in the browser's URL bar — issuer mismatch. Re-run `make k8s-up` with the right value | | A portal returns 502 on `/self-service/…` | the BFF is unreachable from the portal pod: check `kubectl -n big get svc bff` and the BFF's own readiness | | Public register empty after a submit | usually a wiped `emptyDir` database (§6): `make k8s-reseed`. Confirm with `kubectl -n big logs deploy/event-subscriber \| grep 42P01` | | `helm upgrade` fails with `cannot patch … with kind Job` | see §7 — use `make k8s-reseed` | | Pods `Evicted` / `OOMKilled` | the VM is too small (§0) | | A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/` | ## What is not ported - **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built images too, so switching them on means pushing them as well: `K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`. The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent. - **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the chart. The Kubernetes path is verified with §5's smoke test. - **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.