register-referentie/docs/runbooks/kubernetes-talos.md

# Deploying the stack to a single-node Talos cluster

The Helm chart in `infra/helm/big-reference` is a port of `infra/docker-compose.yml`
(ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up
on a Talos VM under virt-manager on a laptop, including the parts that bite.

Compose remains the CI-canonical stack — `make verify`, the acceptance lane and the
Playwright e2e all still drive it. Kubernetes is a second deployment target.

## 0. What you need

On the laptop, four static binaries, all installable to `~/.local/bin` without root:

```bash
curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64
curl -sSLo ~/.local/bin/kubectl   https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane
chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane}
```

Match `talosctl` to the Talos ISO you booted (`talosctl version --insecure -n <ip>` reports
the server's tag). `crane` is what pushes images to a plain-HTTP registry without a
root-level Docker daemon change — see §2.

**VM sizing.** 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare
(measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The
chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the
only budget there is. Resize a stopped VM with:

```bash
virsh -c qemu:///system destroy talos      # it's in maintenance mode; nothing is lost
virsh -c qemu:///system setmaxmem talos 10G --config
virsh -c qemu:///system setmem    talos 10G --config
virsh -c qemu:///system setvcpus  talos 6 --config --maximum
virsh -c qemu:///system setvcpus  talos 6 --config
```

Two addresses matter throughout:

| Name | Meaning | Example |
|---|---|---|
| `TALOS_HOST` | the VM's IP — used by the browser, `talosctl` and `kubectl` | `192.168.122.33` |
| `K8S_REGISTRY` | `TALOS_HOST:30500` — the in-cluster registry (§2) | `192.168.122.33:30500` |

Find the VM's address with `virsh -c qemu:///system net-dhcp-leases default`.

## 1. Install Talos onto the VM

### The virt-manager trap

virt-manager treats the install ISO as one-shot: on the VM's **first shutdown** it ejects
the CD and rewrites the boot order to `hd`. A Talos VM booted from `metal-amd64.iso` runs
entirely in RAM, so the disk is still empty — the next start lands on
`Boot failed: not a bootable disk`. Put the ISO back before installing:

```bash
virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert
virt-xml -c qemu:///system talos --edit --boot cdrom,hd
virsh -c qemu:///system start talos
```

Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio
it is `/dev/vda`, and Talos's default selector expects `/dev/sda`:

```bash
talosctl get disks --insecure -n <TALOS_HOST> -e <TALOS_HOST>
```

### Generate the machine config

Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular
`machine.install` is now `UnattendedInstallConfig`, and patching the old field is rejected
with *"UnattendedInstallConfig config is incompatible with v1alpha1 config"*. Write
`patch.yaml` as a multi-document patch:

```yaml
machine:
  certSANs:
    - 192.168.122.33
  registries:
    mirrors:
      # The in-cluster registry (§2) speaks plain HTTP.
      "192.168.122.33:30500":
        endpoints:
          - http://192.168.122.33:30500
---
apiVersion: v1alpha1
kind: UnattendedInstallConfig
provisioning:
  diskSelector:
    match: disk.dev_path == "/dev/vda"
```

```bash
talosctl gen config big https://<TALOS_HOST>:6443 --output-dir ~/.talos/big --config-patch @patch.yaml
talosctl apply-config --insecure -n <TALOS_HOST> -e <TALOS_HOST> --file ~/.talos/big/controlplane.yaml
```

Talos installs to the disk and **kexecs straight into the installed system**, so the CD
boot order doesn't get in the way here. Then point the client at the node and bootstrap:

```bash
talosctl config merge ~/.talos/big/talosconfig
talosctl config endpoint <TALOS_HOST>
talosctl config node <TALOS_HOST>
talosctl bootstrap                      # wait for `talosctl version` to answer first
talosctl kubeconfig -f ~/.kube/config
```

A single-node cluster must run workloads on the control plane, or CoreDNS never schedules:

```bash
kubectl taint node --all node-role.kubernetes.io/control-plane-
```

Finally, make a VM restart boot the installed system rather than the ISO (takes effect at
the next full power cycle):

```bash
virsh -c qemu:///system change-media talos sda --eject --config
virt-xml -c qemu:///system talos --edit --boot hd
```

## 2. A registry the node can pull from

Talos has no Docker daemon and no way to side-load an image, so this repo's images have to
come from a registry. The registry runs **inside the cluster**, published on NodePort
30500 (`infra/helm/registry.yaml`):

```bash
make k8s-registry
```

Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port
opened on firewalld's `libvirt` zone (`sudo firewall-cmd --zone=libvirt --add-port=5000/tcp`),
which needs root. Pushing from the laptop *to* the node is outbound and always allowed, and
the node pulls from its own NodePort. If you do open that port, put a registry on the
laptop instead and point `K8S_REGISTRY` at `<laptop-ip>:5000` — the mirror patch in §1 has
an entry ready for it.

Its storage is `emptyDir`, so if the registry pod is ever replaced, re-run `make k8s-images`.

## 3. Build and push the images

```bash
make k8s-images K8S_REGISTRY=<TALOS_HOST>:30500
```

This builds the nine images with `docker compose build` — same contexts and Dockerfiles as
compose, no second build definition — then `docker save | crane push --insecure` each one.
`docker push` is not used: the registry speaks plain HTTP, which the Docker daemon refuses
without a root-level `insecure-registries` entry, while crane just takes `--insecure`.

## 4. Deploy

```bash
make k8s-up TALOS_HOST=<TALOS_HOST> K8S_REGISTRY=<TALOS_HOST>:30500
```

That does two things:

1. `make k8s-seed` — creates the ConfigMaps the chart mounts, from the config files that
   already live in this repo (`infra/helm/seed-configmaps.sh`): the four
   `setup_configuration/data.yaml` files, the Keycloak realm exports, the BPMN + DMN, and
   the two bootstrap scripts. Re-run it after editing any of them.
2. `helm upgrade --install` of the chart into namespace `big`.

First bring-up takes a few minutes: the four Django services migrate their databases and
apply their `setup_configuration`, Flowable creates its schema, and the bootstrap Jobs
deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement.

```bash
kubectl -n big get pods -w
kubectl -n big get jobs        # all four must reach COMPLETIONS 1/1
```

The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere
specific:

| Job | What breaks without it |
|---|---|
| `flowable-init` | no `registratie` process, no diploma DMN |
| `registerrecord-init` | the register has no RegisterRecord objecttype, so writes are refused |
| `seed-zaaktype` | the ACL can't resolve `BIG-REGISTRATIE`, so no zaak is created |
| `nrc-subscribe` | register writes never reach the projection — the public register stays empty |

## 5. Use it

### The portals must be reached over `localhost`

The portals' OIDC flow uses PKCE, which needs `crypto.subtle` — and browsers only expose
that in a **secure context**: HTTPS, or an origin on `localhost`/`127.0.0.1`. A NodePort on
the VM's IP is neither, so `http://<TALOS_HOST>:30140` fails before it can even build the
authorize URL:

```
ERROR TypeError: Cannot read properties of undefined (reading 'digest')
    at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize
```

So deploy with `TALOS_HOST=localhost` — which pins Keycloak's issuer and the portals'
`config.json` authority to `http://localhost:30180` — and forward the browser-facing
services to those same ports:

```bash
make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500
make k8s-portals          # stays in the foreground; Ctrl-C stops all five forwards
```

| URL (needs `make k8s-portals`) | What |
|---|---|
| `http://localhost:30140` | self-service portal (DigiD) |
| `http://localhost:30141` | openbaar register (anonymous) |
| `http://localhost:30142` | behandel portal (medewerker) |
| `http://localhost:30143` | beheer portal (medewerker) |
| `http://localhost:30180` | Keycloak (admin/admin) |

The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed
string, so the port the browser uses has to match the one baked into `config.json`.

This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
`localhost:8180`); only the addresses differ.

All of this is what §10 removes: with a public domain the portals have real certificates,
so the browser gets its secure context and no forwarding is involved.

### The admin UIs work straight off the NodePorts

These are server-rendered and need no secure context, so they are reachable at the VM's
address with no forwarding:

| URL | What |
|---|---|
| `http://<TALOS_HOST>:30000` | OpenZaak admin (admin/admin) |
| `http://<TALOS_HOST>:30001` | Open Notificaties admin (admin/admin) |
| `http://<TALOS_HOST>:30020` / `:30021` | Objecttypen / Objecten admin |
| `http://<TALOS_HOST>:30080` | BFF (`/health`) |
| `http://<TALOS_HOST>:30090` | Flowable REST (rest-admin/test) |

### Credentials

Log in with the test users from `docs/synthetic-data.md` (all password `test123`, e.g.
`jan-burger` for self-service, `merel-behandelaar` for behandel). The `medewerker` realm
enforces MFA (ADR-0031) — print a current code with
`python3 infra/keycloak/check_realms.py otp`. Walk the flow in `docs/demo-script.md`.

`TALOS_HOST` is not cosmetic: it pins Keycloak's issuer (`KC_HOSTNAME`) and the portals'
OIDC authority to the same string, which is what makes a browser token pass the BFF's
validation (ADR-0010). Change it and you must re-run `make k8s-up` — the chart rolls the
portals for you, because their `config.json` is a subPath mount and would otherwise keep
serving the old authority.

### Smoke-test the whole chain without a browser

With the forwards running:

```bash
TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \
  -d grant_type=password -d client_id=big-portal \
  -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token)

# through the portal's Caddy, so this also proves the BFF reverse proxy
curl -s -X POST http://localhost:30140/self-service/registrations \
  -H "Authorization: Bearer $TOK" -H 'Content-Length: 0'
# → {"registrationId":"…","status":"Ingediend"}

curl -s http://localhost:30141/openbaar/register
# → [{"id":"…","status":"INGEDIEND","reference":"<the registrationId>"}]
```

The second call proves the whole Common Ground path: portal → BFF → domain → Flowable →
ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register.

## 6. Keeping the databases (recommended if you iterate on the chart)

By default every database is an `emptyDir`: no CSI driver needed, and the data lives as
long as the pod. Note what that means in practice — **any** change to a database pod's
template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack
then needs its bootstrap re-run:

```bash
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...
```

which re-runs the four Jobs *and* restarts `event-subscriber` + `projection-api`, because
those two create the projection schema on start and otherwise keep writing to a
schema-less database (`relation "processed_notifications" does not exist`). For persistence, install Rancher's local-path-provisioner — on Talos it
must write under `/var` and its namespace needs the privileged Pod Security label:

```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31
patches:
  - patch: |-
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: local-path-config
        namespace: local-path-storage
      data:
        config.json: |-
          { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] }
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: local-path-storage
        labels:
          pod-security.kubernetes.io/enforce: privileged
```

```bash
kubectl apply -k .
make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path'
```

The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the data behind;
`make k8s-purge` drops the namespace and with it the volumes.

## 7. Day-to-day

```bash
make k8s-lint                      # render + schema-check the chart, no cluster needed
make k8s-drift                     # fail if compose and the chart describe different stacks
make k8s-portals                   # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=...   # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
make k8s-seed                      # after editing a data.yaml, a realm export, or the BPMN
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...   # re-run the bootstrap Jobs + reset the projection schema
make k8s-down                      # uninstall, keep the database PVCs
make k8s-purge                     # uninstall and drop the namespace
```

This repo's images are pulled with `imagePullPolicy: Always` (the `dev` tag is mutable), so
`kubectl -n big rollout restart deploy/<name>` after `make k8s-images` picks up a rebuild.
Upstream images stay `IfNotPresent`: their tags are pinned, and keeping them out of the pod
template avoids needless churn — a changed template makes a Job unpatchable.

`k8s-reseed` is also the path for *changing* a Job in the chart: a Job's pod template is
immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`.

## 8. When it doesn't work

| Symptom | Cause |
|---|---|
| `Boot failed: not a bootable disk` | virt-manager ejected the install ISO on first shutdown — see §1 |
| The VM comes back in maintenance mode after a restart | the ISO is still attached and boots first; eject it and set `--boot hd` (§1) |
| `apply-config` rejects the patch with *"incompatible with v1alpha1"* | Talos ≥1.14 owns that field in its own config document — patch the document, not `machine.*` (§1) |
| CoreDNS `Pending` forever | the control-plane taint is still on the only node (§1) |
| `ImagePullBackOff` … `pull QPS exceeded` | transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry |
| `ImagePullBackOff` on a `register-referentie/*` image | the registry mirror patch is missing: `talosctl get registriesconfig` |
| Pod stuck in `ContainerCreating`, event names a ConfigMap | `make k8s-seed` |
| `seed-zaaktype` retrying | publishing a zaaktype validates the resultaattype against `selectielijst.openzaak.nl`, so this one Job needs outbound internet from the VM (ADR-0006) |
| `TypeError: Cannot read properties of undefined (reading 'digest')` on a portal | not a secure context: `crypto.subtle` is absent on `http://<ip>`. Use `localhost` + `make k8s-portals` (§5) |
| Login redirects but the portal stays logged out, or the BFF answers 401 | `TALOS_HOST` doesn't match the address in the browser's URL bar — issuer mismatch. Re-run `make k8s-up` with the right value |
| A portal returns 502 on `/self-service/…` | the BFF is unreachable from the portal pod: check `kubectl -n big get svc bff` and the BFF's own readiness |
| Public register empty after a submit | usually a wiped `emptyDir` database (§6): `make k8s-reseed`. Confirm with `kubectl -n big logs deploy/event-subscriber \| grep 42P01` |
| `helm upgrade` fails with `cannot patch … with kind Job` | see §7 — use `make k8s-reseed` |
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |

## 10. Publishing it on a public domain

By default the stack has no hostname: it is reached on NodePorts, and §5's secure-context
problem forces `TALOS_HOST=localhost` plus five SSH forwards. Setting `public.domain` puts a
Caddy deployment in front of it that terminates TLS for real hostnames (ADR-0035), and the
forwards go away.

### Once, outside the cluster

**DNS** — five A records to the *host's* public address (the cluster is behind it):

```
register.<domain>  mijn.<domain>  behandel.<domain>  beheer.<domain>  auth.<domain>  →  46.224.220.37
```

**The host's forward** — the public IP is on the Fedora host, so it has to hand 80/443 to
the node. This is the only host-side configuration, and it is dumb layer 4:

```bash
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32080
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=443:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32443
sudo firewall-cmd --permanent --zone=public --add-masquerade
sudo firewall-cmd --reload
```

`--add-masquerade` is what makes the return path work: without it the node answers the
client's address directly and the reply never goes back through the host.

**A StorageClass.** Caddy's certificates live in `/data`, which is an `emptyDir` unless
`persistence.storageClass` is set (§6). Let's Encrypt allows five duplicate certificates per
week, so on an `emptyDir` a handful of pod restarts leaves the edge serving an untrusted
certificate until the limit resets. Install local-path first (§6).

### Deploy

```bash
make k8s-up TALOS_HOST=<domain-facing name> K8S_REGISTRY=<TALOS_VM_IP>:30500 \
  K8S_SET='--set public.domain=<domain> --set public.email=<ops address> --set persistence.storageClass=local-path'
```

`public.domain` is the only switch: with it empty nothing in `templates/edge.yaml` renders
and the stack behaves exactly as §4 describes. With it set, `KC_HOSTNAME` and the portals'
`config.json` both become `https://auth.<domain>` — one helper builds both, so the issuer
and the authority cannot drift (ADR-0010).

Watch the first certificate being issued:

```bash
kubectl -n big logs deploy/caddy-edge -f        # "certificate obtained successfully"
curl -sSI https://register.<domain>/openbaar/register | head -1
```

### When it doesn't work

| Symptom | Cause |
|---|---|
| ACME fails with `connection refused` or a timeout on the HTTP-01 challenge | the host's 80 → 32080 forward is missing, or `--add-masquerade` is |
| ACME fails with `NXDOMAIN` / `no such host` | the A record isn't there yet. Caddy retries with backoff; fix DNS and it recovers |
| An untrusted certificate after several restarts | the Let's Encrypt duplicate limit, from certificates on an `emptyDir` — see above |
| The portal loads but login bounces back logged out | `public.domain` changed without the portals rolling. The chart hashes the issuer into their pod template, so `helm upgrade` should do it — check `kubectl -n big describe deploy/self-service` |
| `404` from the edge on a name that should work | the name isn't in `public.routes`; Caddy answers 404 for a Host it has no site block for |

## What is not ported

- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
  images too, so switching them on means pushing them as well:
  `K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
  The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
  chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
  clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
  image bump that skipped the chart fails there, but nothing deploys the chart in CI.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.