register-referentie/docs/runbooks/kubernetes-talos.md

# Deploying the stack to a single-node Talos cluster

The Helm chart in `infra/helm/big-reference` is a port of `infra/docker-compose.yml`
(ADR-0033). This runbook is the walkthrough that was actually used to bring the stack up
on a Talos VM under virt-manager on a laptop, including the parts that bite.

Compose remains the CI-canonical stack — `make verify`, the acceptance lane and the
Playwright e2e all still drive it. Kubernetes is a second deployment target.

## 0. What you need

On the laptop, four static binaries, all installable to `~/.local/bin` without root:

```bash
curl -sSLo ~/.local/bin/talosctl https://github.com/siderolabs/talos/releases/download/v1.14.0/talosctl-linux-amd64
curl -sSLo ~/.local/bin/kubectl   https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > ~/.local/bin/helm
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > ~/.local/bin/crane
chmod +x ~/.local/bin/{talosctl,kubectl,helm,crane}
```

Match `talosctl` to the Talos ISO you booted (`talosctl version --insecure -n <ip>` reports
the server's tag). `crane` is what pushes images to a plain-HTTP registry without a
root-level Docker daemon change — see §2.

**VM sizing.** 6 vCPU / 10 GB RAM / 27 GB disk runs the whole stack with room to spare
(measured: ~4.4 GB used, 5.4 GB available with all 29 pods up). 4 GB is not enough. The
chart sets no resource requests or limits on purpose — on a single node the VM's RAM is the
only budget there is. Resize a stopped VM with:

```bash
virsh -c qemu:///system destroy talos      # it's in maintenance mode; nothing is lost
virsh -c qemu:///system setmaxmem talos 10G --config
virsh -c qemu:///system setmem    talos 10G --config
virsh -c qemu:///system setvcpus  talos 6 --config --maximum
virsh -c qemu:///system setvcpus  talos 6 --config
```

Two addresses matter throughout:

| Name | Meaning | Example |
|---|---|---|
| `TALOS_HOST` | the VM's IP — used by the browser, `talosctl` and `kubectl` | `192.168.122.33` |
| `K8S_REGISTRY` | `TALOS_HOST:30500` — the in-cluster registry (§2) | `192.168.122.33:30500` |

Find the VM's address with `virsh -c qemu:///system net-dhcp-leases default`.

## 1. Install Talos onto the VM

### The virt-manager trap

virt-manager treats the install ISO as one-shot: on the VM's **first shutdown** it ejects
the CD and rewrites the boot order to `hd`. A Talos VM booted from `metal-amd64.iso` runs
entirely in RAM, so the disk is still empty — the next start lands on
`Boot failed: not a bootable disk`. Put the ISO back before installing:

```bash
virsh -c qemu:///system change-media talos sda /path/to/metal-amd64.iso --config --insert
virt-xml -c qemu:///system talos --edit --boot cdrom,hd
virsh -c qemu:///system start talos
```

Wait for the maintenance-mode API, then confirm the install disk's device name — on virtio
it is `/dev/vda`, and Talos's default selector expects `/dev/sda`:

```bash
talosctl get disks --insecure -n <TALOS_HOST> -e <TALOS_HOST>
```

### Generate the machine config

Talos 1.14 moved several v1alpha1 fields into their own config documents. In particular
`machine.install` is now `UnattendedInstallConfig`, and patching the old field is rejected
with *"UnattendedInstallConfig config is incompatible with v1alpha1 config"*. Write
`patch.yaml` as a multi-document patch:

```yaml
machine:
  certSANs:
    - 192.168.122.33
  registries:
    mirrors:
      # The in-cluster registry (§2) speaks plain HTTP.
      "192.168.122.33:30500":
        endpoints:
          - http://192.168.122.33:30500
---
apiVersion: v1alpha1
kind: UnattendedInstallConfig
provisioning:
  diskSelector:
    match: disk.dev_path == "/dev/vda"
```

```bash
talosctl gen config big https://<TALOS_HOST>:6443 --output-dir ~/.talos/big --config-patch @patch.yaml
talosctl apply-config --insecure -n <TALOS_HOST> -e <TALOS_HOST> --file ~/.talos/big/controlplane.yaml
```

Talos installs to the disk and **kexecs straight into the installed system**, so the CD
boot order doesn't get in the way here. Then point the client at the node and bootstrap:

```bash
talosctl config merge ~/.talos/big/talosconfig
talosctl config endpoint <TALOS_HOST>
talosctl config node <TALOS_HOST>
talosctl bootstrap                      # wait for `talosctl version` to answer first
talosctl kubeconfig -f ~/.kube/config
```

A single-node cluster must run workloads on the control plane, or CoreDNS never schedules:

```bash
kubectl taint node --all node-role.kubernetes.io/control-plane-
```

Finally, make a VM restart boot the installed system rather than the ISO (takes effect at
the next full power cycle):

```bash
virsh -c qemu:///system change-media talos sda --eject --config
virt-xml -c qemu:///system talos --edit --boot hd
```

## 2. A registry the node can pull from

Talos has no Docker daemon and no way to side-load an image, so this repo's images have to
come from a registry. The registry runs **inside the cluster**, published on NodePort
30500 (`infra/helm/registry.yaml`):

```bash
make k8s-registry
```

Why in-cluster rather than on the laptop: a laptop-side registry needs an inbound port
opened on firewalld's `libvirt` zone (`sudo firewall-cmd --zone=libvirt --add-port=5000/tcp`),
which needs root. Pushing from the laptop *to* the node is outbound and always allowed, and
the node pulls from its own NodePort. If you do open that port, put a registry on the
laptop instead and point `K8S_REGISTRY` at `<laptop-ip>:5000` — the mirror patch in §1 has
an entry ready for it.

Its storage is `emptyDir`, so if the registry pod is ever replaced, re-run `make k8s-images`.

## 3. Build and push the images

```bash
make k8s-images K8S_REGISTRY=<TALOS_HOST>:30500
```

This builds the nine images with `docker compose build` — same contexts and Dockerfiles as
compose, no second build definition — then `docker save | crane push --insecure` each one.
`docker push` is not used: the registry speaks plain HTTP, which the Docker daemon refuses
without a root-level `insecure-registries` entry, while crane just takes `--insecure`.

## 4. Deploy

```bash
make k8s-up TALOS_HOST=<TALOS_HOST> K8S_REGISTRY=<TALOS_HOST>:30500
```

That does two things:

1. `make k8s-seed` — creates the ConfigMaps the chart mounts, from the config files that
   already live in this repo (`infra/helm/seed-configmaps.sh`): the four
   `setup_configuration/data.yaml` files, the Keycloak realm exports, the BPMN + DMN, and
   the two bootstrap scripts. Re-run it after editing any of them.
2. `helm upgrade --install` of the chart into namespace `big`.

First bring-up takes a few minutes: the four Django services migrate their databases and
apply their `setup_configuration`, Flowable creates its schema, and the bootstrap Jobs
deploy the BPMN/DMN, seed the zaaktype and register the NRC abonnement.

```bash
kubectl -n big get pods -w
kubectl -n big get jobs        # all four must reach COMPLETIONS 1/1
```

The Jobs are the stack's wiring; if one is not complete, the flow is broken somewhere
specific:

| Job | What breaks without it |
|---|---|
| `flowable-init` | no `registratie` process, no diploma DMN |
| `registerrecord-init` | the register has no RegisterRecord objecttype, so writes are refused |
| `seed-zaaktype` | the ACL can't resolve `BIG-REGISTRATIE`, so no zaak is created |
| `nrc-subscribe` | register writes never reach the projection — the public register stays empty |

## 5. Use it

### The portals must be reached over `localhost`

The portals' OIDC flow uses PKCE, which needs `crypto.subtle` — and browsers only expose
that in a **secure context**: HTTPS, or an origin on `localhost`/`127.0.0.1`. A NodePort on
the VM's IP is neither, so `http://<TALOS_HOST>:30140` fails before it can even build the
authorize URL:

```
ERROR TypeError: Cannot read properties of undefined (reading 'digest')
    at t.calcHash → t.generateCodeChallenge → t.createUrlCodeFlowAuthorize
```

So deploy with `TALOS_HOST=localhost` — which pins Keycloak's issuer and the portals'
`config.json` authority to `http://localhost:30180` — and forward the browser-facing
services to those same ports:

```bash
make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500
make k8s-portals          # stays in the foreground; Ctrl-C stops all five forwards
```

| URL (needs `make k8s-portals`) | What |
|---|---|
| `http://localhost:30140` | self-service portal (DigiD) |
| `http://localhost:30141` | openbaar register (anonymous) |
| `http://localhost:30142` | behandel portal (medewerker) |
| `http://localhost:30143` | beheer portal (medewerker) |
| `http://localhost:30180` | Keycloak (admin/admin) |

The port numbers are deliberately the NodePort numbers: Keycloak's issuer is one fixed
string, so the port the browser uses has to match the one baked into `config.json`.

This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
`localhost:8180`); only the addresses differ.

### The admin UIs work straight off the NodePorts

These are server-rendered and need no secure context, so they are reachable at the VM's
address with no forwarding:

| URL | What |
|---|---|
| `http://<TALOS_HOST>:30000` | OpenZaak admin (admin/admin) |
| `http://<TALOS_HOST>:30001` | Open Notificaties admin (admin/admin) |
| `http://<TALOS_HOST>:30020` / `:30021` | Objecttypen / Objecten admin |
| `http://<TALOS_HOST>:30080` | BFF (`/health`) |
| `http://<TALOS_HOST>:30090` | Flowable REST (rest-admin/test) |

### Credentials

Log in with the test users from `docs/synthetic-data.md` (all password `test123`, e.g.
`jan-burger` for self-service, `merel-behandelaar` for behandel). The `medewerker` realm
enforces MFA (ADR-0031) — print a current code with
`python3 infra/keycloak/check_realms.py otp`. Walk the flow in `docs/demo-script.md`.

`TALOS_HOST` is not cosmetic: it pins Keycloak's issuer (`KC_HOSTNAME`) and the portals'
OIDC authority to the same string, which is what makes a browser token pass the BFF's
validation (ADR-0010). Change it and you must re-run `make k8s-up` — the chart rolls the
portals for you, because their `config.json` is a subPath mount and would otherwise keep
serving the old authority.

### Smoke-test the whole chain without a browser

With the forwards running:

```bash
TOK=$(curl -s -X POST http://localhost:30180/realms/digid/protocol/openid-connect/token \
  -d grant_type=password -d client_id=big-portal \
  -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token)

# through the portal's Caddy, so this also proves the BFF reverse proxy
curl -s -X POST http://localhost:30140/self-service/registrations \
  -H "Authorization: Bearer $TOK" -H 'Content-Length: 0'
# → {"registrationId":"…","status":"Ingediend"}

curl -s http://localhost:30141/openbaar/register
# → [{"id":"…","status":"INGEDIEND","reference":"<the registrationId>"}]
```

The second call proves the whole Common Ground path: portal → BFF → domain → Flowable →
ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → openbaar register.

## 6. Keeping the databases (recommended if you iterate on the chart)

By default every database is an `emptyDir`: no CSI driver needed, and the data lives as
long as the pod. Note what that means in practice — **any** change to a database pod's
template (an image policy, an env value, a probe) recreates the pod and wipes it. The stack
then needs its bootstrap re-run:

```bash
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...
```

which re-runs the four Jobs *and* restarts `event-subscriber` + `projection-api`, because
those two create the projection schema on start and otherwise keep writing to a
schema-less database (`relation "processed_notifications" does not exist`). For persistence, install Rancher's local-path-provisioner — on Talos it
must write under `/var` and its namespace needs the privileged Pod Security label:

```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/rancher/local-path-provisioner/deploy?ref=v0.0.31
patches:
  - patch: |-
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: local-path-config
        namespace: local-path-storage
      data:
        config.json: |-
          { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/var/local-path-provisioner"] } ] }
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: local-path-storage
        labels:
          pod-security.kubernetes.io/enforce: privileged
```

```bash
kubectl apply -k .
make k8s-up TALOS_HOST=... K8S_REGISTRY=... K8S_SET='--set persistence.storageClass=local-path'
```

The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the data behind;
`make k8s-purge` drops the namespace and with it the volumes.

## 7. Day-to-day

```bash
make k8s-lint                      # render + schema-check the chart, no cluster needed
make k8s-drift                     # fail if compose and the chart describe different stacks
make k8s-portals                   # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=...   # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
make k8s-seed                      # after editing a data.yaml, a realm export, or the BPMN
make k8s-reseed TALOS_HOST=... K8S_REGISTRY=...   # re-run the bootstrap Jobs + reset the projection schema
make k8s-down                      # uninstall, keep the database PVCs
make k8s-purge                     # uninstall and drop the namespace
```

This repo's images are pulled with `imagePullPolicy: Always` (the `dev` tag is mutable), so
`kubectl -n big rollout restart deploy/<name>` after `make k8s-images` picks up a rebuild.
Upstream images stay `IfNotPresent`: their tags are pinned, and keeping them out of the pod
template avoids needless churn — a changed template makes a Job unpatchable.

`k8s-reseed` is also the path for *changing* a Job in the chart: a Job's pod template is
immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`.

## 8. When it doesn't work

| Symptom | Cause |
|---|---|
| `Boot failed: not a bootable disk` | virt-manager ejected the install ISO on first shutdown — see §1 |
| The VM comes back in maintenance mode after a restart | the ISO is still attached and boots first; eject it and set `--boot hd` (§1) |
| `apply-config` rejects the patch with *"incompatible with v1alpha1"* | Talos ≥1.14 owns that field in its own config document — patch the document, not `machine.*` (§1) |
| CoreDNS `Pending` forever | the control-plane taint is still on the only node (§1) |
| `ImagePullBackOff` … `pull QPS exceeded` | transient: the kubelet rate-limits pulls when ~30 pods start at once. It recovers on retry |
| `ImagePullBackOff` on a `register-referentie/*` image | the registry mirror patch is missing: `talosctl get registriesconfig` |
| Pod stuck in `ContainerCreating`, event names a ConfigMap | `make k8s-seed` |
| `seed-zaaktype` retrying | publishing a zaaktype validates the resultaattype against `selectielijst.openzaak.nl`, so this one Job needs outbound internet from the VM (ADR-0006) |
| `TypeError: Cannot read properties of undefined (reading 'digest')` on a portal | not a secure context: `crypto.subtle` is absent on `http://<ip>`. Use `localhost` + `make k8s-portals` (§5) |
| Login redirects but the portal stays logged out, or the BFF answers 401 | `TALOS_HOST` doesn't match the address in the browser's URL bar — issuer mismatch. Re-run `make k8s-up` with the right value |
| A portal returns 502 on `/self-service/…` | the BFF is unreachable from the portal pod: check `kubectl -n big get svc bff` and the BFF's own readiness |
| Public register empty after a submit | usually a wiped `emptyDir` database (§6): `make k8s-reseed`. Confirm with `kubectl -n big logs deploy/event-subscriber \| grep 42P01` |
| `helm upgrade` fails with `cannot patch … with kind Job` | see §7 — use `make k8s-reseed` |
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |

## 9. Deploying on merge to main

`.gitea/workflows/deploy.yaml` runs the §3–§4 steps against the **lab server's** Talos VM
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
merge gate, so the workflow deploys without re-running the checks.

**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
gives you a node that still carries the control-plane taint and knows nothing about the
plain-HTTP registry, and the deploy hits those in that order: the `registry` pod sits
`Pending` until `rollout status` times out, and once that is fixed every repo image fails to
pull. Two separate fixes:

```bash
# on the Fedora host — 1. let workloads onto the only node (§1)
export KUBECONFIG=~/talos-kubeconfig-local
kubectl taint node --all node-role.kubernetes.io/control-plane-

# 2. trust the in-cluster registry over plain HTTP (§2)
cat > /tmp/registry-patch.yaml <<'YAML'
machine:
  registries:
    mirrors:
      "<TALOS_VM_IP>:30500":
        endpoints:
          - http://<TALOS_VM_IP>:30500
YAML
talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/registry-patch.yaml
```

Keep those two apart. On Talos 1.14 a patch that also sets
`cluster.allowSchedulingOnControlPlanes` is rejected with *".cluster.allowSchedulingOnControlPlanes
is already set in v1alpha1 config"* — the field moved out of the v1alpha1 schema, the same way
`machine.install` did (§1) — and the rejection takes the whole patch with it, so the mirror
silently doesn't land either. `kubectl taint` is the documented way (§1); it is undone if the
node ever re-registers, which is a reboot, not a deploy.

The cluster's API and registry are not exposed publicly, so the job forwards them over the
same SSH hop the Gitea-runner pipeline uses:

```
ssh -p 6667 user@labs.respellion.tech  -L 6443 -L 30500 -L 30141  →  <TALOS_VM_IP>
```

Consequences worth knowing:

- Images are **pushed** to `localhost:30500` (the tunnel) and **pulled** by the node from
  `<TALOS_VM_IP>:30500` (its own NodePort, the address in the Talos registry-mirror patch).
  Same registry, two names — hence the two `K8S_REGISTRY` values in the workflow.
- It calls `make k8s-reseed`, not `make k8s-up`: the bootstrap Jobs are idempotent, and
  deleting them first is what stops a changed Job template from wedging `helm upgrade` (§7).
- `dev` is a mutable tag, so a `rollout restart` of the nine repo deployments is what
  actually puts the new images in the pods.
- Deploys **queue** (`cancel-in-progress: false`): a helm upgrade killed half-way leaves the
  release in `pending-upgrade`, which has to be unwedged by hand.

Settings, all on the repository in Gitea:

| Kind | Name | What |
|---|---|---|
| Secret | `TALOS_SSH_KEY` | private key for `user@labs.respellion.tech` (the Fedora host) |
| Secret | `TALOS_KUBECONFIG` | base64 of the kubeconfig, **`server: https://127.0.0.1:6443`** — Talos puts `127.0.0.1` in the apiserver cert SANs, so TLS still verifies through the tunnel |
| Variable | `TALOS_VM_IP` | the VM's libvirt address (default `192.168.122.173`) |
| Variable | `TALOS_HOST` | the browser-facing host baked into Keycloak's issuer (default `localhost`, see §5) |

The last step smokes `GET /openbaar/register` through the openbaar portal, which exercises
portal → Caddy → BFF → projection. An empty register passes; a 502 does not.

Not covered: the portals still need `make k8s-portals` (or an SSH forward) to be usable in a
browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the
upgrade path.

## What is not ported

- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
  images too, so switching them on means pushing them as well:
  `K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
  The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
  chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
  clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
  image bump that skipped the chart fails there, but nothing deploys the chart in CI.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.