ci(deploy): publish the stack over TLS when PUBLIC_DOMAIN is set (refs #175)

The in-cluster edge (#178) is off unless the chart is given a domain, so pass
one through from a repository variable. Unset, the deploy is exactly what it was
— NodePorts, and the portals reachable only over the SSH forwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitea/workflows/ci.yaml

@@ -207,14 +207,8 @@ jobs:
   # dispatched (gitea-actions-gotchas.md §7). Default `if: success()` dispatches normally. Cost: a
   # failing mutation ratchet now skips verify-stack instead of running it anyway; the fix-and-re-push
   # re-run exercises verify-stack, so we still get the signal.
-  #
-  # Main only, not on PRs: the runner shares the lab node with the deployed stack, and a second
-  # full stack per PR was what got the runner OOM-killed (#182). PRs still gate on every job above;
-  # the live-stack check runs once per merge. A plain event `if` keeps the implicit success(), so it
-  # is not the status-function case from gotchas §7.
   verify-stack:
     needs: [mutation]
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
     runs-on: ubuntu-latest
     steps:
       - uses: https://github.com/actions/checkout@v4

.gitea/workflows/deploy.yaml

@@ -27,13 +27,10 @@ jobs:
       # `kubectl port-forward` — runbook §5. Override with repo variables.
       TALOS_VM_IP: ${{ vars.TALOS_VM_IP }}
       TALOS_HOST: ${{ vars.TALOS_HOST }}
-      # Set it when the labs Caddy publishes the portals: Keycloak's public https
+      # Set it and the stack is published over TLS on <sub>.<domain> by the
-      # origin, e.g. https://big-auth.labs.respellion.tech (runbook, "Publishing
+      # in-cluster edge (ADR-0035, runbook §10). Empty = NodePorts, as before.
-      # through the labs Caddy").
+      PUBLIC_DOMAIN: ${{ vars.PUBLIC_DOMAIN }}
-      KEYCLOAK_URL: ${{ vars.KEYCLOAK_URL }}
+      PUBLIC_EMAIL: ${{ vars.PUBLIC_EMAIL }}
-      # `true` fills in the medewerker OTP step for the public demo (chart value
-      # demo.otpAutofill). The fixture secret is committed: demo only.
-      OTP_AUTOFILL: ${{ vars.OTP_AUTOFILL }}
     steps:
       - uses: https://github.com/actions/checkout@v4
 
@@ -97,10 +94,12 @@ jobs:
       # Job template from wedging the upgrade (`cannot patch … with kind Job`).
       - name: Deploy the chart
         run: |
+          set -euo pipefail
+          publish="${PUBLIC_DOMAIN:+--set public.domain=$PUBLIC_DOMAIN --set public.email=${PUBLIC_EMAIL:-}}"
           make k8s-reseed \
             TALOS_HOST=${TALOS_HOST:-localhost} \
             K8S_REGISTRY=${TALOS_VM_IP:-192.168.122.173}:30500 \
-            K8S_SET="${KEYCLOAK_URL:+--set keycloakUrl=$KEYCLOAK_URL} --set demo.otpAutofill=${OTP_AUTOFILL:-false}"
+            K8S_SET="$publish"
 
       # `dev` is a mutable tag and helm sees an unchanged pod template, so the
       # new images only land on a restart (pullPolicy is already Always).
@@ -116,6 +115,12 @@ jobs:
       - name: Smoke the public register
         run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
 
-      - name: Pods on failure
+      # Cluster-wide, not just `big`: the first thing that can fail is the registry
+      # in its own namespace, and a scheduling problem shows up in the events, not
+      # in `rollout status` — which only ever says "timed out waiting".
+      - name: Pods and events on failure
         if: failure()
-        run: kubectl -n big get pods,jobs || true
+        run: |
+          kubectl get pods -A -o wide || true
+          kubectl -n big get jobs || true
+          kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true

docs/runbooks/ci.md

@@ -21,7 +21,7 @@ and CI cannot drift:
 | `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
 | `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
 | `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
-| `verify-stack` | **push to `main` only, skipped on PRs** (#182) — the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
+| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
 
 > **Why one `verify-stack` job, not three.** The single self-hosted runner runs jobs
 > **sequentially**, so booting OpenZaak once (instead of once per check) is the

docs/runbooks/kubernetes-talos.md

@@ -366,6 +366,36 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
 every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
 merge gate, so the workflow deploys without re-running the checks.
 
+**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
+gives you a node that still carries the control-plane taint and knows nothing about the
+plain-HTTP registry, and the deploy hits those in that order: the `registry` pod sits
+`Pending` until `rollout status` times out, and once that is fixed every repo image fails to
+pull. Two separate fixes:
+
+```bash
+# on the Fedora host — 1. let workloads onto the only node (§1)
+export KUBECONFIG=~/talos-kubeconfig-local
+kubectl taint node --all node-role.kubernetes.io/control-plane-
+
+# 2. trust the in-cluster registry over plain HTTP (§2)
+cat > /tmp/registry-patch.yaml <<'YAML'
+machine:
+  registries:
+    mirrors:
+      "<TALOS_VM_IP>:30500":
+        endpoints:
+          - http://<TALOS_VM_IP>:30500
+YAML
+talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/registry-patch.yaml
+```
+
+Keep those two apart. On Talos 1.14 a patch that also sets
+`cluster.allowSchedulingOnControlPlanes` is rejected with *".cluster.allowSchedulingOnControlPlanes
+is already set in v1alpha1 config"* — the field moved out of the v1alpha1 schema, the same way
+`machine.install` did (§1) — and the rejection takes the whole patch with it, so the mirror
+silently doesn't land either. `kubectl taint` is the documented way (§1); it is undone if the
+node ever re-registers, which is a reboot, not a deploy.
+
 The cluster's API and registry are not exposed publicly, so the job forwards them over the
 same SSH hop the Gitea-runner pipeline uses:
 
@@ -397,58 +427,30 @@ Settings, all on the repository in Gitea:
 The last step smokes `GET /openbaar/register` through the openbaar portal, which exercises
 portal → Caddy → BFF → projection. An empty register passes; a 502 does not.
 
+### Reaching the portals from a laptop
+
+The deployed portals are pinned to `http://localhost:30180` for Keycloak (§5), so a browser
+needs **all five** browser-facing ports on its own localhost — the portal alone is not
+enough, and a missing Keycloak shows up as `ERR_CONNECTION_REFUSED` on
+`/realms/*/.well-known/openid-configuration` followed by an opaque `ERROR Error: [object Object]`.
+`make k8s-portals` does this when kubectl can reach the cluster; through the lab server one
+SSH does it without a kubeconfig at all:
+
+```bash
+ssh -N -p 6667 \
+  -L 30140:<TALOS_VM_IP>:30140 -L 30141:<TALOS_VM_IP>:30141 \
+  -L 30142:<TALOS_VM_IP>:30142 -L 30143:<TALOS_VM_IP>:30143 \
+  -L 30180:<TALOS_VM_IP>:30180 \
+  user@labs.respellion.tech
+```
+
+Then the §5 table's URLs work as written. The admin UIs (OpenZaak, Flowable, …) need no
+forward — they are server-rendered, so the VM's address is fine.
+
 Not covered: the portals still need `make k8s-portals` (or an SSH forward) to be usable in a
 browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the
 upgrade path.
 
-## Publishing through the labs Caddy
-
-The portals can be reached on real hostnames through the Caddy that already fronts
-`*.labs.respellion.tech` (repo `Infra`, `infra/development/`). The chain:
-
-```
-browser → Caddy (labs server, TLS) → openssh-server:3014x/30180
-        → reverse SSH tunnel → Fedora host → <TALOS_VM_IP>:3014x/30180 (NodePorts)
-```
-
-| URL | NodePort |
-|---|---|
-| `https://big-register.labs.respellion.tech` | 30141 openbaar |
-| `https://big-mijn.labs.respellion.tech` | 30140 self-service |
-| `https://big-behandel.labs.respellion.tech` | 30142 behandel |
-| `https://big-beheer.labs.respellion.tech` | 30143 beheer |
-| `https://big-auth.labs.respellion.tech` | 30180 Keycloak (`/admin` blocked) |
-
-HTTPS makes the portals a secure context, so PKCE works without port-forwards — but
-Keycloak's issuer must be the public origin. Deploy with it:
-
-```bash
-make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500 \
-  K8S_SET="--set keycloakUrl=https://big-auth.labs.respellion.tech"
-```
-
-For deploy-on-merge, set the repository variable `KEYCLOAK_URL` to the same value.
-With it set, the `localhost` port-forwards (§5) no longer log in: the issuer is one string.
-
-Staff logins still hit the enforced OTP step. For a demo, set the repository variable
-`OTP_AUTOFILL=true` (chart value `demo.otpAutofill`): Keycloak then uses the `big-demo`
-theme, which fills in and submits the code from the fixture secret, so the step is visible
-but needs no authenticator. Keycloak restarts when the value flips. Demo only — the secret
-is committed.
-
-The theme lives in `infra/keycloak/themes/big-demo/` and is seeded as the `rr-kc-theme`
-ConfigMap by `infra/helm/seed-configmaps.sh` on every deploy. Keycloak runs `start-dev`,
-which doesn't cache themes, so an edit shows up about a minute after the ConfigMap changes.
-A *new* theme file also needs a key in the seed script and a path in the keycloak `files`
-in `values.yaml`.
-
-One-time setup:
-
-1. Fedora host: install `infra/development/big-portals-tunnel.service` from the Infra repo
-   (instructions in the file).
-2. Labs server: deploy the Infra `Caddyfile` + `compose.yml` (Caddy joins the
-   `openssh_default` network to reach the tunnel ends).
-
 ## What is not ported
 
 - **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built

infra/docker-compose.yml

@@ -57,9 +57,6 @@ services:
       # share this anchor and ignore it — they don't run uwsgi.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
-      # Same lever for oz-celery: unset, the worker forks one process per CPU (22 on the lab node,
-      # ~225 MB each), which OOM-killed the shared runner mid-verify-stack. Only celery reads it.
-      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: openzaak.conf.docker
       SECRET_KEY: ${OZ_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: oz-db
@@ -147,8 +144,6 @@ services:
       # 1 uWSGI worker, not the image default of 4×4 (#147) — see the oz-env note above.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
-      # Two celery workers, not one per CPU — see the oz-env note above.
-      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: nrc.conf.docker
       SECRET_KEY: ${NRC_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: nrc-db

infra/helm/big-reference/templates/_helpers.tpl

@@ -94,10 +94,6 @@ volumes:
       {{- with .defaultMode }}
       defaultMode: {{ . }}
       {{- end }}
-      {{- with .items }}
-      items:
-{{- toYaml . | nindent 8 }}
-      {{- end }}
 {{- end }}
 {{- with $w.data }}
   - name: data
@@ -129,26 +125,14 @@ volumes:
 {{/*
 Env list from a map. Every value is run through `tpl`, so values.yaml can name
 cluster-internal hosts ({{ .Release.Namespace }}) and the node address
-({{ .Values.host }}) without the chart hard-coding either. A value that renders
+({{ .Values.host }}) without the chart hard-coding either.
-empty is left out, which is how a setting is made conditional on a chart value.
 */}}
 {{- define "big.env" -}}
 {{- $root := index . 0 -}}
 {{- range $k, $v := index . 1 }}
-{{- $val := tpl (toString $v) $root }}
-{{- if $val }}
 - name: {{ $k }}
-  value: {{ $val | quote }}
+  value: {{ tpl (toString $v) $root | quote }}
 {{- end }}
-{{- end }}
-{{- end -}}
-
-{{/*
-The origin a browser reaches Keycloak on: the issuer Keycloak pins and the
-authority the portals use, from one place so they cannot drift (ADR-0010).
-*/}}
-{{- define "big.keycloakUrl" -}}
-{{- .Values.keycloakUrl | default (printf "http://%s:%v" .Values.host (index .Values.nodePorts "keycloak")) -}}
 {{- end -}}
 
 {{- define "big.labels" -}}

infra/helm/big-reference/templates/config.yaml

@@ -40,5 +40,5 @@ metadata:
 {{- include "big.labels" (dict "root" $ "name" (printf "portal-config-%s" $realm)) | nindent 4 }}
 data:
   config.json: |
-    { "authority": "{{ include "big.keycloakUrl" $ }}/realms/{{ $realm }}" }
+    { "authority": "{{ printf "http://%s:%v" $.Values.host (index $.Values.nodePorts "keycloak") }}/realms/{{ $realm }}" }
 {{- end }}

infra/helm/big-reference/templates/deployments.yaml

@@ -28,7 +28,7 @@ spec:
       {{- range $w.files }}
       {{- if hasPrefix "portal-config-" .configMap }}
       annotations:
-        checksum/portal-config: {{ include "big.keycloakUrl" $ | sha256sum }}
+        checksum/portal-config: {{ printf "%s|%v" $.Values.host (index $.Values.nodePorts "keycloak") | sha256sum }}
       {{- end }}
       {{- end }}
       labels:

infra/helm/big-reference/values.yaml

@@ -25,17 +25,6 @@
 # string, so browser tokens and the BFF's discovered issuer agree.
 host: 192.168.122.100
 
-# Set when a TLS proxy outside the cluster publishes Keycloak: the full origin, no
-# trailing slash. It replaces `host` + Keycloak's NodePort as the issuer and the
-# portals' authority (runbook, "Publishing through the labs Caddy").
-keycloakUrl: ""
-
-demo:
-  # Fill in and submit the medewerker OTP step from the fixture secret, so a public
-  # demo shows MFA enforced without an authenticator: makes the big-demo theme
-  # (infra/keycloak/themes/big-demo) Keycloak's default. Demo only: the secret is committed.
-  otpAutofill: false
-
 # Set when pulling from a private registry (e.g. the Gitea Container Registry).
 imagePullSecrets: []
 
@@ -82,7 +71,6 @@ envGroups:
   oz:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
-    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: openzaak.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: oz-db
@@ -105,7 +93,6 @@ envGroups:
   nrc:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
-    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: nrc.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: nrc-db
@@ -281,31 +268,13 @@ workloads:
       # Pin the issuer to the address the browser uses, and let backchannel calls
       # keep using keycloak:8080 — the BFF discovers metadata in-cluster and gets
       # this issuer back, which is what browser tokens carry (infra/host-browser.yml).
-      KC_HOSTNAME: '{{ include "big.keycloakUrl" . }}'
+      KC_HOSTNAME: "http://{{ .Values.host }}:{{ index .Values.nodePorts \"keycloak\" }}"
       KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
-      # Only rendered with demo.otpAutofill (big.env skips empty values); off, Keycloak
-      # keeps its stock theme and the mounted big-demo theme is unused.
-      KC_SPI_THEME_DEFAULT: '{{ if .Values.demo.otpAutofill }}big-demo{{ end }}'
-      # Behind a TLS proxy (keycloakUrl) the dynamic backchannel URLs — token,
-      # userinfo, certs — take their scheme from the request, which reaches Keycloak
-      # as plain http; trusting X-Forwarded-Proto keeps them https so the browser
-      # doesn't block them as mixed content. In-cluster calls send no such header.
-      KC_PROXY_HEADERS: xforwarded
     ports: [{ name: http, port: 8080 }]
     # TCP, not /health/ready on the management port: nothing here gates on realm
     # import, and a wrong health path would leave the Service with no endpoints.
     probe: { tcpSocket: { port: 8080 }, initialDelaySeconds: 15 }
-    files:
+    files: [{ configMap: rr-kc-realms, mountPath: /opt/keycloak/data/import }]
-      - { configMap: rr-kc-realms, mountPath: /opt/keycloak/data/import }
-      # infra/keycloak/themes/big-demo, seeded by infra/helm/seed-configmaps.sh.
-      - configMap: rr-kc-theme
-        mountPath: /opt/keycloak/themes/big-demo
-        items:
-          - { key: login.properties, path: login/theme.properties }
-          - { key: otp-autofill.js, path: login/resources/js/otp-autofill.js }
-          - { key: account.properties, path: account/theme.properties }
-          - { key: admin.properties, path: admin/theme.properties }
-          - { key: email.properties, path: email/theme.properties }
 
   # ── Flowable (S-03) ─────────────────────────────────────────────────────────
   flowable-db:

infra/helm/seed-configmaps.sh

@@ -30,15 +30,6 @@ seed() {  # name  <kubectl --from-file args...>
 seed rr-oz-config            --from-file="$repo/infra/openzaak/setup_configuration/"
 seed rr-nrc-config           --from-file="$repo/infra/opennotificaties/setup_configuration/"
 seed rr-kc-realms            --from-file="$repo/infra/keycloak/realms/"
-# The big-demo login theme (demo.otpAutofill). ConfigMap keys are flat, so each
-# file gets a key here and its path back in the keycloak `files` in values.yaml.
-theme="$repo/infra/keycloak/themes/big-demo"
-seed rr-kc-theme \
-  --from-file=login.properties="$theme/login/theme.properties" \
-  --from-file=otp-autofill.js="$theme/login/resources/js/otp-autofill.js" \
-  --from-file=account.properties="$theme/account/theme.properties" \
-  --from-file=admin.properties="$theme/admin/theme.properties" \
-  --from-file=email.properties="$theme/email/theme.properties"
 seed rr-objecttypen-config   --from-file="$repo/infra/objecttypen/setup_configuration/"
 seed rr-objecten-config      --from-file="$repo/infra/objecten/setup_configuration/"
 # register.py + the RegisterRecord JSON schema (the __pycache__ dir is skipped:

infra/keycloak/themes/big-demo/account/theme.properties

@@ -1,4 +0,0 @@
-# The chart makes big-demo the default for every theme type, and Keycloak does not
-# fall back for a type a theme lacks (the account page then fails), so each type is
-# declared as a plain child of Keycloak 26's own default.
-parent=keycloak.v3

infra/keycloak/themes/big-demo/admin/theme.properties

@@ -1,4 +0,0 @@
-# The chart makes big-demo the default for every theme type, and Keycloak does not
-# fall back for a type a theme lacks (the admin page then fails), so each type is
-# declared as a plain child of Keycloak 26's own default.
-parent=keycloak.v2

infra/keycloak/themes/big-demo/email/theme.properties

@@ -1,4 +0,0 @@
-# The chart makes big-demo the default for every theme type, and Keycloak does not
-# fall back for a type a theme lacks (the email page then fails), so each type is
-# declared as a plain child of Keycloak 26's own default.
-parent=keycloak

infra/keycloak/themes/big-demo/login/resources/js/otp-autofill.js

@@ -1,24 +0,0 @@
-// RFC 6238 with Keycloak's default policy (HmacSHA1, 6 digits, 30 s) over the
-// raw bytes of the medewerker fixture secret — same as tests/e2e/keycloak-login.ts.
-document.addEventListener('DOMContentLoaded', async () => {
-  const input = document.querySelector('input[name="otp"]');
-  if (!input || !input.form) return;
-  const key = await crypto.subtle.importKey('raw',
-    new TextEncoder().encode('BIGMEDEWERKEROTPSEED'), { name: 'HMAC', hash: 'SHA-1' }, false, ['sign']);
-  // A code is single-use, so a second login in the same window spends the next
-  // counter (Keycloak's look-ahead accepts it). Past that, fill but don't submit,
-  // so a rejected code can't turn into a submit loop.
-  const now = Math.floor(Date.now() / 30000);
-  let last = -1;
-  try { last = Number(sessionStorage.getItem('big-otp-counter')) || -1; } catch {}
-  const counter = Math.max(now, last + 1);
-  const msg = new DataView(new ArrayBuffer(8));
-  msg.setBigUint64(0, BigInt(counter));
-  const mac = new Uint8Array(await crypto.subtle.sign('HMAC', key, msg.buffer));
-  const o = mac[19] & 0x0f;
-  const n = ((mac[o] & 0x7f) << 24 | mac[o + 1] << 16 | mac[o + 2] << 8 | mac[o + 3]) % 1e6;
-  input.value = String(n).padStart(6, '0');
-  if (counter > now + 1) return;
-  try { sessionStorage.setItem('big-otp-counter', String(counter)); } catch {}
-  input.form.requestSubmit();
-});

infra/keycloak/themes/big-demo/login/theme.properties

@@ -1,11 +0,0 @@
-# Demo login theme for the public Talos deployment: keycloak.v2 plus a script that
-# fills in and submits the medewerker OTP step from the committed fixture secret
-# (docs/runbooks/keycloak.md). Only used when the chart's demo.otpAutofill is on —
-# it then becomes Keycloak's default theme. Never enable it anywhere real.
-#
-# Add styles, messages or template overrides here as in any Keycloak theme
-# (https://www.keycloak.org/ui-customization/themes); new files must also be
-# listed in infra/helm/seed-configmaps.sh and the keycloak `files` in values.yaml.
-parent=keycloak.v2
-import=common/keycloak
-scripts=js/otp-autofill.js