fix(infra): cap celery workers at 2 so the shared node stops OOM-killing CI (closes #182) (#183)

## What & why

`verify-stack` is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with `CELERY_WORKER_CONCURRENCY` unset, `oz-celery` and `nrc-celery` each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182.

This sets `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups:
- **compose** (`&oz-env`, `&nrc-env`): what `verify-stack` starts inside `dind`.
- **chart** (`envGroups.oz` / `.nrc`): the deployed demo on the same node.

Both images' `/celery_worker.sh` honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. `objecten-celery` already defaults to 1.

Closes #182

## Definition of Done

- [x] Linked Gitea issue (above).
- [ ] Failing test committed before the implementation. *(Resource setting; the evidence is the OOM log in #182.)*
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [ ] CI green. This PR's `verify-stack` run is the check.
- [x] `docker compose config` renders the variable into all 7 services on the two anchors.
- [x] Docs: comments next to the setting, following the existing uWSGI notes.

## Notes for reviewers

- `make k8s-lint` and `make k8s-drift` pass.
- **Not applied live.** I couldn't patch the running cluster from my session. After merge, the deploy updates the `oz-env` / `nrc-env` ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once:
  `kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery`
- **Why 2 and not 1:** this matches `UWSGI_THREADS: "2"`, and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages).
- **Longer term:** CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #183

.gitea/workflows/deploy.yaml

@@ -27,6 +27,10 @@ jobs:
       # `kubectl port-forward` — runbook §5. Override with repo variables.
       TALOS_VM_IP: ${{ vars.TALOS_VM_IP }}
       TALOS_HOST: ${{ vars.TALOS_HOST }}
+      # Set it when the labs Caddy publishes the portals: Keycloak's public https
+      # origin, e.g. https://big-auth.labs.respellion.tech (runbook, "Publishing
+      # through the labs Caddy").
+      KEYCLOAK_URL: ${{ vars.KEYCLOAK_URL }}
     steps:
       - uses: https://github.com/actions/checkout@v4
 
@@ -89,7 +93,11 @@ jobs:
       # The jobs are idempotent, and deleting them first is what keeps a changed
       # Job template from wedging the upgrade (`cannot patch … with kind Job`).
       - name: Deploy the chart
-        run: make k8s-reseed TALOS_HOST=${TALOS_HOST:-localhost} K8S_REGISTRY=${TALOS_VM_IP:-192.168.122.173}:30500
+        run: |
+          make k8s-reseed \
+            TALOS_HOST=${TALOS_HOST:-localhost} \
+            K8S_REGISTRY=${TALOS_VM_IP:-192.168.122.173}:30500 \
+            K8S_SET="${KEYCLOAK_URL:+--set keycloakUrl=$KEYCLOAK_URL}"
 
       # `dev` is a mutable tag and helm sees an unchanged pod template, so the
       # new images only land on a restart (pullPolicy is already Always).

docs/runbooks/kubernetes-talos.md

@@ -401,6 +401,42 @@ Not covered: the portals still need `make k8s-portals` (or an SSH forward) to be
 browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the
 upgrade path.
 
+## Publishing through the labs Caddy
+
+The portals can be reached on real hostnames through the Caddy that already fronts
+`*.labs.respellion.tech` (repo `Infra`, `infra/development/`). The chain:
+
+```
+browser → Caddy (labs server, TLS) → openssh-server:3014x/30180
+        → reverse SSH tunnel → Fedora host → <TALOS_VM_IP>:3014x/30180 (NodePorts)
+```
+
+| URL | NodePort |
+|---|---|
+| `https://big-register.labs.respellion.tech` | 30141 openbaar |
+| `https://big-mijn.labs.respellion.tech` | 30140 self-service |
+| `https://big-behandel.labs.respellion.tech` | 30142 behandel |
+| `https://big-beheer.labs.respellion.tech` | 30143 beheer |
+| `https://big-auth.labs.respellion.tech` | 30180 Keycloak (`/admin` blocked) |
+
+HTTPS makes the portals a secure context, so PKCE works without port-forwards — but
+Keycloak's issuer must be the public origin. Deploy with it:
+
+```bash
+make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500 \
+  K8S_SET="--set keycloakUrl=https://big-auth.labs.respellion.tech"
+```
+
+For deploy-on-merge, set the repository variable `KEYCLOAK_URL` to the same value.
+With it set, the `localhost` port-forwards (§5) no longer log in: the issuer is one string.
+
+One-time setup:
+
+1. Fedora host: install `infra/development/big-portals-tunnel.service` from the Infra repo
+   (instructions in the file).
+2. Labs server: deploy the Infra `Caddyfile` + `compose.yml` (Caddy joins the
+   `openssh_default` network to reach the tunnel ends).
+
 ## What is not ported
 
 - **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built

infra/docker-compose.yml

@@ -57,6 +57,9 @@ services:
       # share this anchor and ignore it — they don't run uwsgi.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
+      # Same lever for oz-celery: unset, the worker forks one process per CPU (22 on the lab node,
+      # ~225 MB each), which OOM-killed the shared runner mid-verify-stack. Only celery reads it.
+      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: openzaak.conf.docker
       SECRET_KEY: ${OZ_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: oz-db
@@ -144,6 +147,8 @@ services:
       # 1 uWSGI worker, not the image default of 4×4 (#147) — see the oz-env note above.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
+      # Two celery workers, not one per CPU — see the oz-env note above.
+      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: nrc.conf.docker
       SECRET_KEY: ${NRC_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: nrc-db

infra/helm/big-reference/templates/_helpers.tpl

@@ -135,6 +135,14 @@ cluster-internal hosts ({{ .Release.Namespace }}) and the node address
 {{- end }}
 {{- end -}}
 
+{{/*
+The origin a browser reaches Keycloak on: the issuer Keycloak pins and the
+authority the portals use, from one place so they cannot drift (ADR-0010).
+*/}}
+{{- define "big.keycloakUrl" -}}
+{{- .Values.keycloakUrl | default (printf "http://%s:%v" .Values.host (index .Values.nodePorts "keycloak")) -}}
+{{- end -}}
+
 {{- define "big.labels" -}}
 app.kubernetes.io/name: {{ .name }}
 app.kubernetes.io/instance: {{ .root.Release.Name }}

infra/helm/big-reference/templates/config.yaml

@@ -40,5 +40,5 @@ metadata:
 {{- include "big.labels" (dict "root" $ "name" (printf "portal-config-%s" $realm)) | nindent 4 }}
 data:
   config.json: |
-    { "authority": "{{ printf "http://%s:%v" $.Values.host (index $.Values.nodePorts "keycloak") }}/realms/{{ $realm }}" }
+    { "authority": "{{ include "big.keycloakUrl" $ }}/realms/{{ $realm }}" }
 {{- end }}

infra/helm/big-reference/templates/deployments.yaml

@@ -28,7 +28,7 @@ spec:
       {{- range $w.files }}
       {{- if hasPrefix "portal-config-" .configMap }}
       annotations:
-        checksum/portal-config: {{ printf "%s|%v" $.Values.host (index $.Values.nodePorts "keycloak") | sha256sum }}
+        checksum/portal-config: {{ include "big.keycloakUrl" $ | sha256sum }}
       {{- end }}
       {{- end }}
       labels:

infra/helm/big-reference/values.yaml

@@ -25,6 +25,11 @@
 # string, so browser tokens and the BFF's discovered issuer agree.
 host: 192.168.122.100
 
+# Set when a TLS proxy outside the cluster publishes Keycloak: the full origin, no
+# trailing slash. It replaces `host` + Keycloak's NodePort as the issuer and the
+# portals' authority (runbook, "Publishing through the labs Caddy").
+keycloakUrl: ""
+
 # Set when pulling from a private registry (e.g. the Gitea Container Registry).
 imagePullSecrets: []
 
@@ -71,6 +76,7 @@ envGroups:
   oz:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
+    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: openzaak.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: oz-db
@@ -93,6 +99,7 @@ envGroups:
   nrc:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
+    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: nrc.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: nrc-db
@@ -268,7 +275,7 @@ workloads:
       # Pin the issuer to the address the browser uses, and let backchannel calls
       # keep using keycloak:8080 — the BFF discovers metadata in-cluster and gets
       # this issuer back, which is what browser tokens carry (infra/host-browser.yml).
-      KC_HOSTNAME: "http://{{ .Values.host }}:{{ index .Values.nodePorts \"keycloak\" }}"
+      KC_HOSTNAME: '{{ include "big.keycloakUrl" . }}'
       KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
     ports: [{ name: http, port: 8080 }]
     # TCP, not /health/ready on the management port: nothing here gates on realm