Compare commits

..
Author SHA1 Message Date
notandClaude Opus 5 399d110663 docs(arch): ADR-0035 and the runbook section for publishing the stack (refs #177)
CI / k8s (pull_request) Successful in 6s
CI / build (pull_request) Successful in 1m28s
CI / lint (pull_request) Successful in 1m51s
CI / unit (pull_request) Successful in 1m12s
CI / frontend (pull_request) Successful in 2m11s
CI / mutation (pull_request) Successful in 3m45s
CI / verify-stack (pull_request) Successful in 8m15s
The ADR records why the edge is in the cluster rather than on the Fedora host —
routing and certificates should be state a `helm upgrade` can see — and the
three costs that buys: the host forward nobody in the cluster can repair, the
Let's Encrypt rate limit that makes `persistence.storageClass` non-optional, and
publishing behandel and beheer to the internet behind synthetic accounts.

Runbook §10 is the operational half: the five DNS records, the two firewalld
rules (including the masquerade that makes the return path work), and the
symptoms each missing piece produces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
notandClaude Opus 5 56cba9c340 feat(k8s): terminate TLS in the cluster for a public domain (refs #177)
`public.domain` is the whole switch. Empty — the default, and what compose, CI
and a laptop cluster use — renders nothing new and leaves every manifest as it
was. Set it and templates/edge.yaml adds a Caddy deployment that gets its own
certificates from Let's Encrypt and proxies the five browser-facing hostnames to
the ClusterIP services, so a public deployment doesn't use their NodePorts at
all.

Caddy rather than an ingress controller because the four portals already run
caddy:2-alpine (ADR-0034, whose ceiling note called exactly this out): no new
dependency, no cert-manager, no CRDs, no Ingress objects for five hostnames that
never change. The Fedora host keeps only a layer-4 forward of 80/443, because
the public IP is there and nothing in the cluster can claim it.

KC_HOSTNAME and the portals' config.json now both come from `big.keycloakUrl`,
so the issuer a token carries and the authority the BFF discovers are one string
by construction (ADR-0010) rather than by two templates agreeing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
notandClaude Opus 5 88fda30008 test(k8s): pin the issuer, the portal authority and the public edge (refs #177)
Keycloak pins one issuer and each portal is configured with one authority; when
they drift the symptom lands three services away — a login that bounces back
logged out, or a 401 from the BFF (ADR-0010) — so assert they are the same
string. The same check states what publishing the stack has to mean: with
`public.domain` set the five hostnames are served and both halves become
`https://auth.<domain>`, and with it empty nothing of the edge renders, which is
what compose, CI and a laptop cluster depend on.

Red: the chart has no `public.domain`, so setting it changes nothing and no
hostname is published.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
17 changed files with 408 additions and 305 deletions
-121
View File
@@ -1,121 +0,0 @@
name: Deploy to Talos
# A merge to main ships the stack to the Talos cluster on the lab server
# (docs/runbooks/kubernetes-talos.md §9). PR CI is the merge gate, so main is
# green by construction — this workflow only deploys.
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
# Queue deploys, never cancel one: a helm upgrade killed half-way leaves the
# release in `pending-upgrade` and the next run has to be unwedged by hand.
concurrency:
group: deploy-talos
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-latest
env:
# The Talos VM as seen from the Fedora host (libvirt guest IP), and the
# address a browser uses to reach the cluster. `localhost` is deliberate:
# the portals' PKCE needs a secure context, so they are reached over
# `kubectl port-forward` — runbook §5. Override with repo variables.
TALOS_VM_IP: ${{ vars.TALOS_VM_IP }}
TALOS_HOST: ${{ vars.TALOS_HOST }}
# Set it when the labs Caddy publishes the portals: Keycloak's public https
# origin, e.g. https://big-auth.labs.respellion.tech (runbook, "Publishing
# through the labs Caddy").
KEYCLOAK_URL: ${{ vars.KEYCLOAK_URL }}
# `true` fills in the medewerker OTP step for the public demo (chart value
# demo.otpAutofill). The fixture secret is committed: demo only.
OTP_AUTOFILL: ${{ vars.OTP_AUTOFILL }}
steps:
- uses: https://github.com/actions/checkout@v4
# Pinned static binaries, the same URLs the Talos runbook §0 gives a
# developer and the same helm the `k8s` CI job uses — no action to vet.
- name: Install kubectl, helm and crane
run: |
set -euo pipefail
bin="$HOME/.local/bin"; mkdir -p "$bin"
curl -sSLo "$bin/kubectl" https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > "$bin/helm"
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > "$bin/crane"
chmod +x "$bin"/{kubectl,helm,crane}
echo "$bin" >> "$GITHUB_PATH"
# The cluster's API and its registry are only reachable through the Fedora
# host, so forward both to the runner. 30141 is the openbaar portal, for
# the smoke at the end.
- name: Tunnel the Talos API + registry through the Fedora host
env:
SSH_KEY: ${{ secrets.TALOS_SSH_KEY }}
run: |
set -euo pipefail
: "${TALOS_VM_IP:=192.168.122.173}"
umask 077
printf '%s\n' "$SSH_KEY" > ~/.ssh_talos
ssh -i ~/.ssh_talos -o StrictHostKeyChecking=no -o IdentitiesOnly=yes \
-o ExitOnForwardFailure=yes -p 6667 -f -N \
-L 6443:$TALOS_VM_IP:6443 \
-L 30500:$TALOS_VM_IP:30500 \
-L 30141:$TALOS_VM_IP:30141 \
user@labs.respellion.tech
# The kubeconfig's server must be https://127.0.0.1:6443 — Talos puts
# 127.0.0.1 in the apiserver cert SANs, so TLS verification still holds
# through the tunnel.
- name: Write the kubeconfig
env:
KUBECONFIG_B64: ${{ secrets.TALOS_KUBECONFIG }}
run: |
set -euo pipefail
umask 077
base64 -d <<< "$KUBECONFIG_B64" > "$RUNNER_TEMP/kubeconfig"
echo "KUBECONFIG=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_ENV"
kubectl --kubeconfig "$RUNNER_TEMP/kubeconfig" get nodes
# Idempotent; also makes a first deploy onto a bare cluster work. The
# registry's storage is an emptyDir, so a replaced pod loses the images —
# which the push in the next step puts back anyway.
- name: Ensure the in-cluster registry
run: make k8s-registry
# Push through the tunnel (localhost), pull from the node's own NodePort
# (the address in the Talos registry-mirror patch) — same registry, two
# names, so the two `make` calls get different K8S_REGISTRY values.
- name: Build and push the images
run: make k8s-images K8S_REGISTRY=localhost:30500
# k8s-reseed = seed configmaps + helm upgrade + re-run the bootstrap jobs.
# The jobs are idempotent, and deleting them first is what keeps a changed
# Job template from wedging the upgrade (`cannot patch … with kind Job`).
- name: Deploy the chart
run: |
make k8s-reseed \
TALOS_HOST=${TALOS_HOST:-localhost} \
K8S_REGISTRY=${TALOS_VM_IP:-192.168.122.173}:30500 \
K8S_SET="${KEYCLOAK_URL:+--set keycloakUrl=$KEYCLOAK_URL} --set demo.otpAutofill=${OTP_AUTOFILL:-false}"
# `dev` is a mutable tag and helm sees an unchanged pod template, so the
# new images only land on a restart (pullPolicy is already Always).
- name: Roll the services onto the new images
run: |
set -euo pipefail
svcs="acl domain bff event-subscriber projection-api self-service openbaar behandel beheer"
kubectl -n big rollout restart deploy $svcs
kubectl -n big rollout status --timeout=300s deploy $svcs
# Proves portal → Caddy → BFF → projection end to end. An empty register is
# a pass; a 502 or a timeout is not.
- name: Smoke the public register
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
- name: Pods on failure
if: failure()
run: kubectl -n big get pods,jobs || true
+1
View File
@@ -352,6 +352,7 @@ K8S_IMAGES := acl domain bff event-subscriber projection-api self-service open
k8s-lint:
helm lint $(K8S_CHART)
helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null
python3 infra/helm/check-issuer.py
## k8s-drift: fail if compose and the Helm chart describe different stacks
# Compose is CI-canonical (ADR-0033) and the chart is a transcription of it; this
@@ -0,0 +1,102 @@
# ADR-0035: The public TLS edge is a Caddy deployment in the cluster
- **Status:** Accepted
- **Date:** 2026-09-18
- **Deciders:** Respellion engineering
- **Slice:** [#177](https://git.labs.respellion.tech/eho/register-referentie/issues/177)
## Context
The stack deploys to a Talos VM on the lab server (ADR-0033, issue #175). Until now it was
only usable through five SSH port-forwards: the portals' OIDC flow uses PKCE, PKCE needs
`crypto.subtle`, and browsers expose that only in a **secure context** — HTTPS or an origin
on `localhost`. A NodePort on the VM's address is neither, so the deployment was pinned to
`host: localhost` and every viewer had to forward all five browser-facing ports (a portal
without Keycloak on the same `localhost:30180` fails on the discovery document).
That is not a demo anyone can be sent a link to. We want public hostnames with real
certificates — and we want the routing and the certificates to be cluster state, not
host-side configuration that no `helm upgrade` can see.
The public IP is on the Fedora host (`46.224.220.37`); the cluster is a libvirt guest
behind it.
## Decision
**Terminate TLS in the cluster, with a Caddy deployment rendered by the chart
(`templates/edge.yaml`), and give the Fedora host nothing but a layer-4 forward.**
- `public.domain` is the single switch. Empty — the default, and what compose and CI use —
renders nothing: the stack is reached on its NodePorts and `host` pins the OIDC origin
exactly as before. Set it, and the edge appears.
- `public.routes` maps a subdomain to an in-cluster `service:port`. Caddy proxies to the
**ClusterIP** services, so a public deployment does not use the browser-facing NodePorts
at all.
- Caddy obtains and renews certificates itself (ACME HTTP-01). There is no cert-manager.
- The host forwards `:80`/`:443` to two NodePorts with two `firewall-cmd
--add-forward-port` rules. No TLS, no routing, no per-service knowledge there — adding a
portal is a chart change, not a host change.
- `KC_HOSTNAME` and the portals' `config.json` stop being `host` + NodePort. Both now come
from one helper, `big.keycloakUrl`, so the issuer Keycloak pins and the authority the
portals are configured with cannot drift apart (ADR-0010).
### Alternatives considered
- **Caddy on the Fedora host.** Fewest moving parts — but the routing table and the
certificates would live outside the cluster, in a file no deployment touches, and adding
a portal would mean editing a host we deploy to over SSH. Rejected on exactly the ground
this ADR exists to record.
- **Traefik or ingress-nginx, plus cert-manager.** The conventional answer, and the right
one for a cluster with many teams and changing hostnames. Here it buys a controller, a
set of CRDs and Ingress objects to describe five hostnames that never change — and
cert-manager to do what Caddy already does unprompted.
- **A `LoadBalancer` service (MetalLB).** Solves address allocation, which is not the
problem; the node has exactly one address and it still is not the public one.
- **Keep the SSH forwards.** Free, and genuinely fine for one developer. It is not a demo
you can send to someone.
## Consequences
**Positive**
- No new dependency: the four portals already run `caddy:2-alpine` (ADR-0034), whose
ceiling note called this out — *"a real hostname makes TLS a one-line `Caddyfile`
change"*. This is that change.
- Routing is cluster state: `kubectl -n big get cm caddy-edge-config -o yaml` is the whole
truth about what is published, and `helm upgrade` is how it changes.
- The secure context is real, so `TALOS_HOST=localhost` and the five forwards disappear —
and with them the class of failure where a mismatched issuer logs the user out silently.
- Nothing changes for compose, CI or a laptop cluster: with `public.domain` empty the
rendered manifests are byte-identical to before.
**Negative / costs**
- The host forward is irreducible. Two firewalld rules, applied by hand once, with `sudo`
on a machine our pipeline reaches only over SSH. If someone rebuilds that host, the stack
is unreachable until they are re-applied, and nothing in the cluster can tell them so.
- **Certificates need a volume.** On the default `emptyDir` every pod restart asks Let's
Encrypt again, and its duplicate-certificate limit is five per week — a handful of
restarts and the edge serves an untrusted certificate for a week. `persistence.storageClass`
stops being optional for anything public (runbook §6).
- **All five hostnames are published, including `behandel` and `beheer`**, which approve
registrations and administer the register. They are protected by synthetic accounts with
well-known passwords, and by MFA on the medewerker realm (ADR-0031). That is a deliberate
choice for a demonstration environment holding synthetic data only, and it is the reason
this bullet is in the ADR rather than in a comment: if this stack ever holds anything
real, this decision is the first one to revisit.
- One more workload in the chart with no counterpart in compose — compose has no edge
because it has no hostname. The drift check (`make k8s-drift`) renders the defaults, so
it does not see it.
- `auth` is load-bearing: `big.keycloakUrl` builds the issuer from that subdomain, so
renaming the key in `public.routes` without the helper breaks every login. Both carry a
comment saying so.
- ponytail ceiling: one replica, no HSTS, no security headers beyond Caddy's defaults, no
rate limiting, and HTTP-01 rather than DNS-01 (so a wildcard certificate is not
available). Upgrade path in that order; DNS-01 first if the subdomain list ever grows.
## Coupling rules touched (CLAUDE.md §8)
None. §8.3 holds — the browser reaches a portal, the portal reverse-proxies its own BFF
group, and the edge is in front of all of it. The edge terminates TLS and routes by
hostname; it does not know what any service does.
-4
View File
@@ -31,10 +31,6 @@ and CI cannot drift:
> services by **container IP** (the runner can't reach published ports — see
> [gitea-actions-gotchas.md §5/§6](gitea-actions-gotchas.md)).
A second workflow, `.gitea/workflows/deploy.yaml`, deploys the stack to the Talos
cluster on the lab server when a PR is merged to `main` — see
[kubernetes-talos.md §9](kubernetes-talos.md) for its secrets and the SSH tunnel it needs.
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
`https://github.com/actions/setup-dotnet@v4`) per CLAUDE.md §8.7 and §15 — Gitea
Actions resolves them from GitHub.
+50 -74
View File
@@ -222,6 +222,9 @@ string, so the port the browser uses has to match the one baked into `config.jso
This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
`localhost:8180`); only the addresses differ.
All of this is what §10 removes: with a public domain the portals have real certificates,
so the browser gets its secure context and no forwarding is involved.
### The admin UIs work straight off the NodePorts
These are server-rendered and need no secure context, so they are reachable at the VM's
@@ -360,94 +363,67 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |
## 9. Deploying on merge to main
## 10. Publishing it on a public domain
`.gitea/workflows/deploy.yaml` runs the §3–§4 steps against the **lab server's** Talos VM
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
merge gate, so the workflow deploys without re-running the checks.
By default the stack has no hostname: it is reached on NodePorts, and §5's secure-context
problem forces `TALOS_HOST=localhost` plus five SSH forwards. Setting `public.domain` puts a
Caddy deployment in front of it that terminates TLS for real hostnames (ADR-0035), and the
forwards go away.
The cluster's API and registry are not exposed publicly, so the job forwards them over the
same SSH hop the Gitea-runner pipeline uses:
### Once, outside the cluster
**DNS** — five A records to the *host's* public address (the cluster is behind it):
```
ssh -p 6667 user@labs.respellion.tech -L 6443 -L 30500 -L 30141 → <TALOS_VM_IP>
register.<domain> mijn.<domain> behandel.<domain> beheer.<domain> auth.<domain> → 46.224.220.37
```
Consequences worth knowing:
- Images are **pushed** to `localhost:30500` (the tunnel) and **pulled** by the node from
`<TALOS_VM_IP>:30500` (its own NodePort, the address in the Talos registry-mirror patch).
Same registry, two names — hence the two `K8S_REGISTRY` values in the workflow.
- It calls `make k8s-reseed`, not `make k8s-up`: the bootstrap Jobs are idempotent, and
deleting them first is what stops a changed Job template from wedging `helm upgrade` (§7).
- `dev` is a mutable tag, so a `rollout restart` of the nine repo deployments is what
actually puts the new images in the pods.
- Deploys **queue** (`cancel-in-progress: false`): a helm upgrade killed half-way leaves the
release in `pending-upgrade`, which has to be unwedged by hand.
Settings, all on the repository in Gitea:
| Kind | Name | What |
|---|---|---|
| Secret | `TALOS_SSH_KEY` | private key for `user@labs.respellion.tech` (the Fedora host) |
| Secret | `TALOS_KUBECONFIG` | base64 of the kubeconfig, **`server: https://127.0.0.1:6443`** — Talos puts `127.0.0.1` in the apiserver cert SANs, so TLS still verifies through the tunnel |
| Variable | `TALOS_VM_IP` | the VM's libvirt address (default `192.168.122.173`) |
| Variable | `TALOS_HOST` | the browser-facing host baked into Keycloak's issuer (default `localhost`, see §5) |
The last step smokes `GET /openbaar/register` through the openbaar portal, which exercises
portal → Caddy → BFF → projection. An empty register passes; a 502 does not.
Not covered: the portals still need `make k8s-portals` (or an SSH forward) to be usable in a
browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the
upgrade path.
## Publishing through the labs Caddy
The portals can be reached on real hostnames through the Caddy that already fronts
`*.labs.respellion.tech` (repo `Infra`, `infra/development/`). The chain:
```
browser → Caddy (labs server, TLS) → openssh-server:3014x/30180
→ reverse SSH tunnel → Fedora host → <TALOS_VM_IP>:3014x/30180 (NodePorts)
```
| URL | NodePort |
|---|---|
| `https://big-register.labs.respellion.tech` | 30141 openbaar |
| `https://big-mijn.labs.respellion.tech` | 30140 self-service |
| `https://big-behandel.labs.respellion.tech` | 30142 behandel |
| `https://big-beheer.labs.respellion.tech` | 30143 beheer |
| `https://big-auth.labs.respellion.tech` | 30180 Keycloak (`/admin` blocked) |
HTTPS makes the portals a secure context, so PKCE works without port-forwards — but
Keycloak's issuer must be the public origin. Deploy with it:
**The host's forward** — the public IP is on the Fedora host, so it has to hand 80/443 to
the node. This is the only host-side configuration, and it is dumb layer 4:
```bash
make k8s-up TALOS_HOST=localhost K8S_REGISTRY=<TALOS_HOST>:30500 \
K8S_SET="--set keycloakUrl=https://big-auth.labs.respellion.tech"
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32080
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=443:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32443
sudo firewall-cmd --permanent --zone=public --add-masquerade
sudo firewall-cmd --reload
```
For deploy-on-merge, set the repository variable `KEYCLOAK_URL` to the same value.
With it set, the `localhost` port-forwards (§5) no longer log in: the issuer is one string.
`--add-masquerade` is what makes the return path work: without it the node answers the
client's address directly and the reply never goes back through the host.
Staff logins still hit the enforced OTP step. For a demo, set the repository variable
`OTP_AUTOFILL=true` (chart value `demo.otpAutofill`): Keycloak then uses the `big-demo`
theme, which fills in and submits the code from the fixture secret, so the step is visible
but needs no authenticator. Keycloak restarts when the value flips. Demo only — the secret
is committed.
**A StorageClass.** Caddy's certificates live in `/data`, which is an `emptyDir` unless
`persistence.storageClass` is set (§6). Let's Encrypt allows five duplicate certificates per
week, so on an `emptyDir` a handful of pod restarts leaves the edge serving an untrusted
certificate until the limit resets. Install local-path first (§6).
The theme lives in `infra/keycloak/themes/big-demo/` and is seeded as the `rr-kc-theme`
ConfigMap by `infra/helm/seed-configmaps.sh` on every deploy. Keycloak runs `start-dev`,
which doesn't cache themes, so an edit shows up about a minute after the ConfigMap changes.
A *new* theme file also needs a key in the seed script and a path in the keycloak `files`
in `values.yaml`.
### Deploy
One-time setup:
```bash
make k8s-up TALOS_HOST=<domain-facing name> K8S_REGISTRY=<TALOS_VM_IP>:30500 \
K8S_SET='--set public.domain=<domain> --set public.email=<ops address> --set persistence.storageClass=local-path'
```
1. Fedora host: install `infra/development/big-portals-tunnel.service` from the Infra repo
(instructions in the file).
2. Labs server: deploy the Infra `Caddyfile` + `compose.yml` (Caddy joins the
`openssh_default` network to reach the tunnel ends).
`public.domain` is the only switch: with it empty nothing in `templates/edge.yaml` renders
and the stack behaves exactly as §4 describes. With it set, `KC_HOSTNAME` and the portals'
`config.json` both become `https://auth.<domain>` — one helper builds both, so the issuer
and the authority cannot drift (ADR-0010).
Watch the first certificate being issued:
```bash
kubectl -n big logs deploy/caddy-edge -f # "certificate obtained successfully"
curl -sSI https://register.<domain>/openbaar/register | head -1
```
### When it doesn't work
| Symptom | Cause |
|---|---|
| ACME fails with `connection refused` or a timeout on the HTTP-01 challenge | the host's 80 → 32080 forward is missing, or `--add-masquerade` is |
| ACME fails with `NXDOMAIN` / `no such host` | the A record isn't there yet. Caddy retries with backoff; fix DNS and it recovers |
| An untrusted certificate after several restarts | the Let's Encrypt duplicate limit, from certificates on an `emptyDir` — see above |
| The portal loads but login bounces back logged out | `public.domain` changed without the portals rolling. The chart hashes the issuer into their pod template, so `helm upgrade` should do it — check `kubectl -n big describe deploy/self-service` |
| `404` from the edge on a name that should work | the name isn't in `public.routes`; Caddy answers 404 for a Host it has no site block for |
## What is not ported
-5
View File
@@ -57,9 +57,6 @@ services:
# share this anchor and ignore it — they don't run uwsgi.
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
# Same lever for oz-celery: unset, the worker forks one process per CPU (22 on the lab node,
# ~225 MB each), which OOM-killed the shared runner mid-verify-stack. Only celery reads it.
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: openzaak.conf.docker
SECRET_KEY: ${OZ_SECRET_KEY:-dev-only-not-for-production}
DB_HOST: oz-db
@@ -147,8 +144,6 @@ services:
# 1 uWSGI worker, not the image default of 4×4 (#147) — see the oz-env note above.
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
# Two celery workers, not one per CPU — see the oz-env note above.
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: nrc.conf.docker
SECRET_KEY: ${NRC_SECRET_KEY:-dev-only-not-for-production}
DB_HOST: nrc-db
+11 -13
View File
@@ -94,10 +94,6 @@ volumes:
{{- with .defaultMode }}
defaultMode: {{ . }}
{{- end }}
{{- with .items }}
items:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- with $w.data }}
- name: data
@@ -129,26 +125,28 @@ volumes:
{{/*
Env list from a map. Every value is run through `tpl`, so values.yaml can name
cluster-internal hosts ({{ .Release.Namespace }}) and the node address
({{ .Values.host }}) without the chart hard-coding either. A value that renders
empty is left out, which is how a setting is made conditional on a chart value.
({{ .Values.host }}) without the chart hard-coding either.
*/}}
{{- define "big.env" -}}
{{- $root := index . 0 -}}
{{- range $k, $v := index . 1 }}
{{- $val := tpl (toString $v) $root }}
{{- if $val }}
- name: {{ $k }}
value: {{ $val | quote }}
{{- end }}
value: {{ tpl (toString $v) $root | quote }}
{{- end }}
{{- end -}}
{{/*
The origin a browser reaches Keycloak on: the issuer Keycloak pins and the
authority the portals use, from one place so they cannot drift (ADR-0010).
The origin a browser reaches Keycloak on, and so the issuer its tokens carry and
the authority the portals are configured with (ADR-0010). With a public edge that
is the `auth` hostname on `public.domain` — which must stay in step with the `auth`
key in `public.routes`; without one it is the node address plus Keycloak's NodePort.
*/}}
{{- define "big.keycloakUrl" -}}
{{- .Values.keycloakUrl | default (printf "http://%s:%v" .Values.host (index .Values.nodePorts "keycloak")) -}}
{{- if .Values.public.domain -}}
https://auth.{{ .Values.public.domain }}
{{- else -}}
http://{{ .Values.host }}:{{ index .Values.nodePorts "keycloak" }}
{{- end -}}
{{- end -}}
{{- define "big.labels" -}}
@@ -0,0 +1,136 @@
{{- /*
The public TLS edge (ADR-0035). Rendered only when `public.domain` is set; with it
empty the stack is reached on the NodePorts below and nothing here exists.
Caddy rather than an ingress controller: the four portals already run caddy:2-alpine,
so this adds no dependency, and it does ACME itself — no cert-manager, no CRDs, no
Ingress objects for five hostnames that never change. It proxies to the ClusterIP
services, so the browser-facing NodePorts are not involved in a public deployment.
The public IP lives on the Fedora host, which forwards 80/443 to the two NodePorts
below. That forward is dumb L4 — no TLS, no routing — see the runbook.
*/}}
{{- if .Values.public.domain }}
{{- $pub := .Values.public }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: caddy-edge-config
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
data:
Caddyfile: |
{
{{- with $pub.email }}
email {{ . }}
{{- end }}
}
{{- range $sub, $target := $pub.routes }}
{{ $sub }}.{{ $pub.domain }} {
reverse_proxy {{ $target }}
}
{{- end }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: caddy-edge
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: caddy-edge
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
annotations:
# A ConfigMap mounted with subPath never updates in place, so a changed
# Caddyfile has to roll the pod.
checksum/caddyfile: {{ printf "%s|%v|%v" $pub.domain $pub.email $pub.routes | sha256sum }}
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 8 }}
spec:
containers:
- name: caddy-edge
image: {{ $pub.image }}
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
# TCP, not HTTP: a GET with no matching Host gets a 404 from Caddy, which
# would fail an httpGet probe for a perfectly healthy edge.
readinessProbe:
tcpSocket: { port: 443 }
volumeMounts:
- name: config
mountPath: /etc/caddy/Caddyfile
subPath: Caddyfile
readOnly: true
- name: data
mountPath: /data
- name: run
mountPath: /config
volumes:
- name: config
configMap:
name: caddy-edge-config
- name: run
emptyDir: {}
- name: data
{{- if .Values.persistence.storageClass }}
persistentVolumeClaim:
claimName: caddy-edge-data
{{- else }}
# Certificates live here. On an emptyDir every pod restart asks Let's
# Encrypt again, and its duplicate-certificate limit is five per week —
# set persistence.storageClass for anything that stays up.
emptyDir: {}
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: caddy-edge
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
spec:
type: NodePort
selector:
app.kubernetes.io/name: caddy-edge
app.kubernetes.io/instance: {{ .Release.Name }}
ports:
- name: http
port: 80
targetPort: 80
nodePort: {{ $pub.nodePorts.http }}
- name: https
port: 443
targetPort: 443
nodePort: {{ $pub.nodePorts.https }}
{{- if .Values.persistence.storageClass }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: caddy-edge-data
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
# Keep the certificates when the release is uninstalled — re-issuing them on
# every reinstall is what burns the rate limit.
annotations:
helm.sh/resource-policy: keep
spec:
accessModes: [ReadWriteOnce]
storageClassName: {{ .Values.persistence.storageClass }}
resources:
requests:
storage: 128Mi
{{- end }}
{{- end }}
+27 -32
View File
@@ -25,17 +25,6 @@
# string, so browser tokens and the BFF's discovered issuer agree.
host: 192.168.122.100
# Set when a TLS proxy outside the cluster publishes Keycloak: the full origin, no
# trailing slash. It replaces `host` + Keycloak's NodePort as the issuer and the
# portals' authority (runbook, "Publishing through the labs Caddy").
keycloakUrl: ""
demo:
# Fill in and submit the medewerker OTP step from the fixture secret, so a public
# demo shows MFA enforced without an authenticator: makes the big-demo theme
# (infra/keycloak/themes/big-demo) Keycloak's default. Demo only: the secret is committed.
otpAutofill: false
# Set when pulling from a private registry (e.g. the Gitea Container Registry).
imagePullSecrets: []
@@ -60,6 +49,32 @@ persistence:
# the data across pod restarts.
storageClass: ""
# The public TLS edge (ADR-0035). Empty `domain` = no edge at all: nothing in
# templates/edge.yaml is rendered and the stack is reached on the NodePorts below,
# with `host` above pinning the OIDC origin.
#
# Set it and an in-cluster Caddy terminates TLS for `<sub>.<domain>`, gets its own
# certificates from Let's Encrypt and proxies to the ClusterIP services. The node
# only has to be reachable on the two NodePorts here — the Fedora host forwards
# 80/443 to them (see docs/runbooks/kubernetes-talos.md).
public:
domain: ""
# ACME registration address; Let's Encrypt uses it for expiry warnings.
email: ""
image: docker.io/library/caddy:2-alpine
# <subdomain>: <in-cluster service:port>. `auth` is not free-form — big.keycloakUrl
# builds the pinned issuer from it.
routes:
register: openbaar:80
mijn: self-service:80
behandel: behandel:80
beheer: beheer:80
auth: keycloak:8080
# Where the host's 80/443 forward lands. Not 30080/30443: 30080 is the BFF.
nodePorts:
http: 32080
https: 32443
# The only place a port is published outside the cluster. A workload listed here
# gets a NodePort on its single port; everything else stays ClusterIP.
nodePorts:
@@ -82,7 +97,6 @@ envGroups:
oz:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: openzaak.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: oz-db
@@ -105,7 +119,6 @@ envGroups:
nrc:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: nrc.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: nrc-db
@@ -283,29 +296,11 @@ workloads:
# this issuer back, which is what browser tokens carry (infra/host-browser.yml).
KC_HOSTNAME: '{{ include "big.keycloakUrl" . }}'
KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
# Only rendered with demo.otpAutofill (big.env skips empty values); off, Keycloak
# keeps its stock theme and the mounted big-demo theme is unused.
KC_SPI_THEME_DEFAULT: '{{ if .Values.demo.otpAutofill }}big-demo{{ end }}'
# Behind a TLS proxy (keycloakUrl) the dynamic backchannel URLs — token,
# userinfo, certs — take their scheme from the request, which reaches Keycloak
# as plain http; trusting X-Forwarded-Proto keeps them https so the browser
# doesn't block them as mixed content. In-cluster calls send no such header.
KC_PROXY_HEADERS: xforwarded
ports: [{ name: http, port: 8080 }]
# TCP, not /health/ready on the management port: nothing here gates on realm
# import, and a wrong health path would leave the Service with no endpoints.
probe: { tcpSocket: { port: 8080 }, initialDelaySeconds: 15 }
files:
- { configMap: rr-kc-realms, mountPath: /opt/keycloak/data/import }
# infra/keycloak/themes/big-demo, seeded by infra/helm/seed-configmaps.sh.
- configMap: rr-kc-theme
mountPath: /opt/keycloak/themes/big-demo
items:
- { key: login.properties, path: login/theme.properties }
- { key: otp-autofill.js, path: login/resources/js/otp-autofill.js }
- { key: account.properties, path: account/theme.properties }
- { key: admin.properties, path: admin/theme.properties }
- { key: email.properties, path: email/theme.properties }
files: [{ configMap: rr-kc-realms, mountPath: /opt/keycloak/data/import }]
# ── Flowable (S-03) ─────────────────────────────────────────────────────────
flowable-db:
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Fail when the pinned issuer and the portals' OIDC authority stop agreeing.
Keycloak pins one issuer (`KC_HOSTNAME`) and each portal is configured with one
authority (`config.json`). A browser token carries the first; the BFF validates
against what it discovers from the second (ADR-0010). When the two drift the
symptom is three services away — a login that bounces back logged out, or a 401
from the BFF — so the chart builds both from one helper and this asserts it.
It also pins the two halves of the public edge (ADR-0035): that setting
`public.domain` actually publishes the hostnames, and that leaving it empty
renders no edge at all, which is what compose, CI and a laptop cluster rely on.
Run it with `make k8s-lint`. No cluster needed.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
CHART = Path(__file__).resolve().parent / "big-reference"
DOMAIN = "example.test"
def render(*sets: str) -> str:
argv = ["helm", "template", "big", str(CHART), "-n", "big"]
for s in sets:
argv += ["--set", s]
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
sys.exit(f"helm template failed:\n{proc.stderr}")
return proc.stdout
def issuer(out: str) -> str:
"""The value of KC_HOSTNAME in the rendered manifests."""
m = re.search(r"name: KC_HOSTNAME\n\s+value: \"(\S+)\"", out)
return m[1] if m else ""
def authorities(out: str) -> set[str]:
"""Every portal's OIDC authority, with the realm path stripped."""
found = set()
for line in re.findall(r'\{ "authority": .* \}', out):
url = json.loads(line)["authority"]
found.add(url.rsplit("/realms/", 1)[0])
return found
def main() -> int:
problems = []
public = render(f"public.domain={DOMAIN}")
if issuer(public) != f"https://auth.{DOMAIN}":
problems.append(f" with public.domain set, KC_HOSTNAME is {issuer(public)!r}, not https://auth.{DOMAIN}")
if authorities(public) != {f"https://auth.{DOMAIN}"}:
problems.append(f" with public.domain set, the portals point at {sorted(authorities(public))}")
for host in (f"register.{DOMAIN}", f"mijn.{DOMAIN}", f"behandel.{DOMAIN}", f"beheer.{DOMAIN}", f"auth.{DOMAIN}"):
if host not in public:
problems.append(f" {host} is not published by the edge")
private = render()
if authorities(private) != {issuer(private)}:
problems.append(f" by default the portals point at {sorted(authorities(private))}, the issuer is {issuer(private)!r}")
if "caddy-edge" in private:
problems.append(" the edge renders with no public.domain — compose, CI and a laptop cluster expect nothing")
if problems:
print("the chart's OIDC origin is inconsistent:\n" + "\n".join(problems))
print("\nBoth halves come from the `big.keycloakUrl` helper — change it, not one caller.")
return 1
print(f"issuer + portal authority agree, with and without a public domain")
return 0
if __name__ == "__main__":
raise SystemExit(main())
-9
View File
@@ -30,15 +30,6 @@ seed() { # name <kubectl --from-file args...>
seed rr-oz-config --from-file="$repo/infra/openzaak/setup_configuration/"
seed rr-nrc-config --from-file="$repo/infra/opennotificaties/setup_configuration/"
seed rr-kc-realms --from-file="$repo/infra/keycloak/realms/"
# The big-demo login theme (demo.otpAutofill). ConfigMap keys are flat, so each
# file gets a key here and its path back in the keycloak `files` in values.yaml.
theme="$repo/infra/keycloak/themes/big-demo"
seed rr-kc-theme \
--from-file=login.properties="$theme/login/theme.properties" \
--from-file=otp-autofill.js="$theme/login/resources/js/otp-autofill.js" \
--from-file=account.properties="$theme/account/theme.properties" \
--from-file=admin.properties="$theme/admin/theme.properties" \
--from-file=email.properties="$theme/email/theme.properties"
seed rr-objecttypen-config --from-file="$repo/infra/objecttypen/setup_configuration/"
seed rr-objecten-config --from-file="$repo/infra/objecten/setup_configuration/"
# register.py + the RegisterRecord JSON schema (the __pycache__ dir is skipped:
@@ -1,4 +0,0 @@
# The chart makes big-demo the default for every theme type, and Keycloak does not
# fall back for a type a theme lacks (the account page then fails), so each type is
# declared as a plain child of Keycloak 26's own default.
parent=keycloak.v3
@@ -1,4 +0,0 @@
# The chart makes big-demo the default for every theme type, and Keycloak does not
# fall back for a type a theme lacks (the admin page then fails), so each type is
# declared as a plain child of Keycloak 26's own default.
parent=keycloak.v2
@@ -1,4 +0,0 @@
# The chart makes big-demo the default for every theme type, and Keycloak does not
# fall back for a type a theme lacks (the email page then fails), so each type is
# declared as a plain child of Keycloak 26's own default.
parent=keycloak
@@ -1,24 +0,0 @@
// RFC 6238 with Keycloak's default policy (HmacSHA1, 6 digits, 30 s) over the
// raw bytes of the medewerker fixture secret — same as tests/e2e/keycloak-login.ts.
document.addEventListener('DOMContentLoaded', async () => {
const input = document.querySelector('input[name="otp"]');
if (!input || !input.form) return;
const key = await crypto.subtle.importKey('raw',
new TextEncoder().encode('BIGMEDEWERKEROTPSEED'), { name: 'HMAC', hash: 'SHA-1' }, false, ['sign']);
// A code is single-use, so a second login in the same window spends the next
// counter (Keycloak's look-ahead accepts it). Past that, fill but don't submit,
// so a rejected code can't turn into a submit loop.
const now = Math.floor(Date.now() / 30000);
let last = -1;
try { last = Number(sessionStorage.getItem('big-otp-counter')) || -1; } catch {}
const counter = Math.max(now, last + 1);
const msg = new DataView(new ArrayBuffer(8));
msg.setBigUint64(0, BigInt(counter));
const mac = new Uint8Array(await crypto.subtle.sign('HMAC', key, msg.buffer));
const o = mac[19] & 0x0f;
const n = ((mac[o] & 0x7f) << 24 | mac[o + 1] << 16 | mac[o + 2] << 8 | mac[o + 3]) % 1e6;
input.value = String(n).padStart(6, '0');
if (counter > now + 1) return;
try { sessionStorage.setItem('big-otp-counter', String(counter)); } catch {}
input.form.requestSubmit();
});
@@ -1,11 +0,0 @@
# Demo login theme for the public Talos deployment: keycloak.v2 plus a script that
# fills in and submits the medewerker OTP step from the committed fixture secret
# (docs/runbooks/keycloak.md). Only used when the chart's demo.otpAutofill is on —
# it then becomes Keycloak's default theme. Never enable it anywhere real.
#
# Add styles, messages or template overrides here as in any Keycloak theme
# (https://www.keycloak.org/ui-customization/themes); new files must also be
# listed in infra/helm/seed-configmaps.sh and the keycloak `files` in values.yaml.
parent=keycloak.v2
import=common/keycloak
scripts=js/otp-autofill.js
+1
View File
@@ -56,6 +56,7 @@ nav:
- "ADR-0032: Werkbak live refresh": architecture/adr-0032-werkbak-live-refresh.md
- "ADR-0033: Kubernetes via one Helm chart": architecture/adr-0033-kubernetes-via-one-helm-chart.md
- "ADR-0034: Caddy serves the portals": architecture/adr-0034-caddy-serves-the-portals.md
- "ADR-0035: Public TLS edge in the cluster": architecture/adr-0035-public-tls-edge-in-cluster.md
- FDS-architectuur:
- Overzicht: architecture/fds/README.md
- Componentview (L3): architecture/fds/c4-component-view.md