Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
399d110663 | ||
|
|
56cba9c340 | ||
|
|
88fda30008 | ||
|
|
9d7e8e5b65 |
@@ -41,6 +41,27 @@ jobs:
|
||||
nuget-${{ runner.os }}-
|
||||
- run: make lint
|
||||
|
||||
# The Helm chart's only automated gate: it renders and schema-checks the whole
|
||||
# stack, and checks it still describes the same stack as the compose file
|
||||
# (ADR-0033). No cluster involved — see docs/runbooks/kubernetes-talos.md.
|
||||
k8s:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: https://github.com/actions/checkout@v4
|
||||
# helm as its pinned static binary rather than a marketplace action: one URL,
|
||||
# the same one the Talos runbook §0 gives a developer, and no third-party
|
||||
# action to vet (CLAUDE.md §13). The drift check also needs `docker compose`,
|
||||
# which the runner already has (see docs/runbooks/ci.md).
|
||||
- name: Install helm
|
||||
run: |
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz \
|
||||
| tar xz -O linux-amd64/helm > "$HOME/.local/bin/helm"
|
||||
chmod +x "$HOME/.local/bin/helm"
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
- run: make k8s-lint
|
||||
- run: make k8s-drift
|
||||
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
|
||||
@@ -43,7 +43,7 @@ export DOCKER_HOST := unix://$(PODMAN_SOCK)
|
||||
endif
|
||||
endif
|
||||
|
||||
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
|
||||
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-drift k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
|
||||
|
||||
## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions)
|
||||
## `verify` is the live-stack stage (full stack up once → ACL + notification checks).
|
||||
@@ -352,6 +352,14 @@ K8S_IMAGES := acl domain bff event-subscriber projection-api self-service open
|
||||
k8s-lint:
|
||||
helm lint $(K8S_CHART)
|
||||
helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null
|
||||
python3 infra/helm/check-issuer.py
|
||||
|
||||
## k8s-drift: fail if compose and the Helm chart describe different stacks
|
||||
# Compose is CI-canonical (ADR-0033) and the chart is a transcription of it; this
|
||||
# compares what each one deploys — workload names and resolved images. Needs
|
||||
# `docker compose` and `helm`, no cluster.
|
||||
k8s-drift:
|
||||
python3 infra/helm/check-drift.py
|
||||
|
||||
## k8s-registry: deploy the in-cluster image registry (NodePort 30500)
|
||||
k8s-registry:
|
||||
|
||||
@@ -126,8 +126,9 @@ Consequences of that shape, each chosen deliberately:
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- A second deployment description to keep in step with compose. Nothing enforces that
|
||||
today; a drift check belongs in CI (follow-up).
|
||||
- A second deployment description to keep in step with compose. `make k8s-drift` (#168)
|
||||
now enforces the part that bites — the workload set and the resolved images, with the
|
||||
four deviations below declared — but not per-workload env, ports or volumes.
|
||||
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
|
||||
one surfaces as `ContainerCreating`, not as a clear error.
|
||||
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# ADR-0035: The public TLS edge is a Caddy deployment in the cluster
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-09-18
|
||||
- **Deciders:** Respellion engineering
|
||||
- **Slice:** [#177](https://git.labs.respellion.tech/eho/register-referentie/issues/177)
|
||||
|
||||
## Context
|
||||
|
||||
The stack deploys to a Talos VM on the lab server (ADR-0033, issue #175). Until now it was
|
||||
only usable through five SSH port-forwards: the portals' OIDC flow uses PKCE, PKCE needs
|
||||
`crypto.subtle`, and browsers expose that only in a **secure context** — HTTPS or an origin
|
||||
on `localhost`. A NodePort on the VM's address is neither, so the deployment was pinned to
|
||||
`host: localhost` and every viewer had to forward all five browser-facing ports (a portal
|
||||
without Keycloak on the same `localhost:30180` fails on the discovery document).
|
||||
|
||||
That is not a demo anyone can be sent a link to. We want public hostnames with real
|
||||
certificates — and we want the routing and the certificates to be cluster state, not
|
||||
host-side configuration that no `helm upgrade` can see.
|
||||
|
||||
The public IP is on the Fedora host (`46.224.220.37`); the cluster is a libvirt guest
|
||||
behind it.
|
||||
|
||||
## Decision
|
||||
|
||||
**Terminate TLS in the cluster, with a Caddy deployment rendered by the chart
|
||||
(`templates/edge.yaml`), and give the Fedora host nothing but a layer-4 forward.**
|
||||
|
||||
- `public.domain` is the single switch. Empty — the default, and what compose and CI use —
|
||||
renders nothing: the stack is reached on its NodePorts and `host` pins the OIDC origin
|
||||
exactly as before. Set it, and the edge appears.
|
||||
- `public.routes` maps a subdomain to an in-cluster `service:port`. Caddy proxies to the
|
||||
**ClusterIP** services, so a public deployment does not use the browser-facing NodePorts
|
||||
at all.
|
||||
- Caddy obtains and renews certificates itself (ACME HTTP-01). There is no cert-manager.
|
||||
- The host forwards `:80`/`:443` to two NodePorts with two `firewall-cmd
|
||||
--add-forward-port` rules. No TLS, no routing, no per-service knowledge there — adding a
|
||||
portal is a chart change, not a host change.
|
||||
- `KC_HOSTNAME` and the portals' `config.json` stop being `host` + NodePort. Both now come
|
||||
from one helper, `big.keycloakUrl`, so the issuer Keycloak pins and the authority the
|
||||
portals are configured with cannot drift apart (ADR-0010).
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
- **Caddy on the Fedora host.** Fewest moving parts — but the routing table and the
|
||||
certificates would live outside the cluster, in a file no deployment touches, and adding
|
||||
a portal would mean editing a host we deploy to over SSH. Rejected on exactly the ground
|
||||
this ADR exists to record.
|
||||
- **Traefik or ingress-nginx, plus cert-manager.** The conventional answer, and the right
|
||||
one for a cluster with many teams and changing hostnames. Here it buys a controller, a
|
||||
set of CRDs and Ingress objects to describe five hostnames that never change — and
|
||||
cert-manager to do what Caddy already does unprompted.
|
||||
- **A `LoadBalancer` service (MetalLB).** Solves address allocation, which is not the
|
||||
problem; the node has exactly one address and it still is not the public one.
|
||||
- **Keep the SSH forwards.** Free, and genuinely fine for one developer. It is not a demo
|
||||
you can send to someone.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- No new dependency: the four portals already run `caddy:2-alpine` (ADR-0034), whose
|
||||
ceiling note called this out — *"a real hostname makes TLS a one-line `Caddyfile`
|
||||
change"*. This is that change.
|
||||
- Routing is cluster state: `kubectl -n big get cm caddy-edge-config -o yaml` is the whole
|
||||
truth about what is published, and `helm upgrade` is how it changes.
|
||||
- The secure context is real, so `TALOS_HOST=localhost` and the five forwards disappear —
|
||||
and with them the class of failure where a mismatched issuer logs the user out silently.
|
||||
- Nothing changes for compose, CI or a laptop cluster: with `public.domain` empty the
|
||||
rendered manifests are byte-identical to before.
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- The host forward is irreducible. Two firewalld rules, applied by hand once, with `sudo`
|
||||
on a machine our pipeline reaches only over SSH. If someone rebuilds that host, the stack
|
||||
is unreachable until they are re-applied, and nothing in the cluster can tell them so.
|
||||
- **Certificates need a volume.** On the default `emptyDir` every pod restart asks Let's
|
||||
Encrypt again, and its duplicate-certificate limit is five per week — a handful of
|
||||
restarts and the edge serves an untrusted certificate for a week. `persistence.storageClass`
|
||||
stops being optional for anything public (runbook §6).
|
||||
- **All five hostnames are published, including `behandel` and `beheer`**, which approve
|
||||
registrations and administer the register. They are protected by synthetic accounts with
|
||||
well-known passwords, and by MFA on the medewerker realm (ADR-0031). That is a deliberate
|
||||
choice for a demonstration environment holding synthetic data only, and it is the reason
|
||||
this bullet is in the ADR rather than in a comment: if this stack ever holds anything
|
||||
real, this decision is the first one to revisit.
|
||||
- One more workload in the chart with no counterpart in compose — compose has no edge
|
||||
because it has no hostname. The drift check (`make k8s-drift`) renders the defaults, so
|
||||
it does not see it.
|
||||
- `auth` is load-bearing: `big.keycloakUrl` builds the issuer from that subdomain, so
|
||||
renaming the key in `public.routes` without the helper breaks every login. Both carry a
|
||||
comment saying so.
|
||||
|
||||
- ponytail ceiling: one replica, no HSTS, no security headers beyond Caddy's defaults, no
|
||||
rate limiting, and HTTP-01 rather than DNS-01 (so a wildcard certificate is not
|
||||
available). Upgrade path in that order; DNS-01 first if the subdomain list ever grows.
|
||||
|
||||
## Coupling rules touched (CLAUDE.md §8)
|
||||
|
||||
None. §8.3 holds — the browser reaches a portal, the portal reverse-proxies its own BFF
|
||||
group, and the edge is in front of all of it. The edge terminates TLS and routes by
|
||||
hostname; it does not know what any service does.
|
||||
+6
-2
@@ -2,8 +2,10 @@
|
||||
|
||||
> **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's
|
||||
> hosted `ubuntu-latest` runner — no self-hosted runner required.
|
||||
> **`make ci` is still the local gate** — it runs the exact same checks
|
||||
> (the workflow calls the same `make` targets).
|
||||
> **`make ci` is still the local gate** — it runs the same checks via the same
|
||||
> `make` targets, with one exception: the `k8s` job's targets are not in `make ci`,
|
||||
> because `helm` is optional for everyone not deploying to Kubernetes. Run
|
||||
> `make k8s-lint k8s-drift` by hand after touching the chart or the compose file.
|
||||
|
||||
## The pipeline
|
||||
|
||||
@@ -16,6 +18,8 @@ and CI cannot drift:
|
||||
| `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK |
|
||||
| `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK |
|
||||
| `unit` | `make unit` → `dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK |
|
||||
| `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
|
||||
| `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
|
||||
| `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
|
||||
| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
|
||||
|
||||
|
||||
@@ -222,6 +222,9 @@ string, so the port the browser uses has to match the one baked into `config.jso
|
||||
This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
|
||||
`localhost:8180`); only the addresses differ.
|
||||
|
||||
All of this is what §10 removes: with a public domain the portals have real certificates,
|
||||
so the browser gets its secure context and no forwarding is involved.
|
||||
|
||||
### The admin UIs work straight off the NodePorts
|
||||
|
||||
These are server-rendered and need no secure context, so they are reachable at the VM's
|
||||
@@ -322,6 +325,7 @@ The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the da
|
||||
|
||||
```bash
|
||||
make k8s-lint # render + schema-check the chart, no cluster needed
|
||||
make k8s-drift # fail if compose and the chart describe different stacks
|
||||
make k8s-portals # forward the portals + Keycloak to localhost (browser access)
|
||||
make k8s-images K8S_REGISTRY=... # after changing a service or a portal
|
||||
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
|
||||
@@ -359,6 +363,68 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
|
||||
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
|
||||
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |
|
||||
|
||||
## 10. Publishing it on a public domain
|
||||
|
||||
By default the stack has no hostname: it is reached on NodePorts, and §5's secure-context
|
||||
problem forces `TALOS_HOST=localhost` plus five SSH forwards. Setting `public.domain` puts a
|
||||
Caddy deployment in front of it that terminates TLS for real hostnames (ADR-0035), and the
|
||||
forwards go away.
|
||||
|
||||
### Once, outside the cluster
|
||||
|
||||
**DNS** — five A records to the *host's* public address (the cluster is behind it):
|
||||
|
||||
```
|
||||
register.<domain> mijn.<domain> behandel.<domain> beheer.<domain> auth.<domain> → 46.224.220.37
|
||||
```
|
||||
|
||||
**The host's forward** — the public IP is on the Fedora host, so it has to hand 80/443 to
|
||||
the node. This is the only host-side configuration, and it is dumb layer 4:
|
||||
|
||||
```bash
|
||||
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32080
|
||||
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=443:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32443
|
||||
sudo firewall-cmd --permanent --zone=public --add-masquerade
|
||||
sudo firewall-cmd --reload
|
||||
```
|
||||
|
||||
`--add-masquerade` is what makes the return path work: without it the node answers the
|
||||
client's address directly and the reply never goes back through the host.
|
||||
|
||||
**A StorageClass.** Caddy's certificates live in `/data`, which is an `emptyDir` unless
|
||||
`persistence.storageClass` is set (§6). Let's Encrypt allows five duplicate certificates per
|
||||
week, so on an `emptyDir` a handful of pod restarts leaves the edge serving an untrusted
|
||||
certificate until the limit resets. Install local-path first (§6).
|
||||
|
||||
### Deploy
|
||||
|
||||
```bash
|
||||
make k8s-up TALOS_HOST=<domain-facing name> K8S_REGISTRY=<TALOS_VM_IP>:30500 \
|
||||
K8S_SET='--set public.domain=<domain> --set public.email=<ops address> --set persistence.storageClass=local-path'
|
||||
```
|
||||
|
||||
`public.domain` is the only switch: with it empty nothing in `templates/edge.yaml` renders
|
||||
and the stack behaves exactly as §4 describes. With it set, `KC_HOSTNAME` and the portals'
|
||||
`config.json` both become `https://auth.<domain>` — one helper builds both, so the issuer
|
||||
and the authority cannot drift (ADR-0010).
|
||||
|
||||
Watch the first certificate being issued:
|
||||
|
||||
```bash
|
||||
kubectl -n big logs deploy/caddy-edge -f # "certificate obtained successfully"
|
||||
curl -sSI https://register.<domain>/openbaar/register | head -1
|
||||
```
|
||||
|
||||
### When it doesn't work
|
||||
|
||||
| Symptom | Cause |
|
||||
|---|---|
|
||||
| ACME fails with `connection refused` or a timeout on the HTTP-01 challenge | the host's 80 → 32080 forward is missing, or `--add-masquerade` is |
|
||||
| ACME fails with `NXDOMAIN` / `no such host` | the A record isn't there yet. Caddy retries with backoff; fix DNS and it recovers |
|
||||
| An untrusted certificate after several restarts | the Let's Encrypt duplicate limit, from certificates on an `emptyDir` — see above |
|
||||
| The portal loads but login bounces back logged out | `public.domain` changed without the portals rolling. The chart hashes the issuer into their pod template, so `helm upgrade` should do it — check `kubectl -n big describe deploy/self-service` |
|
||||
| `404` from the edge on a name that should work | the name isn't in `public.routes`; Caddy answers 404 for a Host it has no site block for |
|
||||
|
||||
## What is not ported
|
||||
|
||||
- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
|
||||
@@ -366,5 +432,7 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
|
||||
`K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
|
||||
The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
|
||||
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
|
||||
chart. The Kubernetes path is verified with §5's smoke test.
|
||||
chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
|
||||
clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
|
||||
image bump that skipped the chart fails there, but nothing deploys the chart in CI.
|
||||
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.
|
||||
|
||||
@@ -135,6 +135,20 @@ cluster-internal hosts ({{ .Release.Namespace }}) and the node address
|
||||
{{- end }}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
The origin a browser reaches Keycloak on, and so the issuer its tokens carry and
|
||||
the authority the portals are configured with (ADR-0010). With a public edge that
|
||||
is the `auth` hostname on `public.domain` — which must stay in step with the `auth`
|
||||
key in `public.routes`; without one it is the node address plus Keycloak's NodePort.
|
||||
*/}}
|
||||
{{- define "big.keycloakUrl" -}}
|
||||
{{- if .Values.public.domain -}}
|
||||
https://auth.{{ .Values.public.domain }}
|
||||
{{- else -}}
|
||||
http://{{ .Values.host }}:{{ index .Values.nodePorts "keycloak" }}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "big.labels" -}}
|
||||
app.kubernetes.io/name: {{ .name }}
|
||||
app.kubernetes.io/instance: {{ .root.Release.Name }}
|
||||
|
||||
@@ -40,5 +40,5 @@ metadata:
|
||||
{{- include "big.labels" (dict "root" $ "name" (printf "portal-config-%s" $realm)) | nindent 4 }}
|
||||
data:
|
||||
config.json: |
|
||||
{ "authority": "{{ printf "http://%s:%v" $.Values.host (index $.Values.nodePorts "keycloak") }}/realms/{{ $realm }}" }
|
||||
{ "authority": "{{ include "big.keycloakUrl" $ }}/realms/{{ $realm }}" }
|
||||
{{- end }}
|
||||
|
||||
@@ -28,7 +28,7 @@ spec:
|
||||
{{- range $w.files }}
|
||||
{{- if hasPrefix "portal-config-" .configMap }}
|
||||
annotations:
|
||||
checksum/portal-config: {{ printf "%s|%v" $.Values.host (index $.Values.nodePorts "keycloak") | sha256sum }}
|
||||
checksum/portal-config: {{ include "big.keycloakUrl" $ | sha256sum }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
labels:
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
{{- /*
|
||||
The public TLS edge (ADR-0035). Rendered only when `public.domain` is set; with it
|
||||
empty the stack is reached on the NodePorts below and nothing here exists.
|
||||
|
||||
Caddy rather than an ingress controller: the four portals already run caddy:2-alpine,
|
||||
so this adds no dependency, and it does ACME itself — no cert-manager, no CRDs, no
|
||||
Ingress objects for five hostnames that never change. It proxies to the ClusterIP
|
||||
services, so the browser-facing NodePorts are not involved in a public deployment.
|
||||
|
||||
The public IP lives on the Fedora host, which forwards 80/443 to the two NodePorts
|
||||
below. That forward is dumb L4 — no TLS, no routing — see the runbook.
|
||||
*/}}
|
||||
{{- if .Values.public.domain }}
|
||||
{{- $pub := .Values.public }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: caddy-edge-config
|
||||
labels:
|
||||
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
|
||||
data:
|
||||
Caddyfile: |
|
||||
{
|
||||
{{- with $pub.email }}
|
||||
email {{ . }}
|
||||
{{- end }}
|
||||
}
|
||||
{{- range $sub, $target := $pub.routes }}
|
||||
|
||||
{{ $sub }}.{{ $pub.domain }} {
|
||||
reverse_proxy {{ $target }}
|
||||
}
|
||||
{{- end }}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: caddy-edge
|
||||
labels:
|
||||
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: caddy-edge
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
# A ConfigMap mounted with subPath never updates in place, so a changed
|
||||
# Caddyfile has to roll the pod.
|
||||
checksum/caddyfile: {{ printf "%s|%v|%v" $pub.domain $pub.email $pub.routes | sha256sum }}
|
||||
labels:
|
||||
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 8 }}
|
||||
spec:
|
||||
containers:
|
||||
- name: caddy-edge
|
||||
image: {{ $pub.image }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 80
|
||||
- name: https
|
||||
containerPort: 443
|
||||
# TCP, not HTTP: a GET with no matching Host gets a 404 from Caddy, which
|
||||
# would fail an httpGet probe for a perfectly healthy edge.
|
||||
readinessProbe:
|
||||
tcpSocket: { port: 443 }
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/caddy/Caddyfile
|
||||
subPath: Caddyfile
|
||||
readOnly: true
|
||||
- name: data
|
||||
mountPath: /data
|
||||
- name: run
|
||||
mountPath: /config
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: caddy-edge-config
|
||||
- name: run
|
||||
emptyDir: {}
|
||||
- name: data
|
||||
{{- if .Values.persistence.storageClass }}
|
||||
persistentVolumeClaim:
|
||||
claimName: caddy-edge-data
|
||||
{{- else }}
|
||||
# Certificates live here. On an emptyDir every pod restart asks Let's
|
||||
# Encrypt again, and its duplicate-certificate limit is five per week —
|
||||
# set persistence.storageClass for anything that stays up.
|
||||
emptyDir: {}
|
||||
{{- end }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: caddy-edge
|
||||
labels:
|
||||
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
|
||||
spec:
|
||||
type: NodePort
|
||||
selector:
|
||||
app.kubernetes.io/name: caddy-edge
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 80
|
||||
nodePort: {{ $pub.nodePorts.http }}
|
||||
- name: https
|
||||
port: 443
|
||||
targetPort: 443
|
||||
nodePort: {{ $pub.nodePorts.https }}
|
||||
{{- if .Values.persistence.storageClass }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: caddy-edge-data
|
||||
labels:
|
||||
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
|
||||
# Keep the certificates when the release is uninstalled — re-issuing them on
|
||||
# every reinstall is what burns the rate limit.
|
||||
annotations:
|
||||
helm.sh/resource-policy: keep
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: {{ .Values.persistence.storageClass }}
|
||||
resources:
|
||||
requests:
|
||||
storage: 128Mi
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -49,6 +49,32 @@ persistence:
|
||||
# the data across pod restarts.
|
||||
storageClass: ""
|
||||
|
||||
# The public TLS edge (ADR-0035). Empty `domain` = no edge at all: nothing in
|
||||
# templates/edge.yaml is rendered and the stack is reached on the NodePorts below,
|
||||
# with `host` above pinning the OIDC origin.
|
||||
#
|
||||
# Set it and an in-cluster Caddy terminates TLS for `<sub>.<domain>`, gets its own
|
||||
# certificates from Let's Encrypt and proxies to the ClusterIP services. The node
|
||||
# only has to be reachable on the two NodePorts here — the Fedora host forwards
|
||||
# 80/443 to them (see docs/runbooks/kubernetes-talos.md).
|
||||
public:
|
||||
domain: ""
|
||||
# ACME registration address; Let's Encrypt uses it for expiry warnings.
|
||||
email: ""
|
||||
image: docker.io/library/caddy:2-alpine
|
||||
# <subdomain>: <in-cluster service:port>. `auth` is not free-form — big.keycloakUrl
|
||||
# builds the pinned issuer from it.
|
||||
routes:
|
||||
register: openbaar:80
|
||||
mijn: self-service:80
|
||||
behandel: behandel:80
|
||||
beheer: beheer:80
|
||||
auth: keycloak:8080
|
||||
# Where the host's 80/443 forward lands. Not 30080/30443: 30080 is the BFF.
|
||||
nodePorts:
|
||||
http: 32080
|
||||
https: 32443
|
||||
|
||||
# The only place a port is published outside the cluster. A workload listed here
|
||||
# gets a NodePort on its single port; everything else stays ClusterIP.
|
||||
nodePorts:
|
||||
@@ -268,7 +294,7 @@ workloads:
|
||||
# Pin the issuer to the address the browser uses, and let backchannel calls
|
||||
# keep using keycloak:8080 — the BFF discovers metadata in-cluster and gets
|
||||
# this issuer back, which is what browser tokens carry (infra/host-browser.yml).
|
||||
KC_HOSTNAME: "http://{{ .Values.host }}:{{ index .Values.nodePorts \"keycloak\" }}"
|
||||
KC_HOSTNAME: '{{ include "big.keycloakUrl" . }}'
|
||||
KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
|
||||
ports: [{ name: http, port: 8080 }]
|
||||
# TCP, not /health/ready on the management port: nothing here gates on realm
|
||||
|
||||
Executable
+116
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail when the compose stack and the Helm chart stop describing the same stack.
|
||||
|
||||
`infra/docker-compose.yml` is CI-canonical; `infra/helm/big-reference` is a
|
||||
transcription of it (ADR-0033), and until now nothing kept the two in step — an
|
||||
upstream image bump or a new service applied to only one of them landed
|
||||
unnoticed. This compares what each side actually *deploys*, not the two files:
|
||||
the rendered chart against `docker compose config`. Both tools are already
|
||||
prerequisites of the `k8s-*` make targets.
|
||||
|
||||
Run it with `make k8s-drift`. No cluster needed.
|
||||
|
||||
ponytail: names and images only, as sets — no per-workload env/ports/volumes.
|
||||
Those differ by design in four documented places (ADR-0033), so comparing them
|
||||
would mean re-encoding every deviation field by field; a tag bump and a missing
|
||||
service are the drift that actually bites.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
COMPOSE = ROOT / "infra/docker-compose.yml"
|
||||
CHART = ROOT / "infra/helm/big-reference"
|
||||
|
||||
# The busybox init container that every `waitFor` workload gets exists only in
|
||||
# the chart (compose has `depends_on`). Rendering it under a sentinel makes it
|
||||
# filterable without teaching the check what busybox is.
|
||||
BUSYBOX = "drift-check-ignored-init-image"
|
||||
|
||||
# Differences that Kubernetes forces, not drift (ADR-0033). A name listed here is
|
||||
# expected to be on exactly one side; anything else fails.
|
||||
DEVIATIONS = {
|
||||
# The four Django services apply their own setup_configuration in the web pod
|
||||
# (`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]`) rather than in a
|
||||
# separate init Job. Both that script and /start.sh run `manage.py migrate`, and
|
||||
# Kubernetes has no `depends_on: service_completed_successfully` to serialise them,
|
||||
# so the Job and its web pod migrated the same database concurrently.
|
||||
"oz-init": "folded into the openzaak pod",
|
||||
"nrc-init": "folded into the nrc-web pod",
|
||||
"objecttypen-init": "folded into the objecttypen pod",
|
||||
"objecten-init": "folded into the objecten pod",
|
||||
# Compose seeds these from the host — the verify scripts `docker cp` the two
|
||||
# scripts into a running container, and docker-compose.local.yml carries
|
||||
# `local-seed` + `nrc-subscribe` for `make local`. A cluster has no host to seed
|
||||
# from, so both became Jobs in the chart.
|
||||
"seed-zaaktype": "compose seeds the catalogus from the host (infra/openzaak/seed_catalogus.py)",
|
||||
"nrc-subscribe": "compose registers the abonnement from the host (infra/local/register-abonnement.py)",
|
||||
}
|
||||
|
||||
# Workloads the observability backplane adds. Off by default in both stacks'
|
||||
# defaults, so they are rendered on purpose here — otherwise their images drift
|
||||
# unwatched.
|
||||
OBSERVABILITY = ["tempo", "prometheus", "grafana"]
|
||||
|
||||
|
||||
def compose_services() -> dict[str, str]:
|
||||
"""Service name -> image, with ${TAG:-default} interpolation already applied."""
|
||||
out = run(["docker", "compose", "-f", str(COMPOSE), "config", "--format", "json"])
|
||||
return {name: svc.get("image", "") for name, svc in json.loads(out)["services"].items()}
|
||||
|
||||
|
||||
def chart_workloads() -> dict[str, str]:
|
||||
"""Workload name -> image, read back out of the rendered manifests."""
|
||||
out = run(
|
||||
["helm", "template", "big", str(CHART), "-n", "big", "--set", f"images.busybox={BUSYBOX}"]
|
||||
+ [f"--set=workloads.{w}.enabled=true" for w in OBSERVABILITY]
|
||||
)
|
||||
workloads = {}
|
||||
for doc in out.split("\n---"):
|
||||
if not re.search(r"^kind: (Deployment|Job)$", doc, re.M):
|
||||
continue
|
||||
name = re.search(r"^ name: (\S+)$", doc, re.M)[1]
|
||||
images = [i for i in re.findall(r"^\s+image: (\S+)$", doc, re.M) if i != BUSYBOX]
|
||||
workloads[name] = images[0]
|
||||
return workloads
|
||||
|
||||
|
||||
def run(argv: list[str]) -> str:
|
||||
proc = subprocess.run(argv, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
sys.exit(f"{argv[0]} failed:\n{proc.stderr}")
|
||||
return proc.stdout
|
||||
|
||||
|
||||
def main() -> int:
|
||||
compose, chart = compose_services(), chart_workloads()
|
||||
problems = []
|
||||
|
||||
for name in sorted(set(compose) - set(chart) - set(DEVIATIONS)):
|
||||
problems.append(f" {name}: in docker-compose.yml, not in the chart")
|
||||
for name in sorted(set(chart) - set(compose) - set(DEVIATIONS)):
|
||||
problems.append(f" {name}: in the chart, not in docker-compose.yml")
|
||||
for name in sorted(set(compose) & set(chart)):
|
||||
if compose[name] != chart[name]:
|
||||
problems.append(f" {name}: compose runs {compose[name]}, the chart runs {chart[name]}")
|
||||
|
||||
if problems:
|
||||
print("compose and the Helm chart describe different stacks:\n" + "\n".join(problems))
|
||||
print(
|
||||
"\nPort the change to the other stack, or — if the difference is forced by\n"
|
||||
"Kubernetes — declare it in DEVIATIONS in this file, with the reason."
|
||||
)
|
||||
return 1
|
||||
|
||||
print(f"no drift: {len(chart)} workloads, images identical on both stacks")
|
||||
for name, why in sorted(DEVIATIONS.items()):
|
||||
print(f" deviation (declared): {name} — {why}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail when the pinned issuer and the portals' OIDC authority stop agreeing.
|
||||
|
||||
Keycloak pins one issuer (`KC_HOSTNAME`) and each portal is configured with one
|
||||
authority (`config.json`). A browser token carries the first; the BFF validates
|
||||
against what it discovers from the second (ADR-0010). When the two drift the
|
||||
symptom is three services away — a login that bounces back logged out, or a 401
|
||||
from the BFF — so the chart builds both from one helper and this asserts it.
|
||||
|
||||
It also pins the two halves of the public edge (ADR-0035): that setting
|
||||
`public.domain` actually publishes the hostnames, and that leaving it empty
|
||||
renders no edge at all, which is what compose, CI and a laptop cluster rely on.
|
||||
|
||||
Run it with `make k8s-lint`. No cluster needed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
CHART = Path(__file__).resolve().parent / "big-reference"
|
||||
DOMAIN = "example.test"
|
||||
|
||||
|
||||
def render(*sets: str) -> str:
|
||||
argv = ["helm", "template", "big", str(CHART), "-n", "big"]
|
||||
for s in sets:
|
||||
argv += ["--set", s]
|
||||
proc = subprocess.run(argv, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
sys.exit(f"helm template failed:\n{proc.stderr}")
|
||||
return proc.stdout
|
||||
|
||||
|
||||
def issuer(out: str) -> str:
|
||||
"""The value of KC_HOSTNAME in the rendered manifests."""
|
||||
m = re.search(r"name: KC_HOSTNAME\n\s+value: \"(\S+)\"", out)
|
||||
return m[1] if m else ""
|
||||
|
||||
|
||||
def authorities(out: str) -> set[str]:
|
||||
"""Every portal's OIDC authority, with the realm path stripped."""
|
||||
found = set()
|
||||
for line in re.findall(r'\{ "authority": .* \}', out):
|
||||
url = json.loads(line)["authority"]
|
||||
found.add(url.rsplit("/realms/", 1)[0])
|
||||
return found
|
||||
|
||||
|
||||
def main() -> int:
|
||||
problems = []
|
||||
|
||||
public = render(f"public.domain={DOMAIN}")
|
||||
if issuer(public) != f"https://auth.{DOMAIN}":
|
||||
problems.append(f" with public.domain set, KC_HOSTNAME is {issuer(public)!r}, not https://auth.{DOMAIN}")
|
||||
if authorities(public) != {f"https://auth.{DOMAIN}"}:
|
||||
problems.append(f" with public.domain set, the portals point at {sorted(authorities(public))}")
|
||||
for host in (f"register.{DOMAIN}", f"mijn.{DOMAIN}", f"behandel.{DOMAIN}", f"beheer.{DOMAIN}", f"auth.{DOMAIN}"):
|
||||
if host not in public:
|
||||
problems.append(f" {host} is not published by the edge")
|
||||
|
||||
private = render()
|
||||
if authorities(private) != {issuer(private)}:
|
||||
problems.append(f" by default the portals point at {sorted(authorities(private))}, the issuer is {issuer(private)!r}")
|
||||
if "caddy-edge" in private:
|
||||
problems.append(" the edge renders with no public.domain — compose, CI and a laptop cluster expect nothing")
|
||||
|
||||
if problems:
|
||||
print("the chart's OIDC origin is inconsistent:\n" + "\n".join(problems))
|
||||
print("\nBoth halves come from the `big.keycloakUrl` helper — change it, not one caller.")
|
||||
return 1
|
||||
|
||||
print(f"issuer + portal authority agree, with and without a public domain")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -56,6 +56,7 @@ nav:
|
||||
- "ADR-0032: Werkbak live refresh": architecture/adr-0032-werkbak-live-refresh.md
|
||||
- "ADR-0033: Kubernetes via one Helm chart": architecture/adr-0033-kubernetes-via-one-helm-chart.md
|
||||
- "ADR-0034: Caddy serves the portals": architecture/adr-0034-caddy-serves-the-portals.md
|
||||
- "ADR-0035: Public TLS edge in the cluster": architecture/adr-0035-public-tls-edge-in-cluster.md
|
||||
- FDS-architectuur:
|
||||
- Overzicht: architecture/fds/README.md
|
||||
- Componentview (L3): architecture/fds/c4-component-view.md
|
||||
|
||||
Reference in New Issue
Block a user