Compare commits

..
Author SHA1 Message Date
notandClaude Opus 5 dfa1a370ac ci(deploy): publish the stack over TLS when PUBLIC_DOMAIN is set (refs #175)
The in-cluster edge (#178) is off unless the chart is given a domain, so pass
one through from a repository variable. Unset, the deploy is exactly what it was
— NodePorts, and the portals reachable only over the SSH forwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:31:47 +02:00
notandClaude Opus 5 e8cb1ec7e9 docs(k8s): how to reach the deployed portals from a laptop (refs #175)
The five forwards are not optional and not independent: the portals' OIDC
authority is pinned to localhost:30180, so forwarding the portal without
Keycloak gets ERR_CONNECTION_REFUSED on the discovery document and an opaque
"[object Object]" in the console. One ssh replaces `make k8s-portals` for the
lab server, and needs no kubeconfig.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:19:02 +02:00
notandClaude Opus 5 de6db7d35b docs(k8s): split the taint fix from the registry patch (refs #175)
Talos 1.14 rejects a `patch mc` that sets `cluster.allowSchedulingOnControlPlanes`
— the field left the v1alpha1 schema, the way `machine.install` did — and the
rejection discards the rest of the patch with it, so the registry mirror never
lands and the failure moves from Pending pods to ImagePullBackOff without ever
saying so. Document the two as separate steps, and correct the claim that the
taint has to be patched away: `kubectl taint` is what §1 prescribes and it holds
until the node re-registers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:06:24 +02:00
notandClaude Opus 5 d17e79959b ci(deploy): say why a deploy stalled, and document the cluster prerequisite (refs #175)
The first run against the lab server's VM timed out on `kubectl rollout status`
for the in-cluster registry with nothing but "timed out waiting for the
condition". The cause was three commands up the runbook: that VM was installed
from a stock Talos config, so the only node still carries the control-plane
taint and no pod can schedule — and the missing registry mirror would have
failed the image pulls right after.

`rollout status` can only ever report the symptom, so dump the whole cluster's
pods and the recent events on failure instead of `big`'s pods alone; the
scheduler's "untolerated taint" message is the answer and it lives in the
events. Runbook §9 now opens with the machine-config patch the deploy assumes,
as one applied-live patch rather than a `kubectl taint` that the controller
undoes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:01:47 +02:00
notandClaude Opus 5 fc036d53d5 ci(deploy): deploy the stack to Talos on merge to main (refs #175)
CI / k8s (pull_request) Successful in 5s
CI / lint (pull_request) Successful in 1m25s
CI / build (pull_request) Successful in 1m22s
CI / unit (pull_request) Successful in 1m24s
CI / frontend (pull_request) Successful in 1m45s
CI / mutation (pull_request) Successful in 3m1s
CI / verify-stack (pull_request) Successful in 6m22s
The chart has been deployable by hand since #25 and linted in CI since #168;
this makes a merged PR actually ship it to the lab server's Talos VM.

Neither the Kubernetes API nor the in-cluster registry is publicly reachable,
so the job forwards 6443, 30500 and 30141 over the same SSH hop into the Fedora
host that the Gitea-runner pipeline uses. That splits the registry into two
names for one store: images are pushed through the tunnel to localhost:30500,
and the node pulls them from its own NodePort — the address its registry-mirror
patch trusts over plain HTTP.

It deploys with `make k8s-reseed` rather than `make k8s-up`: the bootstrap Jobs
are idempotent, and deleting them first is what stops a changed Job template
from wedging `helm upgrade`. The nine deployments are then rolled explicitly,
because `dev` is a mutable tag and helm sees an unchanged pod template.

PR CI is the merge gate, so this workflow does not re-run the checks. Deploys
queue instead of cancelling: a `helm upgrade` killed half-way leaves the release
in `pending-upgrade` and needs unwedging by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:40:42 +02:00
not 9d7e8e5b65 ci(k8s): gate the Helm chart in CI + a compose↔chart drift check (closes #168) (#171)
CI / k8s (push) Successful in 5s
CI / lint (push) Successful in 1m27s
CI / build (push) Successful in 1m22s
CI / unit (push) Successful in 1m12s
CI / frontend (push) Successful in 2m7s
CI / mutation (push) Successful in 3m9s
CI / verify-stack (push) Successful in 6m20s
## What & why

The Helm chart landed in #167 with two gaps written into ADR-0033: `make k8s-lint` existed
but no CI job ran it, and *"a second deployment description to keep in step with compose —
nothing enforces that today; a drift check belongs in CI (follow-up)"*. Both are closed here.

**`make k8s-drift`** (`infra/helm/check-drift.py`, stdlib only) compares what each stack
actually deploys rather than diffing two files that differ by design: workload names and
resolved container images, taken from `docker compose config --format json` and a rendered
chart. The six differences that exist today are declared in `DEVIATIONS` with the reason
each was forced — the four `*-init` Django services folded into their web pods, and the two
bootstrap Jobs compose runs from the host — so only a *new* difference fails.

**A `k8s` CI job** runs `k8s-lint` then `k8s-drift` on every push and PR. No cluster, no
marketplace action: helm is fetched as the pinned static binary the Talos runbook already
gives developers.

Closes #168

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — the red commit reports all six
      real differences; the green commit declares them.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issue (`refs #168`).
- [x] CI green — awaiting the run on this PR (`make k8s-lint` and `make k8s-drift` pass locally).
- [x] `docker compose up` unaffected — no service, image or compose file is touched.
- [x] Docs updated — `docs/runbooks/ci.md` (job table + the one place local and CI now
      differ), `docs/runbooks/kubernetes-talos.md` §7/§"not ported", and ADR-0033's cost note.
- [x] No ADR needed: no new dependency (python stdlib, and helm/docker were already
      prerequisites of the `k8s-*` targets), no boundary moved, no §8 rule bent.
- [x] Not user-visible, so no demo note.

## Notes for reviewers

Verified by hand that both drift classes fail the check, not just that it passes today:

- bumping `OPENZAAK_TAG` in compose alone → reports `openzaak` and `oz-celery` with both
  image strings;
- adding a workload to `values.yaml` alone → reports it by name.

Deliberate limits (there is a `ponytail:` note in the script):

- **Names and images only**, as sets — no per-workload env, ports or volumes. Those differ
  by design in four documented places, so comparing them would mean re-encoding every
  deviation field by field for very little more signal.
- **The three observability workloads are rendered with `enabled=true`** by the check, even
  though both stacks default them off, so their images can't drift unwatched.
- **`k8s-lint`/`k8s-drift` are not in `make ci`**, to avoid making `helm` a hard
  prerequisite for everyone. That is now the only local/CI difference; it's called out in
  `docs/runbooks/ci.md`.

Follow-ups filed while reviewing the chart, not addressed here: #169 (the published docs
omit every ADR after 0010 and all runbooks but `ci.md`) and #170 (the production-posture
ADR #25 asked for — secrets are still plain text in `values.yaml`).Reviewed-on: #171
2026-09-18 13:25:24 +00:00
not 17f1f2f809 docs(nav): publish every ADR and runbook, gated by a nav check (closes #169) (#172)
CI / lint (push) Successful in 1m46s
CI / build (push) Successful in 1m29s
CI / unit (push) Successful in 1m5s
CI / frontend (push) Successful in 1m41s
CI / mutation (push) Successful in 3m30s
CI / verify-stack (push) Successful in 5m50s
## What & why

`docs/` is the source of truth (CLAUDE.md §12), but only pages listed in `mkdocs.yml`'s nav
are published — and mkdocs' own `validation.nav.omitted_files: warn` keeps the build green
while dropping the rest. So the site had quietly stopped at **ADR-0010** and
**`runbooks/ci.md`**: 31 pages, including every ADR from 0011 to 0034, six of the seven
runbooks, and `synthetic-data.md`, existed in the repo and nowhere else.

- `infra/check-docs-nav.py` fails when a page under `docs/` is not in the nav. It runs in
  `make lint`, so the existing CI job gates it — python3 only, no new tooling, and no
  mkdocs install needed to check it.
- The nav now lists all 34 ADRs, all 7 runbooks and `synthetic-data.md`.
- ADR-0033's `Slice:` header said "none yet"; #25 closed it.
- The landing page gained a pointer to the Talos runbook.

Closes #169

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the fix — the red commit lists all 31 missing pages.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issue (`refs #169`).
- [ ] CI green — awaiting the run on this PR (`python3 infra/check-docs-nav.py` passes locally;
      `make lint` also needs the .NET SDK, which CI has).
- [x] `docker compose up` unaffected — docs and `mkdocs.yml` only, plus one `make lint` line.
- [x] Docs updated — that is the change.
- [x] No ADR needed: no dependency, no boundary, no §8 rule touched.
- [x] Not user-visible, so no demo note.

## Notes for reviewers

- The check is a **substring test**, not a YAML parse (there's a `ponytail:` note in the
  script): a page's path either appears in `mkdocs.yml` or it doesn't. That keeps it
  dependency-free — `mkdocs.yml` can't be read by `yaml.safe_load` anyway, it carries a
  `!!python/name:` tag for the mermaid fence. It does not check that an entry *points at a
  file that exists*; mkdocs' `not_found: warn` covers that direction.
- ADR labels in the nav are shortened by hand (`"ADR-0013: Behandel-portal wiring"`), since
  several H1s are a full sentence.

**Known gap, not fixed here:** CLAUDE.md §12 says the site is "published via a Gitea Actions
workflow to Gitea Pages", and no such workflow exists — `mkdocs build` is never run, by CI or
by any make target. Gitea has no built-in Pages, so publishing needs a decision (a
`gitea-pages` server, an artifact, or a static host) rather than a patch. Worth its own issue
if the published site is actually wanted; until then this PR makes the nav correct for whoever
runs `mkdocs serve`.Reviewed-on: #172
2026-09-18 12:55:20 +00:00
not 1dd8bd4e1b S-24/#25 · Helm chart + Kubernetes deployment, and Caddy for the portals (#166) (#167)
CI / lint (push) Successful in 1m17s
CI / build (push) Successful in 1m12s
CI / unit (push) Successful in 1m26s
CI / frontend (push) Successful in 2m58s
CI / mutation (push) Successful in 9m1s
CI / verify-stack (push) Successful in 8m53s
## What & why

Two changes, made and verified together on a real cluster.

**S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`,
whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered
by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a
service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without
a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see
below), which is more than it asked for — the part it asked for that is **not** here is the
production-posture write-up (HA, secrets, backup), see Known gaps.

**#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass`
upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search
domains. That had cost two workarounds in one script: rewriting the resolver address for
rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on
Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy
bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's
`BFF_HOST` env are deleted.

Closes #25
Closes #166

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — twice: the Caddyfile contract test
      before the Caddyfiles, `make k8s-lint` before the chart.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`).
- [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new
      unit self-check pass locally; the compose e2e and mutation lanes are CI's).
- [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes —
      the portal images were rebuilt and verified standalone, but a full `make up` run has
      not been done on this branch. Please confirm in review or let CI's smoke test speak.
- [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`,
      `demo-script.md`, and the docs that named nginx.
- [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy).
- [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a
      user-visible slice, and the Caddy swap is invisible to the demo script beyond the
      wording fix included here.

## How it was verified

Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three
minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB.

- Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak +
  Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference
  matching the submitted registration).
- Werkbak read with an MFA'd medewerker token → 200.
- The browser flow driven with Playwright against `http://localhost:30140`: secure context,
  `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**.
- Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies
  its own groups, and a portal does *not* proxy a neighbour's group.

## Notes for reviewers

Three bugs this shook out, each fixed at the cause rather than the symptom:

1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes'
   replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that
   relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec
   `start-dev`. The chart now `fail`s at render time on `command`.
2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run
   `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such
   edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already
   exists`). The four Django services now do both steps in order in the web pod — which
   also deletes four workloads.
3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now
   also restarts `event-subscriber` and `projection-api`, which create the projection
   schema on start and otherwise keep writing to a schema-less database.

Known gaps / follow-ups:

- **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the
  ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine
  for a laptop demo, and exactly what #25's "production posture" ADR should address — I
  suggest a follow-up issue rather than stretching this PR.
- **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into
  `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay
  in step. Worth a small follow-up.
- **This is two slices in one PR.** They were built and verified together and the diff is
  entangled (the chart was written against Caddy from the start), so splitting now would
  mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather.
- **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear.
  One conflict, in the `unit:` target where both branches add a self-check line — resolved
  by keeping both. #161's `infra/host-browser.yml` arrived with
  `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the
  `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a
  path the images no longer have.Reviewed-on: #167
2026-09-10 08:53:58 +00:00
10 changed files with 447 additions and 8 deletions
+21
View File
@@ -41,6 +41,27 @@ jobs:
nuget-${{ runner.os }}- nuget-${{ runner.os }}-
- run: make lint - run: make lint
# The Helm chart's only automated gate: it renders and schema-checks the whole
# stack, and checks it still describes the same stack as the compose file
# (ADR-0033). No cluster involved — see docs/runbooks/kubernetes-talos.md.
k8s:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
# helm as its pinned static binary rather than a marketplace action: one URL,
# the same one the Talos runbook §0 gives a developer, and no third-party
# action to vet (CLAUDE.md §13). The drift check also needs `docker compose`,
# which the runner already has (see docs/runbooks/ci.md).
- name: Install helm
run: |
mkdir -p "$HOME/.local/bin"
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz \
| tar xz -O linux-amd64/helm > "$HOME/.local/bin/helm"
chmod +x "$HOME/.local/bin/helm"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- run: make k8s-lint
- run: make k8s-drift
build: build:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
+126
View File
@@ -0,0 +1,126 @@
name: Deploy to Talos
# A merge to main ships the stack to the Talos cluster on the lab server
# (docs/runbooks/kubernetes-talos.md §9). PR CI is the merge gate, so main is
# green by construction — this workflow only deploys.
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
# Queue deploys, never cancel one: a helm upgrade killed half-way leaves the
# release in `pending-upgrade` and the next run has to be unwedged by hand.
concurrency:
group: deploy-talos
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-latest
env:
# The Talos VM as seen from the Fedora host (libvirt guest IP), and the
# address a browser uses to reach the cluster. `localhost` is deliberate:
# the portals' PKCE needs a secure context, so they are reached over
# `kubectl port-forward` — runbook §5. Override with repo variables.
TALOS_VM_IP: ${{ vars.TALOS_VM_IP }}
TALOS_HOST: ${{ vars.TALOS_HOST }}
# Set it and the stack is published over TLS on <sub>.<domain> by the
# in-cluster edge (ADR-0035, runbook §10). Empty = NodePorts, as before.
PUBLIC_DOMAIN: ${{ vars.PUBLIC_DOMAIN }}
PUBLIC_EMAIL: ${{ vars.PUBLIC_EMAIL }}
steps:
- uses: https://github.com/actions/checkout@v4
# Pinned static binaries, the same URLs the Talos runbook §0 gives a
# developer and the same helm the `k8s` CI job uses — no action to vet.
- name: Install kubectl, helm and crane
run: |
set -euo pipefail
bin="$HOME/.local/bin"; mkdir -p "$bin"
curl -sSLo "$bin/kubectl" https://dl.k8s.io/release/v1.37.0/bin/linux/amd64/kubectl
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz | tar xz -O linux-amd64/helm > "$bin/helm"
curl -sSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz | tar xz -O crane > "$bin/crane"
chmod +x "$bin"/{kubectl,helm,crane}
echo "$bin" >> "$GITHUB_PATH"
# The cluster's API and its registry are only reachable through the Fedora
# host, so forward both to the runner. 30141 is the openbaar portal, for
# the smoke at the end.
- name: Tunnel the Talos API + registry through the Fedora host
env:
SSH_KEY: ${{ secrets.TALOS_SSH_KEY }}
run: |
set -euo pipefail
: "${TALOS_VM_IP:=192.168.122.173}"
umask 077
printf '%s\n' "$SSH_KEY" > ~/.ssh_talos
ssh -i ~/.ssh_talos -o StrictHostKeyChecking=no -o IdentitiesOnly=yes \
-o ExitOnForwardFailure=yes -p 6667 -f -N \
-L 6443:$TALOS_VM_IP:6443 \
-L 30500:$TALOS_VM_IP:30500 \
-L 30141:$TALOS_VM_IP:30141 \
user@labs.respellion.tech
# The kubeconfig's server must be https://127.0.0.1:6443 — Talos puts
# 127.0.0.1 in the apiserver cert SANs, so TLS verification still holds
# through the tunnel.
- name: Write the kubeconfig
env:
KUBECONFIG_B64: ${{ secrets.TALOS_KUBECONFIG }}
run: |
set -euo pipefail
umask 077
base64 -d <<< "$KUBECONFIG_B64" > "$RUNNER_TEMP/kubeconfig"
echo "KUBECONFIG=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_ENV"
kubectl --kubeconfig "$RUNNER_TEMP/kubeconfig" get nodes
# Idempotent; also makes a first deploy onto a bare cluster work. The
# registry's storage is an emptyDir, so a replaced pod loses the images —
# which the push in the next step puts back anyway.
- name: Ensure the in-cluster registry
run: make k8s-registry
# Push through the tunnel (localhost), pull from the node's own NodePort
# (the address in the Talos registry-mirror patch) — same registry, two
# names, so the two `make` calls get different K8S_REGISTRY values.
- name: Build and push the images
run: make k8s-images K8S_REGISTRY=localhost:30500
# k8s-reseed = seed configmaps + helm upgrade + re-run the bootstrap jobs.
# The jobs are idempotent, and deleting them first is what keeps a changed
# Job template from wedging the upgrade (`cannot patch … with kind Job`).
- name: Deploy the chart
run: |
set -euo pipefail
publish="${PUBLIC_DOMAIN:+--set public.domain=$PUBLIC_DOMAIN --set public.email=${PUBLIC_EMAIL:-}}"
make k8s-reseed \
TALOS_HOST=${TALOS_HOST:-localhost} \
K8S_REGISTRY=${TALOS_VM_IP:-192.168.122.173}:30500 \
K8S_SET="$publish"
# `dev` is a mutable tag and helm sees an unchanged pod template, so the
# new images only land on a restart (pullPolicy is already Always).
- name: Roll the services onto the new images
run: |
set -euo pipefail
svcs="acl domain bff event-subscriber projection-api self-service openbaar behandel beheer"
kubectl -n big rollout restart deploy $svcs
kubectl -n big rollout status --timeout=300s deploy $svcs
# Proves portal → Caddy → BFF → projection end to end. An empty register is
# a pass; a 502 or a timeout is not.
- name: Smoke the public register
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
# Cluster-wide, not just `big`: the first thing that can fail is the registry
# in its own namespace, and a scheduling problem shows up in the events, not
# in `rollout status` — which only ever says "timed out waiting".
- name: Pods and events on failure
if: failure()
run: |
kubectl get pods -A -o wide || true
kubectl -n big get jobs || true
kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true
+11 -1
View File
@@ -43,7 +43,7 @@ export DOCKER_HOST := unix://$(PODMAN_SOCK)
endif endif
endif endif
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help .PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-drift k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions) ## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions)
## `verify` is the live-stack stage (full stack up once → ACL + notification checks). ## `verify` is the live-stack stage (full stack up once → ACL + notification checks).
@@ -64,6 +64,9 @@ frontend:
## lint: verify formatting (no changes) ## lint: verify formatting (no changes)
lint: lint:
dotnet format $(SLN) --verify-no-changes dotnet format $(SLN) --verify-no-changes
# Only pages in mkdocs.yml's nav are published, and mkdocs keeps a build green
# when one is missing — so the nav is checked here rather than not at all.
python3 infra/check-docs-nav.py
## build: release build ## build: release build
build: build:
@@ -350,6 +353,13 @@ k8s-lint:
helm lint $(K8S_CHART) helm lint $(K8S_CHART)
helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null
## k8s-drift: fail if compose and the Helm chart describe different stacks
# Compose is CI-canonical (ADR-0033) and the chart is a transcription of it; this
# compares what each one deploys — workload names and resolved images. Needs
# `docker compose` and `helm`, no cluster.
k8s-drift:
python3 infra/helm/check-drift.py
## k8s-registry: deploy the in-cluster image registry (NodePort 30500) ## k8s-registry: deploy the in-cluster image registry (NodePort 30500)
k8s-registry: k8s-registry:
kubectl apply -f infra/helm/registry.yaml kubectl apply -f infra/helm/registry.yaml
@@ -3,8 +3,8 @@
- **Status:** Accepted - **Status:** Accepted
- **Date:** 2026-09-04 - **Date:** 2026-09-04
- **Deciders:** Respellion engineering - **Deciders:** Respellion engineering
- **Slice:** _(none yet — raised directly as a deployment-target request; see - **Slice:** #25 (S-24) — raised directly as a deployment-target request and matched to
"Process note" at the end)_ that issue afterwards; see the "Process note" at the end
## Context ## Context
@@ -126,8 +126,9 @@ Consequences of that shape, each chosen deliberately:
**Negative / costs** **Negative / costs**
- A second deployment description to keep in step with compose. Nothing enforces that - A second deployment description to keep in step with compose. `make k8s-drift` (#168)
today; a drift check belongs in CI (follow-up). now enforces the part that bites — the workload set and the resolved images, with the
four deviations below declared — but not per-workload env, ports or volumes.
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing - `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
one surfaces as `ContainerCreating`, not as a clear error. one surfaces as `ContainerCreating`, not as a clear error.
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches - Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
+2
View File
@@ -14,6 +14,8 @@ should teach.
In Dutch; the strategic framing lives in `Respellion/innovation-lab`. In Dutch; the strategic framing lives in `Respellion/innovation-lab`.
- **[Working in Gitea](gitea-workflow.md)** — issues, milestones, branches, PRs. - **[Working in Gitea](gitea-workflow.md)** — issues, milestones, branches, PRs.
- **[CI runbook](runbooks/ci.md)** — the pipeline and the `make ci` local gate. - **[CI runbook](runbooks/ci.md)** — the pipeline and the `make ci` local gate.
- **[Kubernetes on Talos](runbooks/kubernetes-talos.md)** — the second deployment target:
one Helm chart, a single-node cluster, and the parts that bite (ADR-0033).
## Quickstart ## Quickstart
+10 -2
View File
@@ -2,8 +2,10 @@
> **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's > **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's
> hosted `ubuntu-latest` runner — no self-hosted runner required. > hosted `ubuntu-latest` runner — no self-hosted runner required.
> **`make ci` is still the local gate** — it runs the exact same checks > **`make ci` is still the local gate** — it runs the same checks via the same
> (the workflow calls the same `make` targets). > `make` targets, with one exception: the `k8s` job's targets are not in `make ci`,
> because `helm` is optional for everyone not deploying to Kubernetes. Run
> `make k8s-lint k8s-drift` by hand after touching the chart or the compose file.
## The pipeline ## The pipeline
@@ -16,6 +18,8 @@ and CI cannot drift:
| `lint` | `make lint``dotnet format … --verify-no-changes` | .NET 10 SDK | | `lint` | `make lint``dotnet format … --verify-no-changes` | .NET 10 SDK |
| `build` | `make build``dotnet build … -c Release` | .NET 10 SDK | | `build` | `make build``dotnet build … -c Release` | .NET 10 SDK |
| `unit` | `make unit``dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK | | `unit` | `make unit``dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK |
| `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
| `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
| `mutation` | `make mutation``dotnet tool restore``dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK | | `mutation` | `make mutation``dotnet tool restore``dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) | | `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
@@ -27,6 +31,10 @@ and CI cannot drift:
> services by **container IP** (the runner can't reach published ports — see > services by **container IP** (the runner can't reach published ports — see
> [gitea-actions-gotchas.md §5/§6](gitea-actions-gotchas.md)). > [gitea-actions-gotchas.md §5/§6](gitea-actions-gotchas.md)).
A second workflow, `.gitea/workflows/deploy.yaml`, deploys the stack to the Talos
cluster on the lab server when a PR is merged to `main` — see
[kubernetes-talos.md §9](kubernetes-talos.md) for its secrets and the SSH tunnel it needs.
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`, All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
`https://github.com/actions/setup-dotnet@v4`) per CLAUDE.md §8.7 and §15 — Gitea `https://github.com/actions/setup-dotnet@v4`) per CLAUDE.md §8.7 and §15 — Gitea
Actions resolves them from GitHub. Actions resolves them from GitHub.
+95 -1
View File
@@ -322,6 +322,7 @@ The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the da
```bash ```bash
make k8s-lint # render + schema-check the chart, no cluster needed make k8s-lint # render + schema-check the chart, no cluster needed
make k8s-drift # fail if compose and the chart describe different stacks
make k8s-portals # forward the portals + Keycloak to localhost (browser access) make k8s-portals # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=... # after changing a service or a portal make k8s-images K8S_REGISTRY=... # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=... make k8s-up TALOS_HOST=... K8S_REGISTRY=...
@@ -359,6 +360,97 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) | | Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` | | A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |
## 9. Deploying on merge to main
`.gitea/workflows/deploy.yaml` runs the §3–§4 steps against the **lab server's** Talos VM
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
merge gate, so the workflow deploys without re-running the checks.
**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
gives you a node that still carries the control-plane taint and knows nothing about the
plain-HTTP registry, and the deploy hits those in that order: the `registry` pod sits
`Pending` until `rollout status` times out, and once that is fixed every repo image fails to
pull. Two separate fixes:
```bash
# on the Fedora host — 1. let workloads onto the only node (§1)
export KUBECONFIG=~/talos-kubeconfig-local
kubectl taint node --all node-role.kubernetes.io/control-plane-
# 2. trust the in-cluster registry over plain HTTP (§2)
cat > /tmp/registry-patch.yaml <<'YAML'
machine:
registries:
mirrors:
"<TALOS_VM_IP>:30500":
endpoints:
- http://<TALOS_VM_IP>:30500
YAML
talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/registry-patch.yaml
```
Keep those two apart. On Talos 1.14 a patch that also sets
`cluster.allowSchedulingOnControlPlanes` is rejected with *".cluster.allowSchedulingOnControlPlanes
is already set in v1alpha1 config"* — the field moved out of the v1alpha1 schema, the same way
`machine.install` did (§1) — and the rejection takes the whole patch with it, so the mirror
silently doesn't land either. `kubectl taint` is the documented way (§1); it is undone if the
node ever re-registers, which is a reboot, not a deploy.
The cluster's API and registry are not exposed publicly, so the job forwards them over the
same SSH hop the Gitea-runner pipeline uses:
```
ssh -p 6667 user@labs.respellion.tech -L 6443 -L 30500 -L 30141 → <TALOS_VM_IP>
```
Consequences worth knowing:
- Images are **pushed** to `localhost:30500` (the tunnel) and **pulled** by the node from
`<TALOS_VM_IP>:30500` (its own NodePort, the address in the Talos registry-mirror patch).
Same registry, two names — hence the two `K8S_REGISTRY` values in the workflow.
- It calls `make k8s-reseed`, not `make k8s-up`: the bootstrap Jobs are idempotent, and
deleting them first is what stops a changed Job template from wedging `helm upgrade` (§7).
- `dev` is a mutable tag, so a `rollout restart` of the nine repo deployments is what
actually puts the new images in the pods.
- Deploys **queue** (`cancel-in-progress: false`): a helm upgrade killed half-way leaves the
release in `pending-upgrade`, which has to be unwedged by hand.
Settings, all on the repository in Gitea:
| Kind | Name | What |
|---|---|---|
| Secret | `TALOS_SSH_KEY` | private key for `user@labs.respellion.tech` (the Fedora host) |
| Secret | `TALOS_KUBECONFIG` | base64 of the kubeconfig, **`server: https://127.0.0.1:6443`** — Talos puts `127.0.0.1` in the apiserver cert SANs, so TLS still verifies through the tunnel |
| Variable | `TALOS_VM_IP` | the VM's libvirt address (default `192.168.122.173`) |
| Variable | `TALOS_HOST` | the browser-facing host baked into Keycloak's issuer (default `localhost`, see §5) |
The last step smokes `GET /openbaar/register` through the openbaar portal, which exercises
portal → Caddy → BFF → projection. An empty register passes; a 502 does not.
### Reaching the portals from a laptop
The deployed portals are pinned to `http://localhost:30180` for Keycloak (§5), so a browser
needs **all five** browser-facing ports on its own localhost — the portal alone is not
enough, and a missing Keycloak shows up as `ERR_CONNECTION_REFUSED` on
`/realms/*/.well-known/openid-configuration` followed by an opaque `ERROR Error: [object Object]`.
`make k8s-portals` does this when kubectl can reach the cluster; through the lab server one
SSH does it without a kubeconfig at all:
```bash
ssh -N -p 6667 \
-L 30140:<TALOS_VM_IP>:30140 -L 30141:<TALOS_VM_IP>:30141 \
-L 30142:<TALOS_VM_IP>:30142 -L 30143:<TALOS_VM_IP>:30143 \
-L 30180:<TALOS_VM_IP>:30180 \
user@labs.respellion.tech
```
Then the §5 table's URLs work as written. The admin UIs (OpenZaak, Flowable, …) need no
forward — they are server-rendered, so the VM's address is fine.
Not covered: the portals still need `make k8s-portals` (or an SSH forward) to be usable in a
browser, because PKCE needs a secure context (§5). Giving the server a hostname + TLS is the
upgrade path.
## What is not ported ## What is not ported
- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built - **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
@@ -366,5 +458,7 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
`K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`. `K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent. The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the - **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
chart. The Kubernetes path is verified with §5's smoke test. chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
image bump that skipped the chart fails there, but nothing deploys the chart in CI.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033. - **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
"""Fail when a page under docs/ is missing from mkdocs.yml's nav.
docs/ is the source of truth (CLAUDE.md §12), but only the pages listed in the nav
are published — and mkdocs' own `omitted_files: warn` keeps a build green while
silently dropping them, which is how every ADR after 0010 and every runbook but
ci.md fell off the site.
ponytail: a substring test, not a YAML parse — a page's path either appears in
mkdocs.yml or it doesn't, and that needs no dependency.
"""
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
nav = (ROOT / "mkdocs.yml").read_text()
missing = sorted(
str(page.relative_to(ROOT / "docs"))
for page in (ROOT / "docs").rglob("*.md")
if str(page.relative_to(ROOT / "docs")) not in nav
)
if missing:
print(f"{len(missing)} page(s) under docs/ are not in mkdocs.yml's nav:")
print("\n".join(f" {m}" for m in missing))
sys.exit(1)
print("docs nav complete: every page under docs/ is published")
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Fail when the compose stack and the Helm chart stop describing the same stack.
`infra/docker-compose.yml` is CI-canonical; `infra/helm/big-reference` is a
transcription of it (ADR-0033), and until now nothing kept the two in step — an
upstream image bump or a new service applied to only one of them landed
unnoticed. This compares what each side actually *deploys*, not the two files:
the rendered chart against `docker compose config`. Both tools are already
prerequisites of the `k8s-*` make targets.
Run it with `make k8s-drift`. No cluster needed.
ponytail: names and images only, as sets — no per-workload env/ports/volumes.
Those differ by design in four documented places (ADR-0033), so comparing them
would mean re-encoding every deviation field by field; a tag bump and a missing
service are the drift that actually bites.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
COMPOSE = ROOT / "infra/docker-compose.yml"
CHART = ROOT / "infra/helm/big-reference"
# The busybox init container that every `waitFor` workload gets exists only in
# the chart (compose has `depends_on`). Rendering it under a sentinel makes it
# filterable without teaching the check what busybox is.
BUSYBOX = "drift-check-ignored-init-image"
# Differences that Kubernetes forces, not drift (ADR-0033). A name listed here is
# expected to be on exactly one side; anything else fails.
DEVIATIONS = {
# The four Django services apply their own setup_configuration in the web pod
# (`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]`) rather than in a
# separate init Job. Both that script and /start.sh run `manage.py migrate`, and
# Kubernetes has no `depends_on: service_completed_successfully` to serialise them,
# so the Job and its web pod migrated the same database concurrently.
"oz-init": "folded into the openzaak pod",
"nrc-init": "folded into the nrc-web pod",
"objecttypen-init": "folded into the objecttypen pod",
"objecten-init": "folded into the objecten pod",
# Compose seeds these from the host — the verify scripts `docker cp` the two
# scripts into a running container, and docker-compose.local.yml carries
# `local-seed` + `nrc-subscribe` for `make local`. A cluster has no host to seed
# from, so both became Jobs in the chart.
"seed-zaaktype": "compose seeds the catalogus from the host (infra/openzaak/seed_catalogus.py)",
"nrc-subscribe": "compose registers the abonnement from the host (infra/local/register-abonnement.py)",
}
# Workloads the observability backplane adds. Off by default in both stacks'
# defaults, so they are rendered on purpose here — otherwise their images drift
# unwatched.
OBSERVABILITY = ["tempo", "prometheus", "grafana"]
def compose_services() -> dict[str, str]:
"""Service name -> image, with ${TAG:-default} interpolation already applied."""
out = run(["docker", "compose", "-f", str(COMPOSE), "config", "--format", "json"])
return {name: svc.get("image", "") for name, svc in json.loads(out)["services"].items()}
def chart_workloads() -> dict[str, str]:
"""Workload name -> image, read back out of the rendered manifests."""
out = run(
["helm", "template", "big", str(CHART), "-n", "big", "--set", f"images.busybox={BUSYBOX}"]
+ [f"--set=workloads.{w}.enabled=true" for w in OBSERVABILITY]
)
workloads = {}
for doc in out.split("\n---"):
if not re.search(r"^kind: (Deployment|Job)$", doc, re.M):
continue
name = re.search(r"^ name: (\S+)$", doc, re.M)[1]
images = [i for i in re.findall(r"^\s+image: (\S+)$", doc, re.M) if i != BUSYBOX]
workloads[name] = images[0]
return workloads
def run(argv: list[str]) -> str:
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
sys.exit(f"{argv[0]} failed:\n{proc.stderr}")
return proc.stdout
def main() -> int:
compose, chart = compose_services(), chart_workloads()
problems = []
for name in sorted(set(compose) - set(chart) - set(DEVIATIONS)):
problems.append(f" {name}: in docker-compose.yml, not in the chart")
for name in sorted(set(chart) - set(compose) - set(DEVIATIONS)):
problems.append(f" {name}: in the chart, not in docker-compose.yml")
for name in sorted(set(compose) & set(chart)):
if compose[name] != chart[name]:
problems.append(f" {name}: compose runs {compose[name]}, the chart runs {chart[name]}")
if problems:
print("compose and the Helm chart describe different stacks:\n" + "\n".join(problems))
print(
"\nPort the change to the other stack, or — if the difference is forced by\n"
"Kubernetes — declare it in DEVIATIONS in this file, with the reason."
)
return 1
print(f"no drift: {len(chart)} workloads, images identical on both stacks")
for name, why in sorted(DEVIATIONS.items()):
print(f" deviation (declared): {name}{why}")
return 0
if __name__ == "__main__":
sys.exit(main())
+31
View File
@@ -32,6 +32,30 @@ nav:
- "ADR-0008: Read projection store": architecture/adr-0008-read-projection-store.md - "ADR-0008: Read projection store": architecture/adr-0008-read-projection-store.md
- "ADR-0009: External-task job worker": architecture/adr-0009-external-task-job-worker.md - "ADR-0009: External-task job worker": architecture/adr-0009-external-task-job-worker.md
- "ADR-0010: BFF OIDC validation": architecture/adr-0010-bff-oidc.md - "ADR-0010: BFF OIDC validation": architecture/adr-0010-bff-oidc.md
- "ADR-0011: Approval status flow": architecture/adr-0011-approval-status-flow.md
- "ADR-0012: Citizen reference correlation": architecture/adr-0012-citizen-reference-correlation.md
- "ADR-0013: Behandel-portal wiring": architecture/adr-0013-behandel-portal-wiring.md
- "ADR-0014: Withdrawal cancels the process": architecture/adr-0014-withdrawal-cancels-the-process.md
- "ADR-0015: Beoordeling escalation": architecture/adr-0015-beoordeling-escalation.md
- "ADR-0016: Diploma eligibility DMN": architecture/adr-0016-diploma-eligibility-dmn.md
- "ADR-0017: Document-wait timeout": architecture/adr-0017-document-wait-timeout-cancellation.md
- "ADR-0018: Diploma upload via the ACL": architecture/adr-0018-diploma-upload-via-acl-documenten.md
- "ADR-0019: Zaak cancellation on timeout": architecture/adr-0019-zaak-cancellation-on-timeout.md
- "ADR-0020: Local stack self-seeds": architecture/adr-0020-local-stack-self-seeds.md
- "ADR-0021: Zaaktype by identificatie": architecture/adr-0021-acl-resolves-zaaktype-by-identificatie.md
- "ADR-0022: Quartz scheduler": architecture/adr-0022-quartz-scheduler.md
- "ADR-0023: Observability stack": architecture/adr-0023-observability-stack.md
- "ADR-0024: Prometheus AspNetCore exporter": architecture/adr-0024-prometheus-aspnetcore-exporter.md
- "ADR-0025: BFF reads catalogus via the ACL": architecture/adr-0025-bff-reads-catalogus-via-acl.md
- "ADR-0026: Mutable default-fill store": architecture/adr-0026-mutable-default-fill-store.md
- "ADR-0027: RegisterRecord objecttype": architecture/adr-0027-registerrecord-objecttype-schema.md
- "ADR-0028: Objecten holds the register": architecture/adr-0028-objecten-holds-the-register.md
- "ADR-0029: Objecten publishes to NRC": architecture/adr-0029-objecten-publishes-to-nrc.md
- "ADR-0030: Projection sourced from the register": architecture/adr-0030-projection-sourced-from-the-register.md
- "ADR-0031: MFA on the medewerker realm": architecture/adr-0031-mfa-on-the-medewerker-realm.md
- "ADR-0032: Werkbak live refresh": architecture/adr-0032-werkbak-live-refresh.md
- "ADR-0033: Kubernetes via one Helm chart": architecture/adr-0033-kubernetes-via-one-helm-chart.md
- "ADR-0034: Caddy serves the portals": architecture/adr-0034-caddy-serves-the-portals.md
- FDS-architectuur: - FDS-architectuur:
- Overzicht: architecture/fds/README.md - Overzicht: architecture/fds/README.md
- Componentview (L3): architecture/fds/c4-component-view.md - Componentview (L3): architecture/fds/c4-component-view.md
@@ -46,8 +70,15 @@ nav:
- Working in Gitea: gitea-workflow.md - Working in Gitea: gitea-workflow.md
- Frontend decisions: frontend-decisions.md - Frontend decisions: frontend-decisions.md
- Demo script: demo-script.md - Demo script: demo-script.md
- Synthetic data: synthetic-data.md
- Runbooks: - Runbooks:
- CI: runbooks/ci.md - CI: runbooks/ci.md
- OpenZaak: runbooks/openzaak.md
- Open Notificaties (NRC): runbooks/opennotificaties.md
- Keycloak: runbooks/keycloak.md
- Flowable: runbooks/flowable.md
- Kubernetes on Talos: runbooks/kubernetes-talos.md
- Gitea Actions gotchas: runbooks/gitea-actions-gotchas.md
markdown_extensions: markdown_extensions:
- admonition - admonition