Compare commits

..
Author SHA1 Message Date
not 594fdde227 fix(infra): cap celery workers at 2 so the shared node stops OOM-killing CI (closes #182) (#183)
CI / k8s (push) Successful in 9s
CI / build (push) Successful in 1m39s
CI / lint (push) Successful in 2m0s
CI / mutation (push) Canceled after 0s
CI / verify-stack (push) Canceled after 0s
CI / unit (push) Canceled after 1m9s
CI / frontend (push) Canceled after 1m16s
Deploy to Talos / deploy (push) Successful in 2m31s
## What & why

`verify-stack` is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with `CELERY_WORKER_CONCURRENCY` unset, `oz-celery` and `nrc-celery` each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182.

This sets `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups:
- **compose** (`&oz-env`, `&nrc-env`): what `verify-stack` starts inside `dind`.
- **chart** (`envGroups.oz` / `.nrc`): the deployed demo on the same node.

Both images' `/celery_worker.sh` honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. `objecten-celery` already defaults to 1.

Closes #182

## Definition of Done

- [x] Linked Gitea issue (above).
- [ ] Failing test committed before the implementation. *(Resource setting; the evidence is the OOM log in #182.)*
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [ ] CI green. This PR's `verify-stack` run is the check.
- [x] `docker compose config` renders the variable into all 7 services on the two anchors.
- [x] Docs: comments next to the setting, following the existing uWSGI notes.

## Notes for reviewers

- `make k8s-lint` and `make k8s-drift` pass.
- **Not applied live.** I couldn't patch the running cluster from my session. After merge, the deploy updates the `oz-env` / `nrc-env` ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once:
  `kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery`
- **Why 2 and not 1:** this matches `UWSGI_THREADS: "2"`, and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages).
- **Longer term:** CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #183
2026-09-25 11:11:45 +00:00
not 804031eeb8 feat(k8s): publish the portals through the labs Caddy (refs #177) (#179)
CI / k8s (push) Successful in 14s
CI / build (push) Successful in 2m2s
CI / lint (push) Successful in 2m32s
CI / unit (push) Successful in 1m36s
CI / frontend (push) Successful in 3m3s
Deploy to Talos / deploy (push) Successful in 3m39s
CI / mutation (push) Successful in 5m31s
CI / verify-stack (push) Failing after 17m26s
## What & why

Makes the portals reachable on real hostnames through the Caddy that already fronts `*.labs.respellion.tech`, instead of five SSH port-forwards:

| URL | Service |
|---|---|
| `https://big-register.labs.respellion.tech` | openbaar |
| `https://big-mijn.labs.respellion.tech` | self-service |
| `https://big-behandel.labs.respellion.tech` | behandel |
| `https://big-beheer.labs.respellion.tech` | beheer |
| `https://big-auth.labs.respellion.tech` | Keycloak (`/admin` blocked) |

Chain: browser → labs Caddy (TLS) → `openssh-server` container → reverse SSH tunnel → Fedora host → Talos NodePorts. The Caddy routes and the tunnel unit are already on `main` in the Infra repo (`infra/development/`).

This repo's part:
- **Chart:** a `keycloakUrl` value. When set it replaces `host` + Keycloak's NodePort as the pinned issuer (`KC_HOSTNAME`) and the portals' OIDC authority. Both now come from one helper, `big.keycloakUrl`, so they can't drift apart (ADR-0010). Empty = rendered output identical to today.
- **Deploy workflow:** passes the `KEYCLOAK_URL` repo variable as `--set keycloakUrl=…`.
- **Runbook:** new section "Publishing through the labs Caddy".

Refs #177

## Definition of Done

- [x] Linked Gitea issue (above).
- [ ] Failing test committed before the implementation. *(Infra/config change, no test added.)*
- [x] Implementation makes the test pass; refactor commit if structure improved.
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [ ] CI green — all Gitea Actions jobs (or `make ci` green while no runner exists).
- [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes. *(Compose untouched.)*
- [x] Docs updated if behaviour, contracts, or operations changed.
- [ ] ADR added in `docs/architecture/` if a non-obvious decision was made.
- [ ] Demo note in `docs/demo-script.md` if user-visible.

## Notes for reviewers

- **This takes the option #177 rejects.** #177 proposes an in-cluster Caddy edge (branch `feat/177-public-tls-edge`). This PR uses the existing labs Caddy instead, because it already holds 80/443 and the wildcard certificate. So it only *refs* #177. If we go this way, #177's ADR should record the host-Caddy option instead.
- `make k8s-lint` and `infra/check-docs-nav.py` pass. I rendered the chart with and without `keycloakUrl`: empty gives the same output as before; set, it gives `https://big-auth.labs.respellion.tech` for both the issuer and the authority.
- Once `KEYCLOAK_URL` is set, the `localhost` port-forward workflow (runbook §5) no longer logs in, because the issuer is a single string.
- The portals are public, with no Azure `authorize` in front of them the way `marketing` has one. The test users use `test123`.
- Rollout after merge: install `big-portals-tunnel.service` on the Fedora host, run `docker compose up -d caddy` on the labs server, then set the `KEYCLOAK_URL` variable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #179
2026-09-25 08:16:37 +00:00
2 changed files with 7 additions and 0 deletions
+5
View File
@@ -57,6 +57,9 @@ services:
# share this anchor and ignore it — they don't run uwsgi.
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
# Same lever for oz-celery: unset, the worker forks one process per CPU (22 on the lab node,
# ~225 MB each), which OOM-killed the shared runner mid-verify-stack. Only celery reads it.
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: openzaak.conf.docker
SECRET_KEY: ${OZ_SECRET_KEY:-dev-only-not-for-production}
DB_HOST: oz-db
@@ -144,6 +147,8 @@ services:
# 1 uWSGI worker, not the image default of 4×4 (#147) — see the oz-env note above.
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
# Two celery workers, not one per CPU — see the oz-env note above.
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: nrc.conf.docker
SECRET_KEY: ${NRC_SECRET_KEY:-dev-only-not-for-production}
DB_HOST: nrc-db
+2
View File
@@ -76,6 +76,7 @@ envGroups:
oz:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: openzaak.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: oz-db
@@ -98,6 +99,7 @@ envGroups:
nrc:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: nrc.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: nrc-db