From ba24784586b0a9e9ad10718eee6d36173769be9f Mon Sep 17 00:00:00 2001 From: Edwin van den Houdt Date: Fri, 31 Jul 2026 08:23:01 +0200 Subject: [PATCH] feat(openzaak): one-command UI-OpenZaak bridge, real BSN fix, flake mitigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/openzaak-ui-up.sh brings up the root app and the OpenZaak harness together, wires them onto one docker network, seeds the catalogus, grants the container-alias zaaktype scope, and verifies a real aanvraag submitted through the UI lands in OpenZaak. Along the way: DocumentStore.DemoOwner was reusing the seeded doctor's 11-digit BIG-nummer as a stand-in BSN, which isn't a valid 9-digit BSN shape — OpenZaak rejects it, breaking both submit's rol-creation step and the citizen's own applications list under Zgw:Enabled=true. Fixed to a real elfproef-valid BSN. Also adds mitigation for a still-unexplained per-container flake (every outbound ZGW POST fails as if the body were empty, for that container's whole lifetime) that correlates with host memory pressure: the script now warns when host swap is heavily used, and an opt-in ZgwDiagnosticHandler (ZGW_DEBUG_HTTP=1) logs Content-Length vs. actual bytes sent so the next reproduction can confirm or rule out client-side body corruption. Co-Authored-By: Claude Sonnet 5 --- backend/openzaak/README.md | 36 +++++ .../openzaak/docker-compose.openzaak.bff.yml | 38 +++++ .../src/BigRegister.Api/Data/DocumentStore.cs | 12 +- backend/src/BigRegister.Api/Program.cs | 13 +- .../Zgw/ZgwDiagnosticHandler.cs | 25 +++ .../ZgwDiagnosticHandlerTests.cs | 28 ++++ docker-compose.openzaak.yml | 50 ++++++ scripts/openzaak-ui-up.sh | 150 ++++++++++++++++++ 8 files changed, 348 insertions(+), 4 deletions(-) create mode 100644 backend/openzaak/docker-compose.openzaak.bff.yml create mode 100644 backend/src/BigRegister.Api/Zgw/ZgwDiagnosticHandler.cs create mode 100644 backend/tests/BigRegister.Tests/ZgwDiagnosticHandlerTests.cs create mode 100644 docker-compose.openzaak.yml create mode 100755 scripts/openzaak-ui-up.sh diff --git a/backend/openzaak/README.md b/backend/openzaak/README.md index ce0ed4b..b49488a 100644 --- a/backend/openzaak/README.md +++ b/backend/openzaak/README.md @@ -80,6 +80,42 @@ BFF_AUTH='' ./verify-notificatie.sh # proves a real delivery, To go back to the fast, no-notifications default: `docker compose -f docker-compose.openzaak.yml up -d --remove-orphans` (drops the celery worker, restores `NOTIFICATIONS_DISABLED: 'true'`). +## Testing the Angular UI against this harness + +The base harness above and the app's own root `docker-compose.yml` are independent projects on +purpose (see the top of this file) — this is the opt-in bridge between them, for when you want +to click through the real UI and see an aanvraag land in a real OpenZaak instead of just +running `dotnet test`. One command from the repo root: + +```bash +scripts/openzaak-ui-up.sh +``` + +It brings up the root app (so its docker network exists), brings up this harness plus +`docker-compose.openzaak.bff.yml` (gives this harness's `web` service a dotted alias, +`openzaak.local`, on its own network — the root project's `api` container joins THIS network, +in the opposite direction from the notifications overlay below, to avoid a real alias +collision: the root project's frontend service is also called `web`. The alias needs a dot +because Django's URLValidator rejects a bare hostname in a URL field; this environment's +rootless Podman also can't route container→host-port traffic through `host.docker.internal`, +so container-to-container is the only reliable path either way — see that file's header +comment for the full, empirically-confirmed reasoning), seeds the catalogus, additively +replaces the zrc authorization grant to match the alias (ZGW authorization is scoped by the +*exact* zaaktype URL string, not just the resource; see `scripts/openzaak-ui-up.sh`'s own +comment for why this is a replace, not an add), and brings the root app back up pointed at +OpenZaak (`docker-compose.openzaak.yml` at the repo root). A final self-check submits a +throwaway aanvraag and confirms it actually lands in OpenZaak, restarting `api` (up to 5 +times) if not — see that script for a caveat about an intermittent per-container networking +flake this environment can hit under memory pressure (the script now warns if host swap is +already high going in; `ZGW_DEBUG_HTTP=1` on `api`, see `docker-compose.openzaak.yml`, logs +diagnostics to help nail the cause next time it reproduces). Prints the URLs to check +afterward and the teardown commands. + +Two caveats, both non-fatal (WP-60 catches and flags rather than surfacing an error): +**only `herregistratie` has a seeded zaaktype** here, so submit that wizard to prove a real +write; and **no Documenten content is seeded**, so a document upload's ZGW half no-ops (pick +"per post" in the wizard's document step, or ignore it). + ## Tear down ```bash diff --git a/backend/openzaak/docker-compose.openzaak.bff.yml b/backend/openzaak/docker-compose.openzaak.bff.yml new file mode 100644 index 0000000..b00b904 --- /dev/null +++ b/backend/openzaak/docker-compose.openzaak.bff.yml @@ -0,0 +1,38 @@ +# Opt-in overlay, layered ON TOP of docker-compose.openzaak.yml (never alone): +# +# docker compose -f docker-compose.openzaak.yml -f docker-compose.openzaak.bff.yml up -d +# +# Gives this harness's `web` (the OpenZaak API) an extra, dotted hostname alias +# (`openzaak.local`) on its OWN network, so the root project's `api` container (joined in via +# `docker-compose.openzaak.yml` at the repo root, as an EXTERNAL network) can reach it. See +# `scripts/openzaak-ui-up.sh` for the one-command version that brings both projects up +# together, seeds the catalogus, and grants the extra authorization scope this alias needs. +# +# Three real things this works around, each discovered empirically (curl against the +# running containers), not guessed: +# +# 1. Why container-to-container instead of `http://localhost:8000`: this dev environment's +# rootless Podman drops container→host-port traffic through `host.docker.internal` +# (confirmed for the WP-58 notifications overlay's celery worker — DNS resolves it, every +# TCP connect times out). +# +# 2. Why the ROOT project's `api` joins INTO this project's network (below), not the other way +# around: the root project's frontend service is also called `web`. Docker Compose always +# adds a service's own name as a network alias on every network it joins — so if THIS `web` +# joined the root project's network, "web" would resolve to two different containers there. +# Only `api` crosses into this network, under its own already-unique name. +# +# 3. Why the alias has a dot in it (`openzaak.local`, not e.g. `openzaak`): Django's built-in +# URLValidator rejects a bare, dotless hostname in a URL field (it special-cases exactly +# "localhost"; anything else needs a dot or to be a valid IP). OpenZaak's `zaaktype` field +# (and others) run through this validator — confirmed with a POST referencing +# `http://:8000/...` failing "Voer een geldige URL in" (enter a valid URL) +# before any authorization check even runs. +services: + web: + environment: + # Django rejects any request whose Host header isn't in ALLOWED_HOSTS. + ALLOWED_HOSTS: localhost,127.0.0.1,web,openzaak.local + networks: + default: + aliases: [openzaak.local] diff --git a/backend/src/BigRegister.Api/Data/DocumentStore.cs b/backend/src/BigRegister.Api/Data/DocumentStore.cs index b190cd1..3737f12 100644 --- a/backend/src/BigRegister.Api/Data/DocumentStore.cs +++ b/backend/src/BigRegister.Api/Data/DocumentStore.cs @@ -36,8 +36,16 @@ public sealed record AuditEntry(DateTimeOffset At, string Action, string Documen /// public static class DocumentStore { - /// The single seeded user (the demo has no real auth; ownership = this id). - public const string DemoOwner = "19012345601"; + /// The single seeded user (the demo has no real auth; ownership = this id) — a real, + /// elfproef-valid 9-digit BSN (src/app/shared/kernel/bsn.ts's own checksum), distinct from + /// SeedData.Registration.BigNummer ("19012345601", 11 digits — the seeded doctor's BIG-nummer, + /// a different Dutch identifier scheme). Previously this constant reused that BigNummer value + /// as a stand-in BSN, which is invalid Dutch-BSN shape: harmless against the local store, but + /// a real OpenZaak instance rejects it outright — GET /api/v1/applications 500s (`inpBsn` query + /// filter validation) and every submit's rol-creation POST fails (`inpBsn` max_length) once + /// Zgw:Enabled=true. Not "111222333" or "999888777" — both already mean a different fixture + /// identity (the OpenZaak-harness/unit-test caller, and ApplicationTests' "other citizen"). + public const string DemoOwner = "123456782"; private static readonly object _gate = new(); diff --git a/backend/src/BigRegister.Api/Program.cs b/backend/src/BigRegister.Api/Program.cs index 584f609..5623f6e 100644 --- a/backend/src/BigRegister.Api/Program.cs +++ b/backend/src/BigRegister.Api/Program.cs @@ -61,9 +61,18 @@ if (zgw.Enabled) // WP-60: a bounded client timeout matters once ZgwHttpClient retries — without one, the // sources' sync-over-async call (no CancellationToken threaded through) could block a // thread-pool thread for HttpClient's 100s default times 3 attempts. - builder.Services.AddHttpClient(c => c.Timeout = TimeSpan.FromSeconds(15)); + var zaakClientBuilder = builder.Services.AddHttpClient(c => c.Timeout = TimeSpan.FromSeconds(15)); // WP-51: the documents (Documenten API / DRC) seam — same pattern as IZaakSource above. - builder.Services.AddHttpClient(c => c.Timeout = TimeSpan.FromSeconds(15)); + var documentClientBuilder = builder.Services.AddHttpClient(c => c.Timeout = TimeSpan.FromSeconds(15)); + + // Opt-in diagnostic for the still-unexplained per-container flake (see + // scripts/openzaak-ui-up.sh's header comment) — off by default, zero cost unless set. + if (Environment.GetEnvironmentVariable("ZGW_DEBUG_HTTP") == "1") + { + builder.Services.AddTransient(); + zaakClientBuilder.AddHttpMessageHandler(); + documentClientBuilder.AddHttpMessageHandler(); + } } else { diff --git a/backend/src/BigRegister.Api/Zgw/ZgwDiagnosticHandler.cs b/backend/src/BigRegister.Api/Zgw/ZgwDiagnosticHandler.cs new file mode 100644 index 0000000..e79897f --- /dev/null +++ b/backend/src/BigRegister.Api/Zgw/ZgwDiagnosticHandler.cs @@ -0,0 +1,25 @@ +namespace BigRegister.Api.Zgw; + +/// +/// Opt-in only (wired in Program.cs behind ZGW_DEBUG_HTTP=1) — chases the +/// still-unexplained flake where a freshly-(re)started `api` container has every outbound ZGW +/// POST fail with what looks like an empty body reaching OpenZaak (see +/// scripts/openzaak-ui-up.sh's header comment). Logs the one signal that would actually +/// distinguish "client built an empty body" from "something ate it after send": the declared +/// Content-Length vs. the byte count actually read from the request right before it goes out. +/// A mismatch here would prove client-side corruption; agreement would point downstream instead. +/// +public sealed class ZgwDiagnosticHandler(ILogger logger) : DelegatingHandler +{ + protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + if (request.Content is not null) + { + var bytes = await request.Content.ReadAsByteArrayAsync(cancellationToken); + logger.LogInformation( + "ZGW {Method} {Url}: Content-Length={ContentLength} actualBytes={Actual}", + request.Method, request.RequestUri, request.Content.Headers.ContentLength, bytes.Length); + } + return await base.SendAsync(request, cancellationToken); + } +} diff --git a/backend/tests/BigRegister.Tests/ZgwDiagnosticHandlerTests.cs b/backend/tests/BigRegister.Tests/ZgwDiagnosticHandlerTests.cs new file mode 100644 index 0000000..66cbf29 --- /dev/null +++ b/backend/tests/BigRegister.Tests/ZgwDiagnosticHandlerTests.cs @@ -0,0 +1,28 @@ +using System.Net.Http.Json; +using BigRegister.Api.Zgw; +using Microsoft.Extensions.Logging.Abstractions; + +namespace BigRegister.Tests; + +/// +/// The opt-in diagnostic handler (see ZgwDiagnosticHandler's own doc comment) must be inert on +/// the request/response — it only observes. This doesn't catch the flake itself (that needs a +/// real repro with ZGW_DEBUG_HTTP=1), just proves the hook doesn't alter what's sent or break +/// the response passthrough. +/// +public class ZgwDiagnosticHandlerTests +{ + [Fact] + public async Task Passes_request_and_response_through_unchanged() + { + var stub = new ZgwStubHandler(_ => """{ "ok": true }"""); + var diagnostic = new ZgwDiagnosticHandler(NullLogger.Instance) { InnerHandler = stub }; + using var client = new HttpClient(diagnostic); + + var response = await client.PostAsJsonAsync("https://oz.example/zaken", new { foo = "bar" }); + + Assert.True(response.IsSuccessStatusCode); + Assert.Single(stub.Requests); + Assert.Contains("\"foo\":\"bar\"", stub.BodyOf("https://oz.example/zaken")); + } +} diff --git a/docker-compose.openzaak.yml b/docker-compose.openzaak.yml new file mode 100644 index 0000000..27fab2f --- /dev/null +++ b/docker-compose.openzaak.yml @@ -0,0 +1,50 @@ +# Opt-in overlay, layered ON TOP of docker-compose.yml (never alone): +# +# docker compose -f docker-compose.yml -f docker-compose.openzaak.yml up -d +# +# Points the containerized BFF (`api`) at a real OpenZaak instead of the local SQLite store. +# `api` joins the OpenZaak project's OWN network (external, below) to reach its `web` service +# by the dotted alias `docker-compose.openzaak.bff.yml` (backend/openzaak/) gives it — +# `openzaak.local`, not a bare `openzaak`: Django's URLValidator rejects a dotless hostname +# embedded in a URL field (confirmed empirically — see that file's header comment for the full +# reasoning, including why the join runs in this direction and not the reverse). See +# `scripts/openzaak-ui-up.sh` for the one command that brings both projects up together, seeds +# the catalogus, grants the extra scope this alias needs, and fills in OPENZAAK_ZAAKTYPE_URL. +# +# ClientId/Secret/RSINs match exactly what backend/openzaak/bootstrap-catalogus.sh provisions. +# Only `herregistratie` gets a ZaaktypeUrls entry: the harness seeds exactly one zaaktype +# ("Herregistratie arts") — a registratie/intake submission would hit an unconfigured zaaktype, +# caught by WP-60's retry/flagging (Aanvraag.ZgwError), not surfaced as a UI error. DrcBaseUrl/ +# InformatieobjecttypeUrls are deliberately left unset: this harness seeds no Documenten +# content or scope, so a document upload's ZGW half just no-ops (also caught since WP-60). +# +# ZGW authorization scopes a zaaktype write by the EXACT zaaktype URL string an Applicatie was +# granted for (confirmed empirically: the same zaaktype, referenced via a different hostname +# string, 403s even though the URL itself resolves fine) — bootstrap-catalogus.sh only ever +# grants the `http://localhost:8000/...` form (it runs on the host). scripts/openzaak-ui-up.sh +# additively grants the SAME scope again for the `openzaak.local:8000` form this file's `api` +# actually presents, without touching bootstrap-catalogus.sh's own (host-usable) grant. +services: + api: + environment: + - Zgw__Enabled=true + - Zgw__ZrcBaseUrl=http://openzaak.local:8000/zaken/api/v1 + - Zgw__ZtcBaseUrl=http://openzaak.local:8000/catalogi/api/v1 + # Uncomment to chase the per-container flake (scripts/openzaak-ui-up.sh's header + # comment): logs Content-Length vs. actual bytes sent for every outbound ZGW POST. + # - ZGW_DEBUG_HTTP=1 + - Zgw__ClientId=bigregister-test + - Zgw__Secret=bigregister-test-secret + - Zgw__UserId=bigregister-test + - Zgw__UserRepresentation=Docker compose OpenZaak test + - Zgw__Bronorganisatie=123443210 + - Zgw__VerantwoordelijkeOrganisatie=123443210 + - Zgw__ZaaktypeUrls__herregistratie=${OPENZAAK_ZAAKTYPE_URL:?run backend/openzaak/bootstrap-catalogus.sh and export OPENZAAK_ZAAKTYPE_URL first — see scripts/openzaak-ui-up.sh for the one-command version} + networks: + default: {} + oz: {} + +networks: + oz: + name: openzaak_default + external: true diff --git a/scripts/openzaak-ui-up.sh b/scripts/openzaak-ui-up.sh new file mode 100755 index 0000000..0daf756 --- /dev/null +++ b/scripts/openzaak-ui-up.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# One command to test the Angular UI end-to-end against a real OpenZaak instead of the local +# SQLite store: brings up the OpenZaak harness (backend/openzaak/), seeds its catalogus, wires +# it onto the root app's docker network (docker-compose.openzaak.bff.yml / +# docker-compose.openzaak.yml — see those files for the full "why", it's non-obvious), grants +# the extra authorization scope the container alias needs, and brings the FE+BFF up pointed at +# OpenZaak. +# +# Safe to re-run: every step this chains is already idempotent (bootstrap-catalogus.sh, +# `docker compose up -d`, and the grant-replace below). +# +# Also confirmed empirically, many repeated trials: some freshly-(re)started `api` +# containers have EVERY outbound ZGW POST fail with what looks like an empty body reaching +# OpenZaak ("all fields required"), for that container's entire lifetime — while a plain curl +# to the exact same URL, from inside the exact same container, never fails, even hammered in a +# loop. Ruled out as the cause: HttpClient connection pooling settings, Expect-100-Continue, +# content pre-buffering — none of it made a measurable difference. Best lead so far, and +# reproduced live on this dev host (7.5/8GB swap from long-idle unrelated containers): this +# correlates with the HOST being under heavy memory pressure — a heavier managed runtime +# (dotnet's JIT + GC) is plausibly far more sensitive to that than a lightweight one-shot +# `curl` process. If you hit this, try freeing host memory (stop unrelated containers) before +# assuming it's a code regression. The preflight check and self-check below warn about and +# retry around it either way; set ZGW_DEBUG_HTTP=1 on the `api` container (see +# docker-compose.openzaak.yml) next time it reproduces to log Content-Length vs. actual bytes +# sent, which would confirm (or rule out) client-side body corruption. +set -euo pipefail +cd "$(dirname "$0")/.." + +step() { printf '\n\033[1;36m▶ %s\033[0m\n' "$1"; } + +check_memory_pressure() { + local total used pct + read -r total used <<< "$(free -m | awk '/^Swap:/{print $2, $3}')" + [ "${total:-0}" -gt 0 ] || return 0 + pct=$(( used * 100 / total )) + if [ "$pct" -ge 50 ]; then + echo "⚠ host swap ${pct}% used (${used}MiB/${total}MiB) — this correlates with the" >&2 + echo " known per-container ZGW flake (see comment up top). Consider 'docker ps' and" >&2 + echo " stopping unrelated long-running stacks before continuing." >&2 + fi +} +check_memory_pressure + +step "root app (creates the docker network the OpenZaak harness joins below)" +docker compose up -d + +step "OpenZaak harness + bff overlay" +( cd backend/openzaak && docker compose -f docker-compose.openzaak.yml -f docker-compose.openzaak.bff.yml up -d ) + +step "seed catalogus/zaaktype/zaak (idempotent)" +bootstrap_output=$(cd backend/openzaak && ./bootstrap-catalogus.sh) +echo "$bootstrap_output" + +zaaktype_line=$(printf '%s\n' "$bootstrap_output" | grep -A1 '^Zaaktype (concept)\.\.\.$' | tail -n1) +zaaktype_url=$(printf '%s\n' "$zaaktype_line" | sed -E 's/^ *(exists|created): //') +zaaktype_uuid=$(printf '%s\n' "$zaaktype_url" | sed 's#.*/##') +if [ -z "$zaaktype_uuid" ]; then + echo "Could not find the seeded zaaktype's URL in bootstrap-catalogus.sh's output — see above." >&2 + exit 1 +fi +# bootstrap-catalogus.sh queried (and granted scope) via http://localhost:8000 (run from the +# host); the containerized BFF reaches the same resource via the `openzaak.local` alias +# (docker-compose.openzaak.bff.yml) instead — only the UUID suffix is what actually matters. +container_zaaktype_url="http://openzaak.local:8000/catalogi/api/v1/zaaktypen/${zaaktype_uuid}" +export OPENZAAK_ZAAKTYPE_URL="$container_zaaktype_url" + +step "grant the container-alias zaaktype scope (REPLACES bootstrap-catalogus.sh's own localhost-scoped grant, doesn't add to it — OpenZaak's own zaken-list authorization filter 500s with a RuntimeError, 'are you sure that all paths point to the same resource?', when an Applicatie has two zrc grants for what's really the same zaaktype under two different hostnames; confirmed empirically. One consequence: dotnet test --filter Category=Integration needs bootstrap-catalogus.sh rerun afterward to restore its localhost-scoped grant — it's idempotent, so that's a plain rerun, not a reset)" +( cd backend/openzaak && docker compose -f docker-compose.openzaak.yml exec -T --workdir /app/src web python manage.py shell ) <&2; break; } + if zgw_write_reaches_openzaak; then + verified=true + break + fi + echo " attempt $attempt/$attempts: a write didn't reach OpenZaak — restarting api and giving the network a moment to settle (known flake, see comment up top)..." + sleep 3 + docker compose -f docker-compose.yml -f docker-compose.openzaak.yml restart api +done +if [ "$verified" = true ]; then + echo " verified: a write reaches OpenZaak." +else + check_memory_pressure + echo " Could not verify after $attempts tries. This is the known flake, not necessarily a real failure —" >&2 + echo " keep retrying by hand: 'docker compose -f docker-compose.yml -f docker-compose.openzaak.yml restart api'," >&2 + echo " wait for 'Application started' in 'docker logs -f atomic-design-poc-api-1', then try the UI again." >&2 +fi + +printf '\n\033[1;32m✔ up\033[0m\n' +cat <