docs(architecture): record why single-binary Tempo has its ingester health check off (refs #156)

`verify-tracing` flaked on run 722: `FAIL — no single trace spanned ['bff',
'projection-api']`, green on a plain re-run of the same commit. Not a broken trace
chain — Tempo could not ingest:

    removing distributor_pool failing healthcheck addr=127.0.0.1:9095
      reason="rpc error: code = DeadlineExceeded"
    pusher failed to consume trace data  err="context canceled"   (x18)

Root cause is the mechanism of the data loss, not whatever caused the stall. Tempo
runs single-binary, so distributor and ingester are the *same process* and the
distributor's ingester pool holds exactly one, in-process, member. dskit still
health-checks it over loopback gRPC with a 1s deadline; on the shared runner a
transient stall blows that, the only ingester is dropped from the pool, and every
push fails until the next 15s check interval — spans silently lost. With one
in-process ingester the check can never route around a failure, so it can only ever
discard data.

Fix: `ingester_client.pool_config.healthcheckenabled: false`. This addresses both
candidate triggers (GC pressure near `mem_limit`, CPU contention) at the point where
they turn into lost data, so `mem_limit: 400m` stays untouched — raising it would
risk reintroducing the verify-e2e OOM of #144 on a memory-tight runner.

Also print `tempo_distributor_ingester_clients` on the check's failure path: a
recurrence then names Tempo-dropped-spans instead of costing another container-log
dive, since from the check's side that is indistinguishable from missing
instrumentation.

Verified against the built image: config parses (`-config.verify`), the effective
`/status/config` reports `healthcheckenabled: false`, and the diagnostic reads the
metric off a live Tempo.

docs/architecture/adr-0031-mfa-on-the-medewerker-realm.md

@@ -1,49 +0,0 @@
-# ADR-0031 — MFA on the medewerker realm, with a fixture TOTP secret
-
-- **Status:** Accepted
-- **Date:** 2026-09-03
-- **Slice:** S-15c (Gitea #132)
-
-## Context
-
-Staff (behandelaar, teamlead, beheerder) act on citizens' registrations and on the ACL's
-default-fill: the highest-privilege logins in the platform. The medewerker realm protected
-them with a password alone, while the citizen realms (digid, eherkenning, eidas) mock
-brokers that carry their own assurance levels. A reference application that demonstrates a
-government architecture should show MFA on the staff realm.
-
-Two things had to be decided: **how** to enforce OTP in a realm export, and **how the
-automated checks and a human demo obtain a code** — the e2e drives a real browser login and
-`make keycloak-smoke` drives a real password grant, so neither can scan a QR.
-
-## Decision
-
-**Enforce OTP by giving every seeded medewerker a TOTP credential**, rather than replacing
-Keycloak's browser flow with a copy whose OTP execution is `REQUIRED`.
-
-Keycloak's stock `browser` and `direct grant` flows both contain a *conditional OTP*
-subflow that fires when the user has an OTP credential. Seeding the credential therefore
-turns the challenge on for every seeded user, in both flows, without duplicating ~40 lines
-of flow JSON into the export. `CONFIGURE_TOTP` is additionally set as a **default required
-action**, so a medewerker created later must enrol before their first login.
-
-**The seeded secret is a fixed, committed fixture** (`BIGMEDEWERKEROTPSEED`) shared by all
-medewerkers. Codes are then computable: `infra/keycloak/check_realms.py` (Python, stdlib
-`hmac`) and `tests/e2e/medewerker-login.ts` (Node `crypto`) each implement RFC 6238 in
-about six lines — no OTP dependency on either side, and no enrolment step in the tests.
-
-## Consequences
-
-- A password alone no longer yields a token on the medewerker realm; `check_realms.py`
-  asserts that refusal, so the enforcement cannot silently regress.
-- Every medewerker login in the e2e goes through `loginMedewerker()`, which submits the OTP
-  form. New staff specs must use it.
-- **The secret is public.** It is a demo fixture and worthless outside this synthetic
-  stack, in the same class as the committed `test123` passwords and the mock DigiD broker.
-  A real deployment enrols per-user authenticators (or federates to DigiD Machtigen /
-  eHerkenning at the required assurance level) and seeds no credentials at all.
-- Enforcement is *effectively* realm-wide but *technically* per-user: the conditional
-  subflow is what fires. A medewerker whose OTP credential were removed would fall back to
-  the required action at next login (enrol, then challenge) rather than skipping MFA — an
-  acceptable equivalence for this purpose, and the reason the required action is set.
-- Reversal is a one-file edit: drop the `otp` credentials and the `requiredActions` block.

docs/demo-script.md

@@ -140,8 +140,7 @@ zaaktype cache). Store is in-memory: an edit reverts to the configured env on re
 
 ```bash
 make up
-# 1. Log in as bram-beheerder / test123 + OTP (`python3 infra/keycloak/check_realms.py otp`)
-#    → "Default-fill" tab → change a value → Opslaan.
+# 1. Log in as bram-beheerder / test123 → "Default-fill" tab → change a value → Opslaan.
 open http://localhost:8143/default-fill
 #
 # 2. Automated: the ACL uses the current default-fill per zaak (unit) and the endpoints are behind the
@@ -162,8 +161,7 @@ directly (ADR-0025); managing the default-fill config (S-15b) and MFA (S-15c) co
 
 ```bash
 make up
-# 1. Log in as bram-beheerder / test123 + OTP (`python3 infra/keycloak/check_realms.py otp`)
-#    → the catalogus lists the published zaaktypen.
+# 1. Log in as bram-beheerder / test123 → the catalogus lists the published zaaktypen.
 open http://localhost:8143
 #
 # 2. Automated (a CI verify-stack e2e): a beheerder logs in and sees BIG-REGISTRATIE.
@@ -306,8 +304,7 @@ make verify-local     # → "OK — a fresh local stack completed the flow with
 
 # 3. Or by hand in the browser: log in at http://localhost:8140 (jan-burger / test123), submit +
 #    upload a PDF, then approve it in the werkbak at http://localhost:8142 (merel-behandelaar /
-#    test123 + OTP, see S-15c); it shows as INGESCHREVEN in the openbaar register at
-#    http://localhost:8141.
+#    test123); it shows as INGESCHREVEN in the openbaar register at http://localhost:8141.
 ```
 
 > The zaaktype is discovered by the ACL itself since S-27 (below); `local-seed`'s `acl.env` now
@@ -592,7 +589,7 @@ or **afwijzen** — which also completes the Beoordelen task so the process adva
 
 ```text
 # 1. Open the behandel portal and log in as a behandelaar (medewerker realm):
-#    http://localhost:8142/   →  merel-behandelaar / test123 + OTP
+#    http://localhost:8142/   →  merel-behandelaar / test123
 #
 # 2. The werkbak lists the registrations awaiting beoordeling (referentie / bsn / status).
 #    Find the reference from the submit confirmation and click "Goedkeuren" on that row.
@@ -815,32 +812,3 @@ make verify-domain     # → "the timed-out registration's zaak was cancelled to
 `POST /annuleringen` → ZGW `resultaten` + `statussen` (Geannuleerd); the aggregate then moves to
 `Verlopen`. The ACL cancels the zaak **before** the aggregate is expired, so a failed ZGW call leaves the
 job for redelivery rather than diverging the two (ADR-0019).
-
----
-
-## S-15c — MFA on the medewerker realm (#132, ADR-0031)
-
-**Outcome:** staff logins (behandel + beheer portals) need a **second factor**. The medewerker realm
-seeds every medewerker with a TOTP credential, so Keycloak's conditional-OTP step challenges them in
-both the browser flow and the direct grant; a password alone no longer yields a token. `CONFIGURE_TOTP`
-is a default required action, so a medewerker added later must enrol first. Citizen realms (digid,
-eherkenning, eidas) are unchanged — they mock brokers that carry their own assurance.
-
-```bash
-# 1. Manual: log in to the behandel portal. After username + password Keycloak asks for a code.
-python3 infra/keycloak/check_realms.py otp   # a valid code, right now
-open http://localhost:8142                   # merel-behandelaar / test123 + that code
-#
-# 2. Automated: the realm smoke check asserts the password alone is REFUSED, then that
-#    password + TOTP succeeds and still carries the behandelaar role:
-make keycloak-smoke    # → "medewerker  merel-behandelaar  password-only login refused [OK]"
-#
-# 3. End-to-end: every staff login in the e2e goes through the OTP prompt (loginMedewerker):
-make verify-e2e        # → registration.spec (behandelaar approves), catalogus.spec, default-fill.spec
-```
-
-**The path:** the seeded `otp` credential in `infra/keycloak/realms/medewerker-realm.json` activates
-Keycloak's stock conditional-OTP subflow — no custom browser flow. The fixture secret is shared and
-committed on purpose so the checks can compute codes; a real deployment enrols per-user authenticators
-(ADR-0031).
-

docs/runbooks/keycloak.md

@@ -23,9 +23,6 @@ login per realm and asserts the identifying claim:
 | eidas | pierre-dupont | `eidas_id` |
 | medewerker | merel-behandelaar | role `behandelaar` |
 
-The medewerker row also asserts that the password **alone** is refused — that realm
-enforces MFA (below).
-
 All test users / credentials are in [../synthetic-data.md](../synthetic-data.md).
 
 ## Notes
@@ -38,30 +35,3 @@ All test users / credentials are in [../synthetic-data.md](../synthetic-data.md)
 - **Image** pinned to `quay.io/keycloak/keycloak:26.1`.
 - Claims are injected by OIDC protocol mappers on `big-portal` (user attribute → token
   claim); `medewerker` roles come through `realm_access.roles`.
-
-## MFA on the medewerker realm (S-15c)
-
-Staff logins (behandel + beheer portals) need a second factor; citizen/company realms
-(digid, eherkenning, eidas) do not. Two halves in `medewerker-realm.json`:
-
-- Every seeded medewerker carries a **TOTP credential** with the fixture secret
-  `BIGMEDEWERKEROTPSEED`, so Keycloak's built-in *conditional OTP* step fires on every
-  login — browser flow (an `#otp` prompt after the password) and direct grant (a `totp`
-  form field) alike.
-- `CONFIGURE_TOTP` is a **default required action**, so any medewerker added later must
-  enrol an authenticator before the first login.
-
-See [../architecture/adr-0031-mfa-on-the-medewerker-realm.md](../architecture/adr-0031-mfa-on-the-medewerker-realm.md).
-
-### Getting a code
-
-```bash
-python3 infra/keycloak/check_realms.py otp   # prints a valid 6-digit code right now
-```
-
-Or enrol a phone once: the secret in base32 is `IJEUOTKFIRCVORKSJNCVET2UKBJUKRKE`
-(`otpauth://totp/medewerker?secret=IJEUOTKFIRCVORKSJNCVET2UKBJUKRKE`). The e2e computes its
-own code in `tests/e2e/medewerker-login.ts`.
-
-**Fixture only.** A shared, committed secret is a demo convenience, never a production
-posture — see the ADR's consequences.

docs/synthetic-data.md

@@ -19,11 +19,6 @@ All test users share the password **`test123`**.
 | `eidas` | eIDAS (EU) | `pierre-dupont` | `eidas_id` = `FR/NL/AB-1234-5678` |
 | `medewerker` | Internal staff | `merel-behandelaar` | role `behandelaar` |
 | `medewerker` | Internal staff | `tom-teamlead` | roles `behandelaar`, `teamlead` |
-| `medewerker` | Internal staff | `bram-beheerder` | role `beheerder` |
-
-`medewerker` users additionally need a **second factor**: that realm enforces MFA (S-15c,
-ADR-0031). All three share the fixture TOTP secret `BIGMEDEWERKEROTPSEED`; print a current
-code with `python3 infra/keycloak/check_realms.py otp`.
 
 The identifying claims are injected via OIDC protocol mappers on `big-portal`
 (user-attribute → token claim); `medewerker` roles appear in `realm_access.roles`.
@@ -37,8 +32,5 @@ curl -s -X POST \
   -d username=jan-burger -d password=test123 -d scope=openid | jq -r .access_token
 ```
 
-For a `medewerker` user, add `-d totp=$(python3 infra/keycloak/check_realms.py otp)` —
-without it the grant is refused with `invalid_grant`.
-
 Decode the JWT payload to see the `bsn` claim. `make keycloak-smoke` checks every realm
 automatically.

infra/keycloak/check_realms.py

@@ -1,25 +1,19 @@
 #!/usr/bin/env python3
 """Smoke-check the Keycloak realms: each realm's OIDC login works (password grant)
-and returns its expected identifying claim. The medewerker realm additionally enforces
-MFA (S-15c), so its login must be refused without a TOTP code. Stdlib only.
-Exits non-zero on failure.
+and returns its expected identifying claim. Stdlib only. Exits non-zero on failure.
 """
-import base64, hashlib, hmac, json, struct, sys, time, urllib.error, urllib.parse, urllib.request
+import base64, json, sys, urllib.error, urllib.parse, urllib.request
 
 BASE = "http://localhost:8180"
 CLIENT = "big-portal"
 PWD = "test123"
 
-# Fixture TOTP secret seeded into every medewerker in infra/keycloak/realms/medewerker-realm.json.
-# Keycloak HMACs the raw secret bytes, so no base32 decoding is involved.
-OTP_SECRET = b"BIGMEDEWERKEROTPSEED"
-
-# realm, user, claim ("__roles__" => check realm_access.roles), expected-contains, mfa-enforced
+# realm, user, claim ("__roles__" => check realm_access.roles), expected-contains
 CHECKS = [
-    ("digid", "jan-burger", "bsn", "123456782", False),
-    ("eherkenning", "acme-ondernemer", "kvk", "12345678", False),
-    ("eidas", "pierre-dupont", "eidas_id", "FR/NL", False),
-    ("medewerker", "merel-behandelaar", "__roles__", "behandelaar", True),
+    ("digid", "jan-burger", "bsn", "123456782"),
+    ("eherkenning", "acme-ondernemer", "kvk", "12345678"),
+    ("eidas", "pierre-dupont", "eidas_id", "FR/NL"),
+    ("medewerker", "merel-behandelaar", "__roles__", "behandelaar"),
 ]
 
 
@@ -29,17 +23,10 @@ def decode(jwt):
     return json.loads(base64.urlsafe_b64decode(p))
 
 
-def totp(secret=OTP_SECRET, period=30, digits=6):
-    """RFC 6238 code: HMAC-SHA1 over the 30-second counter, dynamically truncated."""
-    mac = hmac.new(secret, struct.pack(">Q", int(time.time()) // period), hashlib.sha1).digest()
-    o = mac[-1] & 0x0F
-    return str((struct.unpack(">I", mac[o:o + 4])[0] & 0x7FFFFFFF) % 10 ** digits).zfill(digits)
-
-
-def grant(realm, user, **extra):
+def grant(realm, user):
     data = urllib.parse.urlencode({
         "grant_type": "password", "client_id": CLIENT,
-        "username": user, "password": PWD, "scope": "openid", **extra,
+        "username": user, "password": PWD, "scope": "openid",
     }).encode()
     req = urllib.request.Request(
         f"{BASE}/realms/{realm}/protocol/openid-connect/token", data=data,
@@ -48,27 +35,11 @@ def grant(realm, user, **extra):
         return json.loads(r.read())
 
 
-def second_factor_refused(realm, user):
-    """The password alone must not yield a token on an MFA-enforced realm."""
-    try:
-        grant(realm, user)
-    except urllib.error.HTTPError as e:
-        return e.code in (400, 401)
-    return False
-
-
 def main():
     ok = True
-    for realm, user, claim, expect, mfa in CHECKS:
-        extra = {}
-        if mfa:
-            refused = second_factor_refused(realm, user)
-            ok = ok and refused
-            print(f"{realm:12} {user:18} password-only login refused "
-                  f"[{'OK' if refused else 'MFA NOT ENFORCED'}]")
-            extra = {"totp": totp()}
+    for realm, user, claim, expect in CHECKS:
         try:
-            at = decode(grant(realm, user, **extra)["access_token"])
+            at = decode(grant(realm, user)["access_token"])
             if claim == "__roles__":
                 val = at.get("realm_access", {}).get("roles", [])
                 good = expect in val
@@ -86,9 +57,4 @@ def main():
 
 
 if __name__ == "__main__":
-    # `check_realms.py otp` prints a current code for the fixture secret — what a human demoing
-    # the medewerker portals types at Keycloak's OTP prompt (docs/runbooks/keycloak.md).
-    if len(sys.argv) > 1 and sys.argv[1] == "otp":
-        print(totp())
-    else:
-        main()
+    main()

infra/keycloak/realms/medewerker-realm.json

@@ -2,16 +2,6 @@
   "realm": "medewerker",
   "enabled": true,
   "displayName": "Medewerkers",
-  "requiredActions": [
-    {
-      "alias": "CONFIGURE_TOTP",
-      "name": "Configure OTP",
-      "providerId": "CONFIGURE_TOTP",
-      "enabled": true,
-      "defaultAction": true,
-      "priority": 10
-    }
-  ],
   "roles": {
     "realm": [
       { "name": "behandelaar", "description": "Behandelt registratieaanvragen" },
@@ -53,15 +43,7 @@
       "lastName": "Behandelaar",
       "email": "merel@big.example.nl",
       "emailVerified": true,
-      "credentials": [
-        { "type": "password", "value": "test123", "temporary": false },
-        {
-          "type": "otp",
-          "userLabel": "seeded TOTP (fixture)",
-          "secretData": "{\"value\":\"BIGMEDEWERKEROTPSEED\"}",
-          "credentialData": "{\"subType\":\"totp\",\"digits\":6,\"counter\":0,\"period\":30,\"algorithm\":\"HmacSHA1\"}"
-        }
-      ],
+      "credentials": [{ "type": "password", "value": "test123", "temporary": false }],
       "realmRoles": ["behandelaar"]
     },
     {
@@ -71,15 +53,7 @@
       "lastName": "Teamlead",
       "email": "tom@big.example.nl",
       "emailVerified": true,
-      "credentials": [
-        { "type": "password", "value": "test123", "temporary": false },
-        {
-          "type": "otp",
-          "userLabel": "seeded TOTP (fixture)",
-          "secretData": "{\"value\":\"BIGMEDEWERKEROTPSEED\"}",
-          "credentialData": "{\"subType\":\"totp\",\"digits\":6,\"counter\":0,\"period\":30,\"algorithm\":\"HmacSHA1\"}"
-        }
-      ],
+      "credentials": [{ "type": "password", "value": "test123", "temporary": false }],
       "realmRoles": ["behandelaar", "teamlead"]
     },
     {
@@ -89,15 +63,7 @@
       "lastName": "Beheerder",
       "email": "bram@big.example.nl",
       "emailVerified": true,
-      "credentials": [
-        { "type": "password", "value": "test123", "temporary": false },
-        {
-          "type": "otp",
-          "userLabel": "seeded TOTP (fixture)",
-          "secretData": "{\"value\":\"BIGMEDEWERKEROTPSEED\"}",
-          "credentialData": "{\"subType\":\"totp\",\"digits\":6,\"counter\":0,\"period\":30,\"algorithm\":\"HmacSHA1\"}"
-        }
-      ],
+      "credentials": [{ "type": "password", "value": "test123", "temporary": false }],
       "realmRoles": ["beheerder"]
     }
   ]

tests/e2e/catalogus.spec.ts

@@ -1,5 +1,4 @@
 import { expect, test } from '@playwright/test';
-import { loginMedewerker } from './medewerker-login';
 
 // S-15a walking skeleton: a beheerder logs in to the beheer portal (medewerker realm) and sees the
 // read-only ZTC catalogus. The verify stack seeds and publishes the BIG-REGISTRATIE zaaktype (the
@@ -8,9 +7,10 @@ import { loginMedewerker } from './medewerker-login';
 test('a beheerder sees the published zaaktypen in the catalogus', async ({ page }) => {
   await page.goto('http://beheer/');
 
-  // The beheer portal redirects to the Keycloak medewerker realm login (same realm as behandel),
-  // which enforces MFA: password, then a TOTP code.
-  await loginMedewerker(page, 'bram-beheerder');
+  // The beheer portal redirects to the Keycloak medewerker realm login (same realm as behandel).
+  await page.locator('#username').fill('bram-beheerder');
+  await page.locator('#password').fill('test123');
+  await page.locator('#kc-login').click();
 
   await expect(page.getByRole('heading', { name: /Catalogus/i })).toBeVisible();
 

tests/e2e/default-fill.spec.ts

@@ -1,5 +1,4 @@
 import { expect, test } from '@playwright/test';
-import { loginMedewerker } from './medewerker-login';
 
 // S-15b: a beheerder edits the ACL default-fill in the beheer portal and gets a saved confirmation.
 // Runs against the shared verify stack; it edits + saves (the ACL store is in-memory, ADR-0026) and
@@ -7,8 +6,10 @@ import { loginMedewerker } from './medewerker-login';
 test('a beheerder edits and saves the default-fill', async ({ page }) => {
   await page.goto('http://beheer/');
 
-  // Keycloak medewerker-realm login (same realm as behandel) — password + enforced TOTP.
-  await loginMedewerker(page, 'bram-beheerder');
+  // Keycloak medewerker-realm login (same realm as behandel).
+  await page.locator('#username').fill('bram-beheerder');
+  await page.locator('#password').fill('test123');
+  await page.locator('#kc-login').click();
 
   await expect(page.getByRole('heading', { name: /Catalogus/i })).toBeVisible();
 

tests/e2e/medewerker-login.ts

@@ -1,27 +0,0 @@
-import { createHmac } from 'node:crypto';
-import type { Page } from '@playwright/test';
-
-// The medewerker realm enforces MFA (S-15c), so a staff login is two steps: password, then a TOTP
-// code. The realm export seeds every medewerker with this fixture secret — Keycloak HMACs the raw
-// secret bytes — so the e2e can compute a valid code instead of enrolling an authenticator.
-const OTP_SECRET = 'BIGMEDEWERKEROTPSEED';
-
-// RFC 6238 TOTP: HMAC-SHA1 over the 30-second counter, dynamically truncated to 6 digits.
-export function totp(secret = OTP_SECRET, at = Date.now()): string {
-  const counter = Buffer.alloc(8);
-  counter.writeBigUInt64BE(BigInt(Math.floor(at / 1000 / 30)));
-  const mac = createHmac('sha1', secret).update(counter).digest();
-  const offset = mac[mac.length - 1] & 0x0f;
-  return String((mac.readUInt32BE(offset) & 0x7fffffff) % 1_000_000).padStart(6, '0');
-}
-
-export async function loginMedewerker(page: Page, username: string): Promise<void> {
-  await page.locator('#username').fill(username);
-  await page.locator('#password').fill('test123');
-  await page.locator('#kc-login').click();
-
-  // Keycloak's conditional-OTP step. Its lookAheadWindow accepts the neighbouring counters, so a
-  // code computed just before a 30-second boundary still validates — no retry needed.
-  await page.locator('#otp').fill(totp());
-  await page.locator('#kc-login').click();
-}

tests/e2e/registration.spec.ts

@@ -1,5 +1,4 @@
 import { expect, request, test } from '@playwright/test';
-import { loginMedewerker } from './medewerker-login';
 
 // Walking-skeleton happy path (S-08d + S-09 + S-09b + S-12 + S-10a + S-19b-2): a zorgprofessional
 // logs in via mock DigiD and submits through the self-service portal → BFF → domain; the entry
@@ -77,8 +76,9 @@ test('DigiD submit → public INGEDIEND → documenten → behandelaar goedkeurt
   // — the S-12 flow that replaces the temporary admin endpoint. The staff tab switches to the
   // medewerker realm (a different Keycloak realm than the citizen's digid session).
   await staff.goto('http://behandel/');
-  // That realm enforces MFA (S-15c), so the behandelaar logs in with password + TOTP.
-  await loginMedewerker(staff, 'merel-behandelaar');
+  await staff.locator('#username').fill('merel-behandelaar');
+  await staff.locator('#password').fill('test123');
+  await staff.locator('#kc-login').click();
 
   await expect(staff.getByRole('heading', { name: /Werkbak/i })).toBeVisible();