fix(e2e): bound the Playwright run and make a failed login say why (closes #161) (#165)
CI / build (push) Successful in 1m9s
CI / lint (push) Successful in 1m27s
CI / unit (push) Successful in 1m32s
CI / frontend (push) Successful in 3m19s
CI / mutation (push) Successful in 6m24s
CI / verify-stack (push) Successful in 9m33s

## What & why

#161 is really two defects, and the second one is why the first was undiagnosable.

**A wedged suite consumed the job, and took the post-mortem with it.** Nothing bounded the
Playwright run, so CI stopped the job mid-suite — and `if: always()` does not survive that. Run
739's job metadata shows every step after the e2e as a **0-second failure** stamped at the kill:

```
14 failure  09:48:17 -> 10:14:54  Self-service e2e (Playwright …)
15 failure  10:14:54 -> 10:14:54  verify-stack check summary       ← if: always()
16 failure  10:14:54 -> 10:14:54  e2e spec summary                 ← if: always()
17 failure  10:14:54 -> 10:14:54  Dump container logs on failure   ← if: failure()
18 failure  10:14:54 -> 10:14:54  Tear down                        ← if: always()
```

So the per-spec summary, the container-log dump and the teardown never ran, and the log lost
whatever the killed process had buffered — leaving the single `✘` line the issue was filed from.
`globalTimeout` now makes Playwright stop and *report*: the JSON report is written and those steps
still get their turn. (A `timeout-minutes` on the job would have reproduced the same failure, so
there isn't one.) The "~24-minute gap" is that kill, not necessarily a hang — note run 739 shows
`run_attempt: 2`, and `concurrency.cancel-in-progress` kills an in-flight run on any re-run or push.

**A login that never got its form ate the 90-second test timeout.** Playwright actions auto-wait
until the *test* timeout, not `expect.timeout` — so a portal that serves its page but never
bootstraps (its `config.json` fetch or the OIDC discovery behind `authorize()` failed; `main.ts`
only `console.error`s) spent 90s to report `locator.fill: Test timeout of 90000ms exceeded`: the
symptom, not the cause. That is catalogus.spec's 1.8 minutes. Both Keycloak forms are now asserted
visible first, with a 20s budget and a message naming the step that never happened.

Verified against a real blank-bootstrap portal — the beheer image served with a `config.json` that
is not JSON — which fails in **20.2s** with *"the Keycloak login form never appeared — the portal
did not reach Keycloak (check its config.json fetch and the OIDC discovery …)"*.

**And the summary now says why.** The per-spec table (#136) rendered a verdict icon and nothing
else, so even a surviving summary cost a log dive. Failing specs now carry their first error,
flattened for a table cell (ANSI stripped, newlines collapsed, `|` escaped, clipped) — shape
verified against a real @playwright/test 1.61 failing report, with a stdlib assert self-check on
`make unit`.

Closes #161

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation.
- [x] Implementation makes the test pass; refactor commit follows (login helper dedup).
- [x] Conventional Commits referencing the issue (`refs #161`).
- [ ] CI green — all Gitea Actions jobs.
- [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (untouched).
- [x] Docs updated — `docs/runbooks/gitea-actions-gotchas.md` §9.
- [x] ADR — not needed: no boundary, dependency or coupling rule touched (test/CI infra only).
- [x] Demo note — not applicable: nothing user-visible.

## Notes for reviewers

**What this does not do: identify why the beheerder login failed that once.** The evidence to do
that was destroyed by defect 2, which is what this PR fixes. The suite ran green here five times
today (catalogus.spec 1.1–5.3s each) — but a local box is not the loaded CI runner, so that is weak
evidence and I am not claiming the flake is gone. What changes is that the next occurrence is
bounded and self-describing: it fails in 20s naming the failing step, the JSON report survives, and
the summary prints the error. Please keep #161 in mind rather than treating this as proof.

**Two follow-ups I did not pull into this PR:**
- *All four portals show a permanently blank page if their startup fetch fails* — `main.ts` does
  `fetch('config.json').then(bootstrap).catch(console.error)`, one shot, no UI and no recovery. That
  is a real product gap (the deliberately-broken portal above is exactly what a user would see) and
  wants its own slice, not a test-infra PR.
- `retries: 1` is untouched. CLAUDE.md §15 says flaky tests are fixed rather than retried, but
  removing retries while a real flake is unexplained would trade a rare red for a frequent one.
  Worth revisiting once #161 recurs (or doesn't) with the new diagnostics.

The login-helper rename (`medewerker-login.ts` → `keycloak-login.ts`, citizen logins routed through
`loginBurger`) is its own no-behaviour-change commit: the three citizen specs each duplicated the
same three-line login, so guarding the login path once meant routing them through it first.Reviewed-on: #165
This commit was merged in pull request #165.
This commit is contained in:
not
2026-09-04 10:53:35 +00:00
parent 8b206a005f
commit d6b3f9764f
14 changed files with 326 additions and 75 deletions
+44
View File
@@ -245,3 +245,47 @@ the verify-stack check table, and per-spec e2e results (`infra/playwright-summar
- Getting a report out of the e2e container: Playwright writes `playwright-report.json`
inside the container; `infra/run-e2e-check.sh` `docker cp`s it back to the host
(capturing the test exit code first) so the summary step can read it.
---
## 9. `if: always()` does not survive the job being killed — bound the work itself
`if: always()` makes a step run when an *earlier step failed*. It does **not** help when
the job as a whole is stopped: the run's remaining steps are simply never dispatched.
That is how #161 lost its diagnosis. `verify-stack` entered `make verify-e2e` at 09:48:17
and the job ended at 10:14:54 — 26½ minutes later, mid-suite. Every step after the e2e
shows a **0-second `failure`** stamped at that same instant:
```
14 failure 09:48:17 -> 10:14:54 Self-service e2e (Playwright, login → submit → success)
15 failure 10:14:54 -> 10:14:54 verify-stack check summary ← if: always()
16 failure 10:14:54 -> 10:14:54 e2e spec summary ← if: always()
17 failure 10:14:54 -> 10:14:54 Dump container logs on failure ← if: failure()
18 failure 10:14:54 -> 10:14:54 Tear down ← if: always()
```
So the per-spec summary, the container-log dump and the teardown never ran, and the job
log — which also loses whatever the killed process had buffered — ended at a single `✘`
line. A job that dies takes its own post-mortem with it.
**Read the step timings, not just the log.** `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}`
returns every step with `started_at`/`completed_at`; a row of identical zero-length
steps at the end means *killed*, not *silent*. (Job ids come from
`…/actions/runs/{run}/jobs`, and that route returns only the **latest attempt** — a
re-run hides the failed one, so keep the failing job id from the original report. Logs:
`…/actions/jobs/{id}/logs`, see also `gitea-ci-logs`.)
**Conventions that follow:**
- **Bound long-running work inside the tool**, where it can still report. Playwright's
`globalTimeout` (`tests/e2e/playwright.config.ts`) ends the run, writes the JSON
report and exits, so the summary and log-dump steps still get their turn. A
`timeout-minutes` on the job would reproduce the very failure above.
- **Never let an auto-waiting action be the timeout.** Playwright actions (`fill`,
`click`) inherit the *test* timeout, not `expect.timeout`, so a missing element costs
the full 90 s and reports `locator.fill: Test timeout …` — the symptom. Assert the
element visible first with its own budget and a message (`tests/e2e/keycloak-login.ts`).
- Remember `concurrency.cancel-in-progress: true` in `ci.yaml`: a new push to the same
ref, or a re-run, kills the in-flight run the same way. Check `run_attempt` before
concluding a job hung.