not
d6b3f9764f
## What & why #161 is really two defects, and the second one is why the first was undiagnosable. **A wedged suite consumed the job, and took the post-mortem with it.** Nothing bounded the Playwright run, so CI stopped the job mid-suite — and `if: always()` does not survive that. Run 739's job metadata shows every step after the e2e as a **0-second failure** stamped at the kill: ``` 14 failure 09:48:17 -> 10:14:54 Self-service e2e (Playwright …) 15 failure 10:14:54 -> 10:14:54 verify-stack check summary ← if: always() 16 failure 10:14:54 -> 10:14:54 e2e spec summary ← if: always() 17 failure 10:14:54 -> 10:14:54 Dump container logs on failure ← if: failure() 18 failure 10:14:54 -> 10:14:54 Tear down ← if: always() ``` So the per-spec summary, the container-log dump and the teardown never ran, and the log lost whatever the killed process had buffered — leaving the single `✘` line the issue was filed from. `globalTimeout` now makes Playwright stop and *report*: the JSON report is written and those steps still get their turn. (A `timeout-minutes` on the job would have reproduced the same failure, so there isn't one.) The "~24-minute gap" is that kill, not necessarily a hang — note run 739 shows `run_attempt: 2`, and `concurrency.cancel-in-progress` kills an in-flight run on any re-run or push. **A login that never got its form ate the 90-second test timeout.** Playwright actions auto-wait until the *test* timeout, not `expect.timeout` — so a portal that serves its page but never bootstraps (its `config.json` fetch or the OIDC discovery behind `authorize()` failed; `main.ts` only `console.error`s) spent 90s to report `locator.fill: Test timeout of 90000ms exceeded`: the symptom, not the cause. That is catalogus.spec's 1.8 minutes. Both Keycloak forms are now asserted visible first, with a 20s budget and a message naming the step that never happened. Verified against a real blank-bootstrap portal — the beheer image served with a `config.json` that is not JSON — which fails in **20.2s** with *"the Keycloak login form never appeared — the portal did not reach Keycloak (check its config.json fetch and the OIDC discovery …)"*. **And the summary now says why.** The per-spec table (#136) rendered a verdict icon and nothing else, so even a surviving summary cost a log dive. Failing specs now carry their first error, flattened for a table cell (ANSI stripped, newlines collapsed, `|` escaped, clipped) — shape verified against a real @playwright/test 1.61 failing report, with a stdlib assert self-check on `make unit`. Closes #161 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing test committed before the implementation. - [x] Implementation makes the test pass; refactor commit follows (login helper dedup). - [x] Conventional Commits referencing the issue (`refs #161`). - [ ] CI green — all Gitea Actions jobs. - [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (untouched). - [x] Docs updated — `docs/runbooks/gitea-actions-gotchas.md` §9. - [x] ADR — not needed: no boundary, dependency or coupling rule touched (test/CI infra only). - [x] Demo note — not applicable: nothing user-visible. ## Notes for reviewers **What this does not do: identify why the beheerder login failed that once.** The evidence to do that was destroyed by defect 2, which is what this PR fixes. The suite ran green here five times today (catalogus.spec 1.1–5.3s each) — but a local box is not the loaded CI runner, so that is weak evidence and I am not claiming the flake is gone. What changes is that the next occurrence is bounded and self-describing: it fails in 20s naming the failing step, the JSON report survives, and the summary prints the error. Please keep #161 in mind rather than treating this as proof. **Two follow-ups I did not pull into this PR:** - *All four portals show a permanently blank page if their startup fetch fails* — `main.ts` does `fetch('config.json').then(bootstrap).catch(console.error)`, one shot, no UI and no recovery. That is a real product gap (the deliberately-broken portal above is exactly what a user would see) and wants its own slice, not a test-infra PR. - `retries: 1` is untouched. CLAUDE.md §15 says flaky tests are fixed rather than retried, but removing retries while a real flake is unexplained would trade a rare red for a frequent one. Worth revisiting once #161 recurs (or doesn't) with the new diagnostics. The login-helper rename (`medewerker-login.ts` → `keycloak-login.ts`, citizen logins routed through `loginBurger`) is its own no-behaviour-change commit: the three citizen specs each duplicated the same three-line login, so guarding the login path once meant routing them through it first.Reviewed-on: #165
90 lines
3.8 KiB
Python
90 lines
3.8 KiB
Python
#!/usr/bin/env python3
|
||
"""Render a per-spec table from a Playwright JSON report for a Gitea job summary (#136).
|
||
|
||
Reads the JSON report (default: tests/e2e/playwright-report.json) that run-e2e-check.sh copies out
|
||
of the e2e container, and prints a markdown table (one row per spec) to stdout. The CI step
|
||
redirects it into $GITHUB_STEP_SUMMARY. Stdlib only.
|
||
"""
|
||
import json
|
||
import os
|
||
import re
|
||
import sys
|
||
|
||
STATUS_ICON = {"expected": "✅", "unexpected": "❌", "skipped": "⏭️", "flaky": "⚠️"}
|
||
|
||
# A verdict alone still costs a log dive, and a killed or truncated job leaves no log to dive into
|
||
# (#161) — so a failing spec carries its first error into the table. Playwright errors are multi-line
|
||
# with a "Call log:", which a markdown table cell cannot hold, so they are flattened and clipped.
|
||
ERROR_CLIP = 300
|
||
|
||
|
||
def first_error(spec):
|
||
"""The first error message across a spec's test results, flattened for one table cell."""
|
||
for test in spec.get("tests", []):
|
||
for result in test.get("results", []):
|
||
for error in result.get("errors", []):
|
||
message = (error.get("message") or "").strip()
|
||
if not message:
|
||
continue
|
||
# Strip ANSI colour, collapse to one line, and keep it inside the cell.
|
||
message = re.sub(r"\x1b\[[0-9;]*m", "", message)
|
||
message = " ".join(message.split())
|
||
if len(message) > ERROR_CLIP:
|
||
message = message[:ERROR_CLIP - 1].rstrip() + "…"
|
||
# `|` would end the cell early.
|
||
return message.replace("|", "\\|")
|
||
return ""
|
||
|
||
|
||
def walk(suite, out):
|
||
for spec in suite.get("specs", []):
|
||
# A spec's status is carried on its test(s): expected/unexpected/skipped/flaky.
|
||
statuses = [t.get("status") for t in spec.get("tests", [])]
|
||
status = ("unexpected" if "unexpected" in statuses
|
||
else "flaky" if "flaky" in statuses
|
||
else "skipped" if statuses and all(s == "skipped" for s in statuses)
|
||
else "expected" if spec.get("ok", False)
|
||
else "unexpected")
|
||
out.append({"file": spec.get("file") or suite.get("file") or suite.get("title", ""),
|
||
"title": spec.get("title", ""), "status": status,
|
||
"error": first_error(spec) if status in ("unexpected", "flaky") else ""})
|
||
for child in suite.get("suites", []):
|
||
walk(child, out)
|
||
|
||
|
||
def main(path):
|
||
if not os.path.exists(path):
|
||
print("## 🎭 e2e (Playwright)\n\n_No e2e report — the run did not reach the e2e step._")
|
||
return 0
|
||
with open(path) as fh:
|
||
report = json.load(fh)
|
||
specs = []
|
||
for suite in report.get("suites", []):
|
||
walk(suite, specs)
|
||
|
||
print("## 🎭 e2e (Playwright)\n")
|
||
stats = report.get("stats", {})
|
||
if stats:
|
||
print(f"**{stats.get('expected', 0)} passed · {stats.get('unexpected', 0)} failed · "
|
||
f"{stats.get('flaky', 0)} flaky · {stats.get('skipped', 0)} skipped** "
|
||
f"({round(stats.get('duration', 0) / 1000)}s)\n")
|
||
if not specs:
|
||
print("_No specs ran._")
|
||
return 0
|
||
# The failure column only earns its width when something failed.
|
||
if any(s["error"] for s in specs):
|
||
print("| Spec | Result | Why |")
|
||
print("| ---- | :----: | --- |")
|
||
for s in specs:
|
||
print(f"| {s['file']} › {s['title']} | {STATUS_ICON.get(s['status'], '❔')} | {s['error']} |")
|
||
else:
|
||
print("| Spec | Result |")
|
||
print("| ---- | :----: |")
|
||
for s in specs:
|
||
print(f"| {s['file']} › {s['title']} | {STATUS_ICON.get(s['status'], '❔')} |")
|
||
return 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "tests/e2e/playwright-report.json"))
|