fix(e2e): bound the Playwright run and make a failed login say why (closes #161) #165
@@ -71,8 +71,11 @@ build:
|
|||||||
|
|
||||||
## unit: run unit tests (excludes the container-backed Integration lane)
|
## unit: run unit tests (excludes the container-backed Integration lane)
|
||||||
# TRX per test project (→ TestResults/) feeds the CI per-service summary (#136); harmless locally.
|
# TRX per test project (→ TestResults/) feeds the CI per-service summary (#136); harmless locally.
|
||||||
|
# The CI reporting scripts are stdlib Python with their own assert-based self-checks (#161) — they
|
||||||
|
# ride this lane so a broken job summary is caught by CI rather than by the next red pipeline.
|
||||||
unit:
|
unit:
|
||||||
dotnet test $(SLN) -c Release --filter "Category!=Integration" --logger trx --results-directory TestResults
|
dotnet test $(SLN) -c Release --filter "Category!=Integration" --logger trx --results-directory TestResults
|
||||||
|
python3 infra/test_playwright_summary.py
|
||||||
|
|
||||||
## mutation: run the Stryker.NET ratchet on each service with branching logic (fails below baseline)
|
## mutation: run the Stryker.NET ratchet on each service with branching logic (fails below baseline)
|
||||||
# Stryker is pinned as a local dotnet tool (.config/dotnet-tools.json); `tool restore`
|
# Stryker is pinned as a local dotnet tool (.config/dotnet-tools.json); `tool restore`
|
||||||
|
|||||||
@@ -245,3 +245,47 @@ the verify-stack check table, and per-spec e2e results (`infra/playwright-summar
|
|||||||
- Getting a report out of the e2e container: Playwright writes `playwright-report.json`
|
- Getting a report out of the e2e container: Playwright writes `playwright-report.json`
|
||||||
inside the container; `infra/run-e2e-check.sh` `docker cp`s it back to the host
|
inside the container; `infra/run-e2e-check.sh` `docker cp`s it back to the host
|
||||||
(capturing the test exit code first) so the summary step can read it.
|
(capturing the test exit code first) so the summary step can read it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. `if: always()` does not survive the job being killed — bound the work itself
|
||||||
|
|
||||||
|
`if: always()` makes a step run when an *earlier step failed*. It does **not** help when
|
||||||
|
the job as a whole is stopped: the run's remaining steps are simply never dispatched.
|
||||||
|
|
||||||
|
That is how #161 lost its diagnosis. `verify-stack` entered `make verify-e2e` at 09:48:17
|
||||||
|
and the job ended at 10:14:54 — 26½ minutes later, mid-suite. Every step after the e2e
|
||||||
|
shows a **0-second `failure`** stamped at that same instant:
|
||||||
|
|
||||||
|
```
|
||||||
|
14 failure 09:48:17 -> 10:14:54 Self-service e2e (Playwright, login → submit → success)
|
||||||
|
15 failure 10:14:54 -> 10:14:54 verify-stack check summary ← if: always()
|
||||||
|
16 failure 10:14:54 -> 10:14:54 e2e spec summary ← if: always()
|
||||||
|
17 failure 10:14:54 -> 10:14:54 Dump container logs on failure ← if: failure()
|
||||||
|
18 failure 10:14:54 -> 10:14:54 Tear down ← if: always()
|
||||||
|
```
|
||||||
|
|
||||||
|
So the per-spec summary, the container-log dump and the teardown never ran, and the job
|
||||||
|
log — which also loses whatever the killed process had buffered — ended at a single `✘`
|
||||||
|
line. A job that dies takes its own post-mortem with it.
|
||||||
|
|
||||||
|
**Read the step timings, not just the log.** `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}`
|
||||||
|
returns every step with `started_at`/`completed_at`; a row of identical zero-length
|
||||||
|
steps at the end means *killed*, not *silent*. (Job ids come from
|
||||||
|
`…/actions/runs/{run}/jobs`, and that route returns only the **latest attempt** — a
|
||||||
|
re-run hides the failed one, so keep the failing job id from the original report. Logs:
|
||||||
|
`…/actions/jobs/{id}/logs`, see also `gitea-ci-logs`.)
|
||||||
|
|
||||||
|
**Conventions that follow:**
|
||||||
|
|
||||||
|
- **Bound long-running work inside the tool**, where it can still report. Playwright's
|
||||||
|
`globalTimeout` (`tests/e2e/playwright.config.ts`) ends the run, writes the JSON
|
||||||
|
report and exits, so the summary and log-dump steps still get their turn. A
|
||||||
|
`timeout-minutes` on the job would reproduce the very failure above.
|
||||||
|
- **Never let an auto-waiting action be the timeout.** Playwright actions (`fill`,
|
||||||
|
`click`) inherit the *test* timeout, not `expect.timeout`, so a missing element costs
|
||||||
|
the full 90 s and reports `locator.fill: Test timeout …` — the symptom. Assert the
|
||||||
|
element visible first with its own budget and a message (`tests/e2e/keycloak-login.ts`).
|
||||||
|
- Remember `concurrency.cancel-in-progress: true` in `ci.yaml`: a new push to the same
|
||||||
|
ref, or a re-run, kills the in-flight run the same way. Check `run_attempt` before
|
||||||
|
concluding a job hung.
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Overlay: make the CI compose stack usable from a HOST browser.
|
||||||
|
# Same two mechanisms infra/docker-compose.local.yml already uses — pin Keycloak's issuer to the
|
||||||
|
# host-published address, and point each portal's runtime config.json at it. The BFF needs no
|
||||||
|
# change: it discovers metadata over keycloak:8080 and the discovered issuer is the pinned
|
||||||
|
# localhost:8180, which is what browser tokens carry.
|
||||||
|
services:
|
||||||
|
keycloak:
|
||||||
|
environment:
|
||||||
|
KC_HOSTNAME: http://localhost:8180
|
||||||
|
KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
|
||||||
|
self-service:
|
||||||
|
volumes:
|
||||||
|
- ./local-config/self-service.config.json:/usr/share/nginx/html/config.json:ro,z
|
||||||
|
behandel:
|
||||||
|
volumes:
|
||||||
|
- ./local-config/behandel.config.json:/usr/share/nginx/html/config.json:ro,z
|
||||||
|
# beheer is the same medewerker realm as behandel, so it reuses behandel's config verbatim.
|
||||||
|
beheer:
|
||||||
|
volumes:
|
||||||
|
- ./local-config/behandel.config.json:/usr/share/nginx/html/config.json:ro,z
|
||||||
@@ -7,10 +7,34 @@ redirects it into $GITHUB_STEP_SUMMARY. Stdlib only.
|
|||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
STATUS_ICON = {"expected": "✅", "unexpected": "❌", "skipped": "⏭️", "flaky": "⚠️"}
|
STATUS_ICON = {"expected": "✅", "unexpected": "❌", "skipped": "⏭️", "flaky": "⚠️"}
|
||||||
|
|
||||||
|
# A verdict alone still costs a log dive, and a killed or truncated job leaves no log to dive into
|
||||||
|
# (#161) — so a failing spec carries its first error into the table. Playwright errors are multi-line
|
||||||
|
# with a "Call log:", which a markdown table cell cannot hold, so they are flattened and clipped.
|
||||||
|
ERROR_CLIP = 300
|
||||||
|
|
||||||
|
|
||||||
|
def first_error(spec):
|
||||||
|
"""The first error message across a spec's test results, flattened for one table cell."""
|
||||||
|
for test in spec.get("tests", []):
|
||||||
|
for result in test.get("results", []):
|
||||||
|
for error in result.get("errors", []):
|
||||||
|
message = (error.get("message") or "").strip()
|
||||||
|
if not message:
|
||||||
|
continue
|
||||||
|
# Strip ANSI colour, collapse to one line, and keep it inside the cell.
|
||||||
|
message = re.sub(r"\x1b\[[0-9;]*m", "", message)
|
||||||
|
message = " ".join(message.split())
|
||||||
|
if len(message) > ERROR_CLIP:
|
||||||
|
message = message[:ERROR_CLIP - 1].rstrip() + "…"
|
||||||
|
# `|` would end the cell early.
|
||||||
|
return message.replace("|", "\\|")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
def walk(suite, out):
|
def walk(suite, out):
|
||||||
for spec in suite.get("specs", []):
|
for spec in suite.get("specs", []):
|
||||||
@@ -22,7 +46,8 @@ def walk(suite, out):
|
|||||||
else "expected" if spec.get("ok", False)
|
else "expected" if spec.get("ok", False)
|
||||||
else "unexpected")
|
else "unexpected")
|
||||||
out.append({"file": spec.get("file") or suite.get("file") or suite.get("title", ""),
|
out.append({"file": spec.get("file") or suite.get("file") or suite.get("title", ""),
|
||||||
"title": spec.get("title", ""), "status": status})
|
"title": spec.get("title", ""), "status": status,
|
||||||
|
"error": first_error(spec) if status in ("unexpected", "flaky") else ""})
|
||||||
for child in suite.get("suites", []):
|
for child in suite.get("suites", []):
|
||||||
walk(child, out)
|
walk(child, out)
|
||||||
|
|
||||||
@@ -46,10 +71,17 @@ def main(path):
|
|||||||
if not specs:
|
if not specs:
|
||||||
print("_No specs ran._")
|
print("_No specs ran._")
|
||||||
return 0
|
return 0
|
||||||
print("| Spec | Result |")
|
# The failure column only earns its width when something failed.
|
||||||
print("| ---- | :----: |")
|
if any(s["error"] for s in specs):
|
||||||
for s in specs:
|
print("| Spec | Result | Why |")
|
||||||
print(f"| {s['file']} › {s['title']} | {STATUS_ICON.get(s['status'], '❔')} |")
|
print("| ---- | :----: | --- |")
|
||||||
|
for s in specs:
|
||||||
|
print(f"| {s['file']} › {s['title']} | {STATUS_ICON.get(s['status'], '❔')} | {s['error']} |")
|
||||||
|
else:
|
||||||
|
print("| Spec | Result |")
|
||||||
|
print("| ---- | :----: |")
|
||||||
|
for s in specs:
|
||||||
|
print(f"| {s['file']} › {s['title']} | {STATUS_ICON.get(s['status'], '❔')} |")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Self-check for infra/playwright-summary.py — stdlib asserts, no framework.
|
||||||
|
|
||||||
|
Run: python3 infra/test_playwright_summary.py (also runs in `make unit`).
|
||||||
|
|
||||||
|
A red e2e is only useful if the job summary says WHY it failed: #161 lost a 36-minute
|
||||||
|
verify-stack job whose only surviving output was one ✘ line with no assertion detail.
|
||||||
|
"""
|
||||||
|
import importlib.util
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
from contextlib import redirect_stdout
|
||||||
|
|
||||||
|
# The script's filename is not a valid module name, so load it by path.
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"playwright_summary",
|
||||||
|
os.path.join(os.path.dirname(os.path.abspath(__file__)), "playwright-summary.py"),
|
||||||
|
)
|
||||||
|
summary = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(summary)
|
||||||
|
|
||||||
|
|
||||||
|
def render(report):
|
||||||
|
"""Run the renderer over a report dict and return its markdown."""
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh:
|
||||||
|
json.dump(report, fh)
|
||||||
|
path = fh.name
|
||||||
|
try:
|
||||||
|
out = io.StringIO()
|
||||||
|
with redirect_stdout(out):
|
||||||
|
summary.main(path)
|
||||||
|
return out.getvalue()
|
||||||
|
finally:
|
||||||
|
os.unlink(path)
|
||||||
|
|
||||||
|
|
||||||
|
def spec_entry(title, status, errors=()):
|
||||||
|
return {
|
||||||
|
"title": title,
|
||||||
|
"file": "catalogus.spec.ts",
|
||||||
|
"ok": status == "expected",
|
||||||
|
"tests": [{"status": status, "results": [{"errors": [{"message": m} for m in errors]}]}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_failing_spec_reports_why():
|
||||||
|
md = render({
|
||||||
|
"stats": {"expected": 4, "unexpected": 1, "flaky": 0, "skipped": 0, "duration": 108_000},
|
||||||
|
"suites": [{"file": "catalogus.spec.ts", "specs": [
|
||||||
|
spec_entry("a beheerder sees the published zaaktypen in the catalogus", "unexpected",
|
||||||
|
["locator.fill: Test timeout of 90000ms exceeded.\n"
|
||||||
|
"Call log:\n - waiting for locator('#username')\n"]),
|
||||||
|
]}],
|
||||||
|
})
|
||||||
|
assert "❌" in md, md
|
||||||
|
# The point of the slice: the summary names the cause, not just the verdict.
|
||||||
|
assert "Test timeout of 90000ms exceeded" in md, md
|
||||||
|
assert "waiting for locator('#username')" in md, md
|
||||||
|
# A multi-line Playwright error must not break out of its table row.
|
||||||
|
assert not any(line.startswith("Call log:") for line in md.splitlines()), md
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_playwright_error_is_flattened():
|
||||||
|
# A real report's message is multi-line and ANSI-coloured, and embeds the source snippet with
|
||||||
|
# `|` gutters — all three would break the table cell. Shape verified against an actual
|
||||||
|
# @playwright/test 1.61 JSON report.
|
||||||
|
md = render({
|
||||||
|
"stats": {"expected": 0, "unexpected": 1, "flaky": 0, "skipped": 0, "duration": 1_000},
|
||||||
|
"suites": [{"file": "catalogus.spec.ts", "specs": [
|
||||||
|
spec_entry("a beheerder sees the catalogus", "unexpected",
|
||||||
|
["Error: expect(locator).toBeVisible() failed\n\n"
|
||||||
|
"\x1b[2mLocator: \x1b[22mgetByRole('heading')\n"
|
||||||
|
" 12 | await login(page);\n> 13 | await expect(heading).toBeVisible();\n"]),
|
||||||
|
]}],
|
||||||
|
})
|
||||||
|
row = [line for line in md.splitlines() if line.startswith("| catalogus.spec.ts")][0]
|
||||||
|
assert "\x1b" not in row, row
|
||||||
|
assert "Locator: getByRole('heading')" in row, row
|
||||||
|
# Every literal `|` from the snippet gutters is escaped, so the row keeps exactly 3 cells.
|
||||||
|
assert row.count("|") - row.count("\\|") == 4, row
|
||||||
|
|
||||||
|
|
||||||
|
def test_passing_run_stays_quiet():
|
||||||
|
md = render({
|
||||||
|
"stats": {"expected": 1, "unexpected": 0, "flaky": 0, "skipped": 0, "duration": 5_000},
|
||||||
|
"suites": [{"file": "catalogus.spec.ts",
|
||||||
|
"specs": [spec_entry("a beheerder sees the catalogus", "expected")]}],
|
||||||
|
})
|
||||||
|
assert "✅" in md, md
|
||||||
|
assert "timeout" not in md.lower(), md
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_report_is_not_a_crash():
|
||||||
|
out = io.StringIO()
|
||||||
|
with redirect_stdout(out):
|
||||||
|
rc = summary.main("/nonexistent/playwright-report.json")
|
||||||
|
assert rc == 0
|
||||||
|
assert "did not reach the e2e step" in out.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
for name, fn in sorted(globals().items()):
|
||||||
|
if name.startswith("test_") and callable(fn):
|
||||||
|
fn()
|
||||||
|
print(f" ok {name}")
|
||||||
|
print("playwright-summary self-check passed")
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
import { expect, test } from '@playwright/test';
|
import { expect, test } from '@playwright/test';
|
||||||
import { loginMedewerker } from './medewerker-login';
|
import { loginMedewerker } from './keycloak-login';
|
||||||
|
|
||||||
// S-15a walking skeleton: a beheerder logs in to the beheer portal (medewerker realm) and sees the
|
// S-15a walking skeleton: a beheerder logs in to the beheer portal (medewerker realm) and sees the
|
||||||
// read-only ZTC catalogus. The verify stack seeds and publishes the BIG-REGISTRATIE zaaktype (the
|
// read-only ZTC catalogus. The verify stack seeds and publishes the BIG-REGISTRATIE zaaktype (the
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { expect, test } from '@playwright/test';
|
import { expect, test } from '@playwright/test';
|
||||||
import { loginMedewerker } from './medewerker-login';
|
import { loginMedewerker } from './keycloak-login';
|
||||||
|
|
||||||
// S-15b: a beheerder edits the ACL default-fill in the beheer portal and gets a saved confirmation.
|
// S-15b: a beheerder edits the ACL default-fill in the beheer portal and gets a saved confirmation.
|
||||||
// Runs against the shared verify stack; it edits + saves (the ACL store is in-memory, ADR-0026) and
|
// Runs against the shared verify stack; it edits + saves (the ACL store is in-memory, ADR-0026) and
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { expect, test } from '@playwright/test';
|
import { expect, test } from '@playwright/test';
|
||||||
import { OTP_PERIOD_MS, nextUnusedCounter } from './medewerker-login';
|
import { OTP_PERIOD_MS, nextUnusedCounter } from './keycloak-login';
|
||||||
|
|
||||||
// Pure check of the TOTP counter guard in loginMedewerker — no browser, no stack. Keycloak refuses
|
// Pure check of the TOTP counter guard in loginMedewerker — no browser, no stack. Keycloak refuses
|
||||||
// a code it has already accepted (its otpPolicyCodeReusable defaults to false), so two logins as
|
// a code it has already accepted (its otpPolicyCodeReusable defaults to false), so two logins as
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
import { createHmac } from 'node:crypto';
|
||||||
|
import { readFileSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { expect, type Page } from '@playwright/test';
|
||||||
|
|
||||||
|
// Every portal login in the suite goes through this module — citizen realms (mock DigiD) and the
|
||||||
|
// medewerker realm alike — so the shared Keycloak form handling lives in exactly one place.
|
||||||
|
|
||||||
|
// The medewerker realm enforces MFA (S-15c), so a staff login is two steps: password, then a TOTP
|
||||||
|
// code. The realm export seeds every medewerker with this fixture secret — Keycloak HMACs the raw
|
||||||
|
// secret bytes — so the e2e can compute a valid code instead of enrolling an authenticator.
|
||||||
|
const OTP_SECRET = 'BIGMEDEWERKEROTPSEED';
|
||||||
|
|
||||||
|
export const OTP_PERIOD_MS = 30_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How long a Keycloak form gets to appear. Generous enough for a cold first browser launch and a
|
||||||
|
* loaded stack, far short of the 90-second test timeout an auto-waiting action would otherwise eat.
|
||||||
|
*/
|
||||||
|
const FORM_TIMEOUT_MS = 20_000;
|
||||||
|
const FORM_NEVER_APPEARED =
|
||||||
|
'the Keycloak login form never appeared — the portal did not reach Keycloak (check its ' +
|
||||||
|
'config.json fetch and the OIDC discovery on the authority it was built with)';
|
||||||
|
const OTP_NEVER_APPEARED =
|
||||||
|
'the Keycloak OTP form never appeared — the password step did not complete (check the ' +
|
||||||
|
'medewerker realm seeded this user with both a password and a TOTP credential)';
|
||||||
|
|
||||||
|
// RFC 6238 TOTP: HMAC-SHA1 over the 30-second counter, dynamically truncated to 6 digits.
|
||||||
|
export function totp(secret = OTP_SECRET, at = Date.now()): string {
|
||||||
|
const counter = Buffer.alloc(8);
|
||||||
|
counter.writeBigUInt64BE(BigInt(Math.floor(at / OTP_PERIOD_MS)));
|
||||||
|
const mac = createHmac('sha1', secret).update(counter).digest();
|
||||||
|
const offset = mac[mac.length - 1] & 0x0f;
|
||||||
|
return String((mac.readUInt32BE(offset) & 0x7fffffff) % 1_000_000).padStart(6, '0');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Keycloak refuses a TOTP code it has already accepted (its otpPolicyCodeReusable defaults to
|
||||||
|
// false), so two logins as the same medewerker inside one 30-second window would both submit the
|
||||||
|
// same code and the second is rejected. Spend the first counter this medewerker has left.
|
||||||
|
export function nextUnusedCounter(now: number, spent: number): number {
|
||||||
|
return Math.max(Math.floor(now / OTP_PERIOD_MS), spent + 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The spent counter lives on disk rather than in module state: Playwright starts a fresh worker
|
||||||
|
// process for a retry, which would otherwise forget it and resubmit the rejected code.
|
||||||
|
function spendCounter(username: string): number {
|
||||||
|
const file = join(tmpdir(), `otp-counter-${username}`);
|
||||||
|
let spent = -1;
|
||||||
|
try {
|
||||||
|
spent = Number(readFileSync(file, 'utf8')) || -1;
|
||||||
|
} catch {
|
||||||
|
// first login as this medewerker in this run
|
||||||
|
}
|
||||||
|
const counter = nextUnusedCounter(Date.now(), spent);
|
||||||
|
writeFileSync(file, String(counter));
|
||||||
|
return counter;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fill Keycloak's login form. Every portal is guarded, so the first navigation redirects here; the
|
||||||
|
* form ids are stable across themes.
|
||||||
|
*
|
||||||
|
* The form is asserted visible *before* it is filled. A portal that never reaches Keycloak — its
|
||||||
|
* runtime `config.json` fetch or the OIDC discovery behind `authorize()` failed, so it never
|
||||||
|
* bootstrapped and shows a blank page (main.ts only logs to the console) — would otherwise leave
|
||||||
|
* `fill()` auto-waiting until the whole test times out: 90 seconds spent to report
|
||||||
|
* `locator.fill: Test timeout of 90000ms exceeded`, naming the symptom and not the cause. That is
|
||||||
|
* how #161's catalogus.spec burned 1.8 minutes. This fails in a quarter of the time and says which
|
||||||
|
* step never happened.
|
||||||
|
*/
|
||||||
|
async function submitPassword(page: Page, username: string): Promise<void> {
|
||||||
|
await expect(page.locator('#username'), FORM_NEVER_APPEARED).toBeVisible({ timeout: FORM_TIMEOUT_MS });
|
||||||
|
await page.locator('#username').fill(username);
|
||||||
|
await page.locator('#password').fill('test123');
|
||||||
|
await page.locator('#kc-login').click();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A citizen login on a mock-DigiD realm — no second factor (ADR-0031). */
|
||||||
|
export async function loginBurger(page: Page, username: string): Promise<void> {
|
||||||
|
await submitPassword(page, username);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A staff login on the medewerker realm: password, then the enforced TOTP second factor. */
|
||||||
|
export async function loginMedewerker(page: Page, username: string): Promise<void> {
|
||||||
|
await submitPassword(page, username);
|
||||||
|
|
||||||
|
// Keycloak's conditional-OTP step. Same reasoning as the password form above: assert it arrived
|
||||||
|
// rather than letting `fill()` swallow the test timeout.
|
||||||
|
await expect(page.locator('#otp'), OTP_NEVER_APPEARED).toBeVisible({ timeout: FORM_TIMEOUT_MS });
|
||||||
|
|
||||||
|
// Wait out the rest of the window if the counter we may spend is still in the future; Keycloak's
|
||||||
|
// lookAheadWindow would accept the code a moment early, but only by one counter — waiting keeps a
|
||||||
|
// third login in the same window valid too.
|
||||||
|
const counter = spendCounter(username);
|
||||||
|
await page.waitForTimeout(Math.max(0, counter * OTP_PERIOD_MS - Date.now()));
|
||||||
|
await page.locator('#otp').fill(totp(OTP_SECRET, counter * OTP_PERIOD_MS));
|
||||||
|
await page.locator('#kc-login').click();
|
||||||
|
}
|
||||||
@@ -1,57 +0,0 @@
|
|||||||
import { createHmac } from 'node:crypto';
|
|
||||||
import { readFileSync, writeFileSync } from 'node:fs';
|
|
||||||
import { tmpdir } from 'node:os';
|
|
||||||
import { join } from 'node:path';
|
|
||||||
import type { Page } from '@playwright/test';
|
|
||||||
|
|
||||||
// The medewerker realm enforces MFA (S-15c), so a staff login is two steps: password, then a TOTP
|
|
||||||
// code. The realm export seeds every medewerker with this fixture secret — Keycloak HMACs the raw
|
|
||||||
// secret bytes — so the e2e can compute a valid code instead of enrolling an authenticator.
|
|
||||||
const OTP_SECRET = 'BIGMEDEWERKEROTPSEED';
|
|
||||||
|
|
||||||
export const OTP_PERIOD_MS = 30_000;
|
|
||||||
|
|
||||||
// RFC 6238 TOTP: HMAC-SHA1 over the 30-second counter, dynamically truncated to 6 digits.
|
|
||||||
export function totp(secret = OTP_SECRET, at = Date.now()): string {
|
|
||||||
const counter = Buffer.alloc(8);
|
|
||||||
counter.writeBigUInt64BE(BigInt(Math.floor(at / OTP_PERIOD_MS)));
|
|
||||||
const mac = createHmac('sha1', secret).update(counter).digest();
|
|
||||||
const offset = mac[mac.length - 1] & 0x0f;
|
|
||||||
return String((mac.readUInt32BE(offset) & 0x7fffffff) % 1_000_000).padStart(6, '0');
|
|
||||||
}
|
|
||||||
|
|
||||||
// Keycloak refuses a TOTP code it has already accepted (its otpPolicyCodeReusable defaults to
|
|
||||||
// false), so two logins as the same medewerker inside one 30-second window would both submit the
|
|
||||||
// same code and the second is rejected. Spend the first counter this medewerker has left.
|
|
||||||
export function nextUnusedCounter(now: number, spent: number): number {
|
|
||||||
return Math.max(Math.floor(now / OTP_PERIOD_MS), spent + 1);
|
|
||||||
}
|
|
||||||
|
|
||||||
// The spent counter lives on disk rather than in module state: Playwright starts a fresh worker
|
|
||||||
// process for a retry, which would otherwise forget it and resubmit the rejected code.
|
|
||||||
function spendCounter(username: string): number {
|
|
||||||
const file = join(tmpdir(), `otp-counter-${username}`);
|
|
||||||
let spent = -1;
|
|
||||||
try {
|
|
||||||
spent = Number(readFileSync(file, 'utf8')) || -1;
|
|
||||||
} catch {
|
|
||||||
// first login as this medewerker in this run
|
|
||||||
}
|
|
||||||
const counter = nextUnusedCounter(Date.now(), spent);
|
|
||||||
writeFileSync(file, String(counter));
|
|
||||||
return counter;
|
|
||||||
}
|
|
||||||
|
|
||||||
export async function loginMedewerker(page: Page, username: string): Promise<void> {
|
|
||||||
await page.locator('#username').fill(username);
|
|
||||||
await page.locator('#password').fill('test123');
|
|
||||||
await page.locator('#kc-login').click();
|
|
||||||
|
|
||||||
// Keycloak's conditional-OTP step. Wait out the rest of the window if the counter we may spend is
|
|
||||||
// still in the future; its lookAheadWindow would accept the code a moment early, but only by one
|
|
||||||
// counter — waiting keeps a third login in the same window valid too.
|
|
||||||
const counter = spendCounter(username);
|
|
||||||
await page.waitForTimeout(Math.max(0, counter * OTP_PERIOD_MS - Date.now()));
|
|
||||||
await page.locator('#otp').fill(totp(OTP_SECRET, counter * OTP_PERIOD_MS));
|
|
||||||
await page.locator('#kc-login').click();
|
|
||||||
}
|
|
||||||
@@ -15,6 +15,12 @@ export default defineConfig({
|
|||||||
timeout: 90_000,
|
timeout: 90_000,
|
||||||
expect: { timeout: 15_000 },
|
expect: { timeout: 15_000 },
|
||||||
retries: 1,
|
retries: 1,
|
||||||
|
// Bound the whole run, not just each test (#161). A wedged suite used to run until CI killed the
|
||||||
|
// job — which also killed the `if: always()` steps that would have said why: the per-spec summary
|
||||||
|
// and the container-log dump never ran, leaving a 36-minute job whose entire surviving output was
|
||||||
|
// one ✘ line. On `globalTimeout` Playwright stops and *reports*, so the JSON report is written and
|
||||||
|
// those steps still run. Generous over the ~1-minute suite: this is a backstop, not a budget.
|
||||||
|
globalTimeout: 12 * 60_000,
|
||||||
// Run the specs serially. Each spec drives a full `channel: 'chromium'` browser, and the e2e
|
// Run the specs serially. Each spec drives a full `channel: 'chromium'` browser, and the e2e
|
||||||
// shares an 8 GB runner with the entire compose stack (OpenZaak, NRC, Keycloak, Flowable, 4×
|
// shares an 8 GB runner with the entire compose stack (OpenZaak, NRC, Keycloak, Flowable, 4×
|
||||||
// Postgres, every service + 3 portals). Two parallel browsers exhaust memory and the renderer is
|
// Postgres, every service + 3 portals). Two parallel browsers exhaust memory and the renderer is
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { expect, request, test } from '@playwright/test';
|
import { expect, request, test } from '@playwright/test';
|
||||||
import { loginMedewerker } from './medewerker-login';
|
import { loginBurger, loginMedewerker } from './keycloak-login';
|
||||||
|
|
||||||
// Walking-skeleton happy path (S-08d + S-09 + S-09b + S-12 + S-10a + S-19b-2): a zorgprofessional
|
// Walking-skeleton happy path (S-08d + S-09 + S-09b + S-12 + S-10a + S-19b-2): a zorgprofessional
|
||||||
// logs in via mock DigiD and submits through the self-service portal → BFF → domain; the entry
|
// logs in via mock DigiD and submits through the self-service portal → BFF → domain; the entry
|
||||||
@@ -23,9 +23,7 @@ test('DigiD submit → public INGEDIEND → documenten → behandelaar goedkeurt
|
|||||||
// checks submit as jan-burger (bsn 123456782) before the e2e runs on the shared stack, and
|
// checks submit as jan-burger (bsn 123456782) before the e2e runs on the shared stack, and
|
||||||
// resume-on-load (S-26) would otherwise restore one of those on login — so each self-service spec
|
// resume-on-load (S-26) would otherwise restore one of those on login — so each self-service spec
|
||||||
// uses a dedicated citizen no other actor touches.
|
// uses a dedicated citizen no other actor touches.
|
||||||
await page.locator('#username').fill('emma-burger');
|
await loginBurger(page, 'emma-burger');
|
||||||
await page.locator('#password').fill('test123');
|
|
||||||
await page.locator('#kc-login').click();
|
|
||||||
|
|
||||||
// Back on the portal, authenticated.
|
// Back on the portal, authenticated.
|
||||||
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { expect, test } from '@playwright/test';
|
import { expect, test } from '@playwright/test';
|
||||||
|
import { loginBurger } from './keycloak-login';
|
||||||
|
|
||||||
// S-26: a zorgprofessional submits, then reloads the self-service portal. On load the portal asks the
|
// S-26: a zorgprofessional submits, then reloads the self-service portal. On load the portal asks the
|
||||||
// BFF for the caller's current open registration (owner-scoped by the DigiD token's bsn) and restores
|
// BFF for the caller's current open registration (owner-scoped by the DigiD token's bsn) and restores
|
||||||
@@ -9,9 +10,7 @@ test('DigiD submit → reload → self-service restores the existing registratio
|
|||||||
// Its own DigiD user (like every self-service spec): on the shared verify stack, resume-on-load
|
// Its own DigiD user (like every self-service spec): on the shared verify stack, resume-on-load
|
||||||
// (S-26) restores any open registration for the bsn, so each spec uses a dedicated citizen that no
|
// (S-26) restores any open registration for the bsn, so each spec uses a dedicated citizen that no
|
||||||
// other spec or verify-* check touches. This one in particular leaves an open registration.
|
// other spec or verify-* check touches. This one in particular leaves an open registration.
|
||||||
await page.locator('#username').fill('sanne-burger');
|
await loginBurger(page, 'sanne-burger');
|
||||||
await page.locator('#password').fill('test123');
|
|
||||||
await page.locator('#kc-login').click();
|
|
||||||
|
|
||||||
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
||||||
await page.getByRole('button', { name: /indienen/i }).click();
|
await page.getByRole('button', { name: /indienen/i }).click();
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { expect, test } from '@playwright/test';
|
import { expect, test } from '@playwright/test';
|
||||||
|
import { loginBurger } from './keycloak-login';
|
||||||
|
|
||||||
// S-11 (Flow 3): a zorgprofessional logs in via mock DigiD, submits a registration, then withdraws
|
// S-11 (Flow 3): a zorgprofessional logs in via mock DigiD, submits a registration, then withdraws
|
||||||
// it ("trek aanvraag in") from the self-service portal. The withdrawal goes portal → BFF (owner-
|
// it ("trek aanvraag in") from the self-service portal. The withdrawal goes portal → BFF (owner-
|
||||||
@@ -10,9 +11,7 @@ test('DigiD submit → trek aanvraag in → self-service confirms ingetrokken',
|
|||||||
|
|
||||||
// Its own DigiD user — isolated from the verify-* checks (jan-burger/123456782) so resume-on-load
|
// Its own DigiD user — isolated from the verify-* checks (jan-burger/123456782) so resume-on-load
|
||||||
// (S-26) can't restore someone else's registration on the shared stack.
|
// (S-26) can't restore someone else's registration on the shared stack.
|
||||||
await page.locator('#username').fill('lars-burger');
|
await loginBurger(page, 'lars-burger');
|
||||||
await page.locator('#password').fill('test123');
|
|
||||||
await page.locator('#kc-login').click();
|
|
||||||
|
|
||||||
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user