not
d6b3f9764f
## What & why #161 is really two defects, and the second one is why the first was undiagnosable. **A wedged suite consumed the job, and took the post-mortem with it.** Nothing bounded the Playwright run, so CI stopped the job mid-suite — and `if: always()` does not survive that. Run 739's job metadata shows every step after the e2e as a **0-second failure** stamped at the kill: ``` 14 failure 09:48:17 -> 10:14:54 Self-service e2e (Playwright …) 15 failure 10:14:54 -> 10:14:54 verify-stack check summary ← if: always() 16 failure 10:14:54 -> 10:14:54 e2e spec summary ← if: always() 17 failure 10:14:54 -> 10:14:54 Dump container logs on failure ← if: failure() 18 failure 10:14:54 -> 10:14:54 Tear down ← if: always() ``` So the per-spec summary, the container-log dump and the teardown never ran, and the log lost whatever the killed process had buffered — leaving the single `✘` line the issue was filed from. `globalTimeout` now makes Playwright stop and *report*: the JSON report is written and those steps still get their turn. (A `timeout-minutes` on the job would have reproduced the same failure, so there isn't one.) The "~24-minute gap" is that kill, not necessarily a hang — note run 739 shows `run_attempt: 2`, and `concurrency.cancel-in-progress` kills an in-flight run on any re-run or push. **A login that never got its form ate the 90-second test timeout.** Playwright actions auto-wait until the *test* timeout, not `expect.timeout` — so a portal that serves its page but never bootstraps (its `config.json` fetch or the OIDC discovery behind `authorize()` failed; `main.ts` only `console.error`s) spent 90s to report `locator.fill: Test timeout of 90000ms exceeded`: the symptom, not the cause. That is catalogus.spec's 1.8 minutes. Both Keycloak forms are now asserted visible first, with a 20s budget and a message naming the step that never happened. Verified against a real blank-bootstrap portal — the beheer image served with a `config.json` that is not JSON — which fails in **20.2s** with *"the Keycloak login form never appeared — the portal did not reach Keycloak (check its config.json fetch and the OIDC discovery …)"*. **And the summary now says why.** The per-spec table (#136) rendered a verdict icon and nothing else, so even a surviving summary cost a log dive. Failing specs now carry their first error, flattened for a table cell (ANSI stripped, newlines collapsed, `|` escaped, clipped) — shape verified against a real @playwright/test 1.61 failing report, with a stdlib assert self-check on `make unit`. Closes #161 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing test committed before the implementation. - [x] Implementation makes the test pass; refactor commit follows (login helper dedup). - [x] Conventional Commits referencing the issue (`refs #161`). - [ ] CI green — all Gitea Actions jobs. - [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (untouched). - [x] Docs updated — `docs/runbooks/gitea-actions-gotchas.md` §9. - [x] ADR — not needed: no boundary, dependency or coupling rule touched (test/CI infra only). - [x] Demo note — not applicable: nothing user-visible. ## Notes for reviewers **What this does not do: identify why the beheerder login failed that once.** The evidence to do that was destroyed by defect 2, which is what this PR fixes. The suite ran green here five times today (catalogus.spec 1.1–5.3s each) — but a local box is not the loaded CI runner, so that is weak evidence and I am not claiming the flake is gone. What changes is that the next occurrence is bounded and self-describing: it fails in 20s naming the failing step, the JSON report survives, and the summary prints the error. Please keep #161 in mind rather than treating this as proof. **Two follow-ups I did not pull into this PR:** - *All four portals show a permanently blank page if their startup fetch fails* — `main.ts` does `fetch('config.json').then(bootstrap).catch(console.error)`, one shot, no UI and no recovery. That is a real product gap (the deliberately-broken portal above is exactly what a user would see) and wants its own slice, not a test-infra PR. - `retries: 1` is untouched. CLAUDE.md §15 says flaky tests are fixed rather than retried, but removing retries while a real flake is unexplained would trade a rare red for a frequent one. Worth revisiting once #161 recurs (or doesn't) with the new diagnostics. The login-helper rename (`medewerker-login.ts` → `keycloak-login.ts`, citizen logins routed through `loginBurger`) is its own no-behaviour-change commit: the three citizen specs each duplicated the same three-line login, so guarding the login path once meant routing them through it first.Reviewed-on: #165
335 lines
16 KiB
Makefile
335 lines
16 KiB
Makefile
# Developer + CI entrypoints.
|
|
#
|
|
# These targets are the single source of truth for the checks. The Gitea
|
|
# Actions workflow (.gitea/workflows/ci.yaml) invokes the SAME targets, so
|
|
# `make ci` locally runs exactly what the pipeline runs — no drift. Until a
|
|
# self-hosted runner is registered, `make ci` is the gate (see docs/runbooks/ci.md).
|
|
|
|
SLN := register-referentie.slnx
|
|
COMPOSE := infra/docker-compose.yml
|
|
# Long-running services with a healthcheck — the smoke polls these for readiness
|
|
# (infra/wait-healthy.sh). One-shot init jobs (oz-init, nrc-init, flowable-init)
|
|
# are not polled; they only need to have run. See docs/runbooks/gitea-actions-gotchas.md.
|
|
WAIT_SVCS := openzaak nrc-web acl bff domain event-subscriber projection-api self-service openbaar behandel beheer objecttypen objecten
|
|
# Config files (OpenZaak data.yaml, Keycloak realms, Flowable BPMN) are streamed
|
|
# into external named volumes via `docker cp` (infra/seed-config.sh) instead of
|
|
# bind-mounted, because bind mounts don't reach sibling containers on the
|
|
# containerized CI runner. SEED populates them; run it before every `up`. The
|
|
# volumes are `external`, so compose won't remove them — CFG_VOLS lists them for
|
|
# explicit teardown. See docs/runbooks/gitea-actions-gotchas.md.
|
|
SEED := bash infra/seed-config.sh
|
|
CFG_VOLS := rr-oz-config rr-nrc-config rr-kc-realms rr-fl-bpmn rr-objecttypen-config rr-objecten-config rr-registerrecord-config
|
|
# Local-only stack: same services but config is bind-mounted (no seed step), so a
|
|
# plain `docker compose -f infra/docker-compose.local.yml up` works on any local
|
|
# engine. This is the no-make / Windows-friendly path. See that file's header.
|
|
LOCAL_COMPOSE := infra/docker-compose.local.yml
|
|
OZ_COMPOSE := infra/openzaak/docker-compose.yml
|
|
OZ_BASE := http://localhost:8000
|
|
NRC_COMPOSE := infra/opennotificaties/docker-compose.yml
|
|
NRC_BASE := http://localhost:8001
|
|
KC_COMPOSE := infra/keycloak/docker-compose.yml
|
|
KC_BASE := http://localhost:8180
|
|
FL_COMPOSE := infra/flowable/docker-compose.yml
|
|
FL_BASE := http://localhost:8090/flowable-rest/service
|
|
STACK_FILES := -f $(OZ_COMPOSE) -f $(NRC_COMPOSE)
|
|
|
|
# On a rootless Podman dev box, point Docker CLI/Compose at the Podman socket —
|
|
# but only if that socket exists and DOCKER_HOST isn't already set, so real
|
|
# Docker hosts and CI runners are left untouched.
|
|
PODMAN_SOCK := /run/user/$(shell id -u)/podman/podman.sock
|
|
ifeq ($(wildcard $(PODMAN_SOCK)),$(PODMAN_SOCK))
|
|
ifeq ($(origin DOCKER_HOST),undefined)
|
|
export DOCKER_HOST := unix://$(PODMAN_SOCK)
|
|
endif
|
|
endif
|
|
|
|
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down help
|
|
|
|
## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions)
|
|
## `verify` is the live-stack stage (full stack up once → ACL + notification checks).
|
|
ci: lint build unit mutation frontend verify
|
|
|
|
## frontend: install deps and run the Nx lint/test/build for the portals (pnpm + Node required)
|
|
# Tests run in their own phase, ahead of the build. The @angular/build:unit-test
|
|
# (Vitest) runner spawns a worker with a hard-coded 60s/90s startup timeout that is
|
|
# not configurable. When the ~5min production build shares the run-many pool, it
|
|
# starves that worker of CPU on constrained CI runners and Vitest fails with
|
|
# "Timeout waiting for worker to respond". Splitting the phases keeps tests off the
|
|
# heavy build's back so the worker starts well inside its window.
|
|
frontend:
|
|
pnpm install --frozen-lockfile
|
|
pnpm nx run-many -t lint test
|
|
pnpm nx run-many -t build
|
|
|
|
## lint: verify formatting (no changes)
|
|
lint:
|
|
dotnet format $(SLN) --verify-no-changes
|
|
|
|
## build: release build
|
|
build:
|
|
dotnet build $(SLN) -c Release
|
|
|
|
## unit: run unit tests (excludes the container-backed Integration lane)
|
|
# TRX per test project (→ TestResults/) feeds the CI per-service summary (#136); harmless locally.
|
|
# The CI reporting scripts are stdlib Python with their own assert-based self-checks (#161) — they
|
|
# ride this lane so a broken job summary is caught by CI rather than by the next red pipeline.
|
|
unit:
|
|
dotnet test $(SLN) -c Release --filter "Category!=Integration" --logger trx --results-directory TestResults
|
|
python3 infra/test_playwright_summary.py
|
|
|
|
## mutation: run the Stryker.NET ratchet on each service with branching logic (fails below baseline)
|
|
# Stryker is pinned as a local dotnet tool (.config/dotnet-tools.json); `tool restore`
|
|
# makes `make mutation` work from a fresh clone. Each service owns its config + break
|
|
# threshold (the ratchet, CLAUDE.md §5): each services/<svc>/stryker-config.json.
|
|
# Scores never regress below baseline.
|
|
mutation:
|
|
dotnet tool restore
|
|
cd services/acl && dotnet stryker
|
|
cd services/event-subscriber && dotnet stryker
|
|
cd services/domain && dotnet stryker
|
|
cd services/bff && dotnet stryker
|
|
|
|
## smoke: seed config, bring the whole stack up, wait for health-checked services, tear down
|
|
# SEED populates the external config volumes first (upstream images used verbatim;
|
|
# only our acl/bff are built). `up -d --build` starts EVERYTHING. Readiness is
|
|
# checked by infra/wait-healthy.sh polling the durable, health-checked services
|
|
# ($(WAIT_SVCS)) via `docker inspect` — portable across docker compose and
|
|
# podman-compose, and needing no `--wait` flag or host port access. The one-shots
|
|
# (oz-init, flowable-init) aren't polled; they just need to have run.
|
|
smoke:
|
|
$(SEED) oz nrc kc fl objecttypen objecten registerrecord
|
|
docker compose -f $(COMPOSE) up -d --build
|
|
bash -c 'WAIT_TIMEOUT=420 bash infra/wait-healthy.sh $(WAIT_SVCS); rc=$$?; docker compose -f $(COMPOSE) down --volumes; docker volume rm -f $(CFG_VOLS) >/dev/null 2>&1; exit $$rc'
|
|
|
|
## up: seed config volumes and start the full stack (use instead of bare
|
|
## `docker compose up`, which can't self-seed the external config volumes)
|
|
up:
|
|
$(SEED) oz nrc kc fl objecttypen objecten registerrecord
|
|
docker compose -f $(COMPOSE) up -d --build
|
|
|
|
## down: stop and remove the local stack (incl. the external config volumes)
|
|
down:
|
|
docker compose -f $(COMPOSE) down --volumes
|
|
-docker volume rm -f $(CFG_VOLS)
|
|
|
|
## local: bring up the bind-mount stack (no seed step) and wait for health
|
|
## (Windows / no-make users: run `docker compose -f infra/docker-compose.local.yml up -d --build` directly)
|
|
local:
|
|
docker compose -f $(LOCAL_COMPOSE) up -d --build
|
|
WAIT_TIMEOUT=420 bash infra/wait-healthy.sh $(WAIT_SVCS)
|
|
|
|
## verify-local: acceptance check for the local stack (S-B04) — a fresh `make local` completes the
|
|
## whole flow (zaaktype seeded + DMN deployed + NRC abonnement) with NO manual seeding.
|
|
verify-local:
|
|
bash infra/run-local-flow-check.sh
|
|
|
|
## local-down: stop and remove the bind-mount stack
|
|
local-down:
|
|
docker compose -f $(LOCAL_COMPOSE) down --volumes
|
|
|
|
## changelog: regenerate CHANGELOG.md from Conventional Commits (git-cliff)
|
|
changelog:
|
|
git-cliff --output CHANGELOG.md
|
|
|
|
# ── ZGW verification ───────────────────────────────────────────────────────
|
|
# On the single runner CI jobs run sequentially, so the OpenZaak-dependent checks
|
|
# share ONE full-stack bring-up: the `verify-stack` CI job runs `verify-up` then
|
|
# `verify-acl` + `verify-nrc` as steps against the same stack (issue #58). The
|
|
# check logic lives in stack-agnostic runners that reach services by container IP
|
|
# (gitea-actions-gotchas.md §5/§6); `integration` / `verify-notifications` are local
|
|
# convenience wrappers that bring up a lighter stack and call the same runners.
|
|
|
|
## verify-up: bring the FULL stack up and wait for health (CI verify-stack step 1;
|
|
## subsumes the old compose-smoke health gate — the DoD "up reaches green" check).
|
|
verify-up:
|
|
$(SEED) oz nrc kc fl objecttypen objecten registerrecord
|
|
docker compose -f $(COMPOSE) up -d --build
|
|
WAIT_TIMEOUT=420 bash infra/wait-healthy.sh $(WAIT_SVCS)
|
|
|
|
## verify-acl: ACL ↔ OpenZaak integration tests against the already-running stack.
|
|
verify-acl:
|
|
bash infra/run-acl-integration.sh
|
|
|
|
## verify-nrc: OpenZaak → NRC notification delivery against the already-running stack.
|
|
verify-nrc:
|
|
bash infra/run-notification-check.sh
|
|
|
|
## verify-projection: OpenZaak → NRC → Event Subscriber → projection-api end-to-end (S-06),
|
|
## against the already-running stack.
|
|
verify-projection:
|
|
bash infra/run-projection-check.sh
|
|
|
|
## verify-domain: domain → Flowable → ACL → OpenZaak end-to-end (S-05), against the
|
|
## already-running stack. Recreates the acl service to inject the seeded zaaktype URL.
|
|
verify-domain:
|
|
bash infra/run-domain-check.sh
|
|
|
|
## verify-bff: BFF end-to-end (S-07) against the up stack — token validation on self-service
|
|
## + anonymous public-safe openbaar register (ADR-0010).
|
|
verify-bff:
|
|
bash infra/run-bff-check.sh
|
|
|
|
## verify-e2e: walking-skeleton Playwright e2e (S-08d) against the up stack — DigiD login →
|
|
## submit → confirmation, driven inside the compose network.
|
|
verify-e2e:
|
|
bash infra/run-e2e-check.sh
|
|
|
|
## verify-observability: assert the observability backplane (Grafana + provisioned Tempo &
|
|
## Prometheus datasources) is live, against the already-running stack (S-16a).
|
|
verify-observability:
|
|
bash infra/run-observability-check.sh
|
|
|
|
## verify-tracing: assert one connected distributed trace spans the .NET services in Tempo
|
|
## (S-16b), against the already-running stack.
|
|
verify-tracing:
|
|
bash infra/run-tracing-check.sh
|
|
|
|
## verify-metrics: assert the services expose /metrics and Prometheus scrapes the golden
|
|
## signals (S-16c), against the already-running stack.
|
|
verify-metrics:
|
|
bash infra/run-metrics-check.sh
|
|
|
|
## verify-objecttypen: assert the Objecttypen API is up + its static token authenticates
|
|
## (S-18a), against the already-running stack.
|
|
verify-objecttypen:
|
|
bash infra/run-objecttypen-check.sh
|
|
|
|
## verify-objecten: assert the Objecten API is up + its static token authenticates and it
|
|
## trusts the Objecttypen API (S-18b), against the already-running stack.
|
|
verify-objecten:
|
|
bash infra/run-objecten-check.sh
|
|
|
|
## verify-registerrecord: assert the RegisterRecord objecttype is registered + published in the
|
|
## Objecttypen API (S-18c), against the already-running stack.
|
|
verify-registerrecord:
|
|
bash infra/run-registerrecord-check.sh
|
|
|
|
## verify-objecten-notifications: assert a RegisterRecord write in Objecten is DELIVERED as an
|
|
## `objecten` notification via NRC (S-19b-1), against the already-running stack.
|
|
verify-objecten-notifications:
|
|
bash infra/run-objecten-notifications-check.sh
|
|
|
|
## verify: local mirror of the CI verify-stack job — full stack up once, all checks,
|
|
## tear down (always). For fast single-concern local iteration use `integration`
|
|
## (oz-only) or `verify-notifications` (oz+nrc) instead.
|
|
verify:
|
|
$(SEED) oz nrc kc fl objecttypen objecten registerrecord
|
|
docker compose -f $(COMPOSE) up -d --build
|
|
@bash -c 'set -e; rc=0; \
|
|
WAIT_TIMEOUT=420 bash infra/wait-healthy.sh $(WAIT_SVCS) \
|
|
&& bash infra/run-acl-integration.sh \
|
|
&& bash infra/run-notification-check.sh \
|
|
&& bash infra/run-projection-check.sh \
|
|
&& bash infra/run-objecten-notifications-check.sh \
|
|
&& bash infra/run-domain-check.sh \
|
|
&& bash infra/run-bff-check.sh \
|
|
&& bash infra/run-e2e-check.sh || rc=$$?; \
|
|
docker compose -f $(COMPOSE) down --volumes >/dev/null 2>&1; \
|
|
docker volume rm -f $(CFG_VOLS) >/dev/null 2>&1; \
|
|
exit $$rc'
|
|
|
|
## integration: local convenience — ACL integration test against a throwaway
|
|
## OpenZaak-only stack (fast iteration). CI uses verify-acl on the shared stack.
|
|
integration:
|
|
bash infra/run-integration.sh
|
|
|
|
## openzaak-up: start the OpenZaak stack (migrations run on first start)
|
|
openzaak-up:
|
|
$(SEED) oz
|
|
docker compose -f $(OZ_COMPOSE) up -d
|
|
|
|
## openzaak-smoke: start OpenZaak, then assert it is up with auth enforced
|
|
openzaak-smoke: openzaak-up
|
|
@bash -c 'set -e; \
|
|
echo "waiting for OpenZaak to respond..."; \
|
|
for i in $$(seq 1 60); do \
|
|
code=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/zaken/api/v1/zaken || true); \
|
|
[ -n "$$code" ] && [ "$$code" != "000" ] && break; sleep 3; \
|
|
done; \
|
|
echo "GET /zaken/api/v1/zaken (unauth) -> $$code (expect 403, auth enforced)"; test "$$code" = "403"; \
|
|
admin=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/admin/); \
|
|
echo "GET /admin/ -> $$admin (expect 302)"; test "$$admin" = "302"; \
|
|
root=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/zaken/api/v1/); \
|
|
echo "GET /zaken/api/v1/ -> $$root (expect 200)"; test "$$root" = "200"; \
|
|
echo "OpenZaak smoke OK"'
|
|
|
|
## openzaak-seed: bring OpenZaak up and seed the BIG catalogus (idempotent)
|
|
openzaak-seed: openzaak-up
|
|
@bash -c 'for i in $$(seq 1 50); do \
|
|
c=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/catalogi/api/v1/ || true); \
|
|
[ "$$c" = "200" ] && break; sleep 3; done; echo "OpenZaak ready ($$c)"'
|
|
python3 infra/openzaak/seed_catalogus.py
|
|
|
|
## openzaak-down: stop and remove the OpenZaak stack (wipes data)
|
|
openzaak-down:
|
|
docker compose -f $(OZ_COMPOSE) down --volumes
|
|
-docker volume rm -f rr-oz-config
|
|
|
|
## verify-notifications: local convenience — OpenZaak → NRC notification delivery
|
|
## against a throwaway oz+nrc stack (S-01-c). CI uses verify-nrc on the shared stack.
|
|
verify-notifications:
|
|
bash infra/verify-notifications.sh
|
|
|
|
## stack-up: start OpenZaak + Open Notificaties together (shared network), with
|
|
## OpenZaak publishing notifications to NRC (S-01-c).
|
|
stack-up:
|
|
$(SEED) oz nrc
|
|
OZ_NOTIFICATIONS_DISABLED=false docker compose $(STACK_FILES) up -d
|
|
|
|
## stack-smoke: start both, assert OpenZaak (403/302/200) and NRC (302) are reachable
|
|
stack-smoke: stack-up
|
|
@bash -c 'set -e; \
|
|
echo "waiting for OpenZaak + Open Notificaties..."; \
|
|
for i in $$(seq 1 60); do \
|
|
oz=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/admin/ || true); \
|
|
nrc=$$(curl -s -o /dev/null -w "%{http_code}" $(NRC_BASE)/admin/ || true); \
|
|
[ "$$oz" = "302" ] && [ "$$nrc" = "302" ] && break; sleep 3; done; \
|
|
z=$$(curl -s -o /dev/null -w "%{http_code}" $(OZ_BASE)/zaken/api/v1/zaken); \
|
|
echo "OpenZaak /zaken (unauth) -> $$z (expect 403)"; test "$$z" = "403"; \
|
|
echo "OpenZaak /admin/ -> $$oz (expect 302)"; test "$$oz" = "302"; \
|
|
echo "Open Notificaties /admin/-> $$nrc (expect 302)"; test "$$nrc" = "302"; \
|
|
echo "stack smoke OK"'
|
|
|
|
## stack-down: stop and remove both stacks (wipes data)
|
|
stack-down:
|
|
docker compose $(STACK_FILES) down --volumes
|
|
-docker volume rm -f rr-oz-config rr-nrc-config
|
|
|
|
## keycloak-up: start Keycloak with the four imported realms
|
|
keycloak-up:
|
|
$(SEED) kc
|
|
docker compose -f $(KC_COMPOSE) up -d
|
|
|
|
## keycloak-smoke: start Keycloak, then verify each realm logs in + returns its claim
|
|
keycloak-smoke: keycloak-up
|
|
@bash -c 'for i in $$(seq 1 60); do \
|
|
c=$$(curl -s -o /dev/null -w "%{http_code}" $(KC_BASE)/realms/digid/.well-known/openid-configuration || true); \
|
|
[ "$$c" = "200" ] && break; sleep 3; done; echo "Keycloak ready ($$c)"'
|
|
python3 infra/keycloak/check_realms.py
|
|
|
|
## keycloak-down: stop and remove Keycloak
|
|
keycloak-down:
|
|
docker compose -f $(KC_COMPOSE) down --volumes
|
|
-docker volume rm -f rr-kc-realms
|
|
|
|
## flowable-up: start Flowable (deploys registratie.bpmn on boot)
|
|
flowable-up:
|
|
$(SEED) fl
|
|
docker compose -f $(FL_COMPOSE) up -d
|
|
|
|
## flowable-smoke: start Flowable, then verify a started instance waits on the external task
|
|
flowable-smoke: flowable-up
|
|
@bash -c 'for i in $$(seq 1 80); do \
|
|
c=$$(curl -s -o /dev/null -w "%{http_code}" -u rest-admin:test $(FL_BASE)/repository/process-definitions?key=registratie || true); \
|
|
[ "$$c" = "200" ] && break; sleep 3; done; echo "Flowable ready ($$c)"'
|
|
python3 infra/flowable/verify.py
|
|
|
|
## flowable-down: stop and remove Flowable
|
|
flowable-down:
|
|
docker compose -f $(FL_COMPOSE) down --volumes
|
|
-docker volume rm -f rr-fl-bpmn
|
|
|
|
## help: list available targets
|
|
help:
|
|
@grep -E '^## ' $(MAKEFILE_LIST) | sed 's/^## //'
|