register-referentie/docs/runbooks/gitea-actions-gotchas.md

Gitea Actions gotchas

How our CI (Gitea Actions on the hosted ubuntu-latest runner) differs from a local run, and the workarounds in this repo. Referenced by CLAUDE.md §8.7/§15.

One root cause sits under most of this: the runner executes the job inside a container, so when a step runs docker compose up, Compose starts the stack as sibling containers on the host's daemon. Anything that assumes the job and those containers share a filesystem — or a localhost — breaks.

Gotcha	Fix	Lives in
Bind-mounted config arrives empty	docker cp config into external volumes	infra/seed-config.sh
docker compose up --wait is unsupported / flaky	poll health with docker inspect	infra/wait-healthy.sh
pg_isready passes before PostGIS is ready	add a PostGIS_Version() probe	the db healthchecks
upload-artifact@v4 fails ("not supported on GHES")	pin @v3	.gitea/workflows/ci.yaml (mutation job)
upload-artifact@v3 fails with "Artifact service responded with 500"	mark the upload continue-on-error: true (server-side; issue #62)	.gitea/workflows/ci.yaml (mutation job)

---

1. Bind mounts don't reach the containers

Symptom — green locally, but compose-smoke fails with:

oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.

Migrations run fine; only the step that reads a mounted file fails. The same trap hits nrc-init, flowable-init, and keycloak.

Why — a relative bind mount like ./openzaak/setup_configuration:/app/... is resolved by Compose to a path inside the job container (/workspace/.../setup_configuration). The daemon then looks for that path on its own host, doesn't find it, and mounts an empty directory. (It works on a runner that executes jobs on the host — which is why moving to ubuntu-latest exposed it.)

Fix — use the upstream images verbatim (no build) and stream config into external named volumes with docker cp, which copies over the Docker API and so works wherever the daemon runs. infra/seed-config.sh creates each volume, mounts it in a throwaway helper, and copies the files in:

Asset	Volume	Mounted at
OpenZaak data.yaml	rr-oz-config	oz-init:/app/setup_configuration
Keycloak realms	rr-kc-realms	keycloak:/opt/keycloak/data/import
registratie.bpmn	rr-fl-bpmn	flowable-init:/work

The volumes are external: true with fixed names, so they resolve identically under docker compose and podman-compose. make seeds before every up; make down removes them. (Open Notificaties needs nothing — nrc-init migrates only.)

Consequence — bare docker compose up can't self-seed external volumes:

• CI / Linux / macOS: make up or make smoke (seed, then start).
• No-make / Windows: infra/docker-compose.local.yml — a twin stack that bind-mounts the config instead. Bind mounts are fine locally because a local daemon can see your working directory, so docker compose -f infra/docker-compose.local.yml up -d just works.

Why not the obvious alternatives

• Bake config into an image (incl. an inline Dockerfile) — docker compose up would then work unaided, but it's a build; we wanted the upstream images as-is.
• Compose configs: with inline content — Compose writes a client-side temp file and bind-mounts it, hitting the exact same problem.
• A host-executing runner — bind mounts would work with zero seeding, but it reintroduces a self-hosted runner and undoes the move to ubuntu-latest.

---

2. Readiness: poll health, don't use --wait

docker compose up --wait looks ideal but fails us three ways:

• podman-compose doesn't implement it (unrecognized arguments: --wait) — so it would break local dev.
• A project-wide --wait treats a one-shot exiting 0 as a failure unless something depends_on it with service_completed_successfully. flowable-init deploys the BPMN and exits with no dependant, so --wait fails the moment it does — last line container infra-flowable-init-1 exited (0).
• The containerized runner can't reach published host ports, so an external curl localhost:8080/health can't work either.

Fix — infra/wait-healthy.sh polls each durable service (openzaak nrc-web acl bff, listed as WAIT_SVCS in the Makefile) with docker ps + docker inspect '{{.State.Health.Status}}' until it reports healthy. It uses only primitives both runtimes support, reads the in-container healthcheck (no host port needed), and ignores the one-shots (they only need to have run). WAIT_TIMEOUT defaults to 420 s — enough for the cold OpenZaak migrate (~90 s) plus app start.

---

3. pg_isready passes before PostGIS is ready

pg_isready succeeds as soon as the TCP port is open — before the postgis/postgis image has finished running CREATE EXTENSION postgis. An init container that starts migrating in that window can fail on a missing PostGIS. So the db healthchecks add a SELECT PostGIS_Version() probe, making dependents wait for the extension, not just the port.

---

4. actions/upload-artifact@v4 refuses to run on Gitea

Symptom — the mutation job's make mutation step passes (95% score), but the upload step right after it fails the job:

::error::@actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+
are not currently supported on GHES.
❌  Failure - Main https://github.com/actions/upload-artifact@v4

Why — upload-artifact@v4 bundles @actions/artifact v2, which inspects the server URL and hard-aborts on anything that isn't github.com, treating Gitea as an unsupported GitHub Enterprise Server. The check fires regardless of whether the Gitea server can actually store artifacts (1.24+ can). It is the action, not the server, that refuses.

Fix — pin actions/upload-artifact@v3 (and download-artifact@v3 if ever needed). v3 uses the older artifact protocol that Gitea implements, and has no GHES guard. Inputs are the same (name, path, if-no-files-found), so it is a drop-in swap. Do not bump to @v4 until act_runner advertises github.com-compatible artifact support.

Second failure mode — the server's artifact backend returns 500. Even on the correctly-pinned @v3, uploads can fail with:

Create Artifact Container - Attempt 5 of 5 failed with error: Artifact service responded with 500
::error::Create Artifact Container failed: Artifact service responded with 500

This is the Gitea server's artifact storage failing (not the action's GHES guard), so it is outside the repo's control. Because the mutation job's upload steps run with if: always(), that 500 would fail the job even though the ratchet passed. Fix: mark the uploads continue-on-error: true (issue #62). The mutation gate is the Stryker ratchet — make mutation's exit code fails the job on a real regression — so the report upload is best-effort: when the server's artifact storage is restored, reports publish again with no workflow change.

---

5. A runner process can't reach a service container's published port

Symptom — green locally, but a CI step that runs on the runner and talks to a compose service over localhost fails. The ACL integration test's seed died with:

OpenZaak ready (000)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
make: *** [Makefile:114: integration] Error 1

OpenZaak was demonstrably up — uwsgi had been serving for ~2 minutes — yet curl/urllib to localhost:8000 from the runner were refused the whole time.

Why — the same sibling-container split as §1. Compose starts the stack via the host daemon, so ports: ["8000:8000"] publishes to the daemon host, not to the job container. From the runner, localhost:8000 has nothing listening. (make smoke sidesteps this by polling readiness via docker inspect (§2), never a service port.)

Fix — don't talk to service ports from the runner. Either check state via docker inspect (health), or run the client inside the compose network so it reaches the service by name (http://openzaak:8000). For a test/seed that needs the repo's own code, deliver it via a built image (not a bind mount — §1), then docker run --network <stack>_cg ….

Applied — make integration (ADR-0006) and make verify-notifications (ADR-0007) do exactly this: they run the seed/test/driver as containers on the stack network and reach services by container IP (see §6).

---

6. OpenZaak / NRC reject single-label hosts in URLs

Symptom — talking to OpenZaak or NRC by compose service name fails where a URL is validated: catalogus/zaaktype filters, the zaak zaaktype URL, and abonnement callbackUrl come back 400 "Voer een geldige URL in." — even though the host resolves and is reachable.

Why — these apps validate URLs with Django's URLValidator, which rejects a single-label host like openzaak or nrc-web (no dot, and not localhost). localhost passes (so it's invisible in host-port-based local runs); in-network the reality is a service name or an IPv4 literal — and only the IP passes.

Fix — in-network tooling reaches OpenZaak/NRC by container IP (docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'), not service name; the notif verify harness also registers the sink callback by IP. (infra/run-acl-integration.sh, infra/run-notification-check.sh.)

Related — abonnement callbacks must enforce auth. NRC probes a callback when an abonnement is registered and refuses it (no-auth-on-callback-url) unless it returns 401 without the configured Authorization. The verify sink (infra/notification-sink.py) enforces a bearer token for exactly this reason.

---

7. A job with if: ${{ !cancelled() }} (or always()) + needs sticks in "waiting"

Symptom — after upgrading to Gitea 1.27 + act_runner 2.0.0, one job never starts: the run sits in state waiting forever, the job has no logs (never dispatched to a runner), and the other jobs finish normally. main stays pending/red. Seen on the verify-stack job (#134).

Why — Gitea 1.27 reworked cancellation/aggregation: a job gated by a status-function if (always() / cancelled() / !cancelled()) on top of needs now routes through a new transitional Cancelling job state plus a server↔runner capability negotiation ("Requires Gitea Runner 2.0.0"). On the 1.27 + 2.0.0 pairing that handshake doesn't resolve for such a job, so it's never offered to a runner and never leaves waiting. Jobs with no if/needs are unaffected. (Related upstream: go-gitea/gitea#31074, #27116, #35782.)

Fix — don't gate a needs job with a status-function if. Use the default if: success() (i.e. omit the if). If you need "run even when an upstream job fails", prefer serialising with a concurrency group over needs + always().

Also — a run already stuck this way will not clear itself; force-cancel it from the Actions UI (plain cancel can also stall on this version, #35782). Push the workflow fix to produce a fresh run.

---

8. Job summaries ($GITHUB_STEP_SUMMARY) need Gitea ≥1.27 + runner ≥2.0

Markdown a step appends to the $GITHUB_STEP_SUMMARY file renders on the run page (no artifact download). We use it for per-run reports (#136): mutation scores (Stryker markdown reporter), per-service unit results (infra/trx-summary.py over TRX), per-frontend results (infra/vitest-summary.py over each app's vitest JSON), the verify-stack check table, and per-spec e2e results (infra/playwright-summary.py).

Requirements / conventions:

• Requires Gitea ≥ 1.27 (stores/renders summaries) and act_runner ≥ 2.0.0 (uploads them). Older pairings silently skip the upload.
• Guard every write: [ -n "${GITHUB_STEP_SUMMARY:-}" ] || exit 0 — on a runner without support the var is unset and >> "$GITHUB_STEP_SUMMARY" would be an ambiguous-redirect error. The guard makes the step a no-op locally / on old runners.
• Use if: always() (step-level) on summary steps so they render even when the thing they report on failed. Step-level always() is fine on 2.0.0 — unlike the job-level status-function if of §7.
• Getting a report out of the e2e container: Playwright writes playwright-report.json inside the container; infra/run-e2e-check.sh docker cps it back to the host (capturing the test exit code first) so the summary step can read it.

---

9. if: always() does not survive the job being killed — bound the work itself

if: always() makes a step run when an earlier step failed. It does not help when the job as a whole is stopped: the run's remaining steps are simply never dispatched.

That is how #161 lost its diagnosis. verify-stack entered make verify-e2e at 09:48:17 and the job ended at 10:14:54 — 26½ minutes later, mid-suite. Every step after the e2e shows a 0-second failure stamped at that same instant:

14 failure  09:48:17 -> 10:14:54  Self-service e2e (Playwright, login → submit → success)
15 failure  10:14:54 -> 10:14:54  verify-stack check summary       ← if: always()
16 failure  10:14:54 -> 10:14:54  e2e spec summary                 ← if: always()
17 failure  10:14:54 -> 10:14:54  Dump container logs on failure   ← if: failure()
18 failure  10:14:54 -> 10:14:54  Tear down                        ← if: always()

So the per-spec summary, the container-log dump and the teardown never ran, and the job log — which also loses whatever the killed process had buffered — ended at a single ✘ line. A job that dies takes its own post-mortem with it.

Read the step timings, not just the log. GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id} returns every step with started_at/completed_at; a row of identical zero-length steps at the end means killed, not silent. (Job ids come from …/actions/runs/{run}/jobs, and that route returns only the latest attempt — a re-run hides the failed one, so keep the failing job id from the original report. Logs: …/actions/jobs/{id}/logs, see also gitea-ci-logs.)

Conventions that follow:

• Bound long-running work inside the tool, where it can still report. Playwright's globalTimeout (tests/e2e/playwright.config.ts) ends the run, writes the JSON report and exits, so the summary and log-dump steps still get their turn. A timeout-minutes on the job would reproduce the very failure above.
• Never let an auto-waiting action be the timeout. Playwright actions (fill, click) inherit the test timeout, not expect.timeout, so a missing element costs the full 90 s and reports locator.fill: Test timeout … — the symptom. Assert the element visible first with its own budget and a message (tests/e2e/keycloak-login.ts).
• Remember concurrency.cancel-in-progress: true in ci.yaml: a new push to the same ref, or a re-run, kills the in-flight run the same way. Check run_attempt before concluding a job hung.