register-referentie/docs/runbooks/gitea-actions-gotchas.md

# Gitea Actions gotchas

How our CI (Gitea Actions on the hosted **`ubuntu-latest`** runner) differs from a
local run, and the workarounds in this repo. Referenced by `CLAUDE.md` §8.7/§15.

**One root cause sits under most of this:** the runner executes the job **inside a
container**, so when a step runs `docker compose up`, Compose starts the stack as
**sibling containers** on the host's daemon. Anything that assumes the job and
those containers share a filesystem — or a `localhost` — breaks.

| Gotcha | Fix | Lives in |
|---|---|---|
| Bind-mounted config arrives empty | `docker cp` config into external volumes | `infra/seed-config.sh` |
| `docker compose up --wait` is unsupported / flaky | poll health with `docker inspect` | `infra/wait-healthy.sh` |
| `pg_isready` passes before PostGIS is ready | add a `PostGIS_Version()` probe | the db healthchecks |
| `upload-artifact@v4` fails ("not supported on GHES") | pin `@v3` | `.gitea/workflows/ci.yaml` (`mutation` job) |
| `upload-artifact@v3` fails with "Artifact service responded with 500" | mark the upload `continue-on-error: true` (server-side; issue #62) | `.gitea/workflows/ci.yaml` (`mutation` job) |

---

## 1. Bind mounts don't reach the containers

**Symptom** — green locally, but `compose-smoke` fails with:

```
oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.
```

Migrations run fine; only the step that reads a *mounted* file fails. The same
trap hits `nrc-init`, `flowable-init`, and `keycloak`.

**Why** — a relative bind mount like `./openzaak/setup_configuration:/app/...` is
resolved by Compose to a path *inside the job container*
(`/workspace/.../setup_configuration`). The daemon then looks for that path on
*its own host*, doesn't find it, and mounts an **empty directory**. (It works on a
runner that executes jobs on the host — which is why moving to `ubuntu-latest`
exposed it.)

**Fix** — use the upstream images verbatim (no build) and stream config into
**external named volumes** with `docker cp`, which copies over the Docker API and
so works wherever the daemon runs. `infra/seed-config.sh` creates each volume,
mounts it in a throwaway helper, and copies the files in:

| Asset | Volume | Mounted at |
|---|---|---|
| OpenZaak `data.yaml` | `rr-oz-config` | `oz-init:/app/setup_configuration` |
| Keycloak realms | `rr-kc-realms` | `keycloak:/opt/keycloak/data/import` |
| `registratie.bpmn` | `rr-fl-bpmn` | `flowable-init:/work` |

The volumes are `external: true` with fixed names, so they resolve identically
under docker compose and podman-compose. `make` seeds before every `up`; `make
down` removes them. (Open Notificaties needs nothing — `nrc-init` migrates only.)

**Consequence — bare `docker compose up` can't self-seed external volumes:**

- **CI / Linux / macOS:** `make up` or `make smoke` (seed, then start).
- **No-make / Windows:** `infra/docker-compose.local.yml` — a twin stack that
  **bind-mounts** the config instead. Bind mounts are fine *locally* because a
  local daemon can see your working directory, so
  `docker compose -f infra/docker-compose.local.yml up -d` just works.

**Why not the obvious alternatives**

- *Bake config into an image* (incl. an inline Dockerfile) — `docker compose up`
  would then work unaided, but it's a build; we wanted the upstream images as-is.
- *Compose `configs:` with inline `content`* — Compose writes a client-side temp
  file and bind-mounts it, hitting the exact same problem.
- *A host-executing runner* — bind mounts would work with zero seeding, but it
  reintroduces a self-hosted runner and undoes the move to `ubuntu-latest`.

---

## 2. Readiness: poll health, don't use `--wait`

`docker compose up --wait` looks ideal but fails us three ways:

- **podman-compose doesn't implement it** (`unrecognized arguments: --wait`) — so
  it would break local dev.
- A project-wide `--wait` **treats a one-shot exiting `0` as a failure** unless
  something `depends_on` it with `service_completed_successfully`. `flowable-init`
  deploys the BPMN and exits with no dependant, so `--wait` fails the moment it
  does — last line `container infra-flowable-init-1 exited (0)`.
- The containerized runner **can't reach published host ports**, so an external
  `curl localhost:8080/health` can't work either.

**Fix** — `infra/wait-healthy.sh` polls each durable service (`openzaak nrc-web
acl bff`, listed as `WAIT_SVCS` in the `Makefile`) with `docker ps` + `docker
inspect '{{.State.Health.Status}}'` until it reports `healthy`. It uses only
primitives both runtimes support, reads the **in-container** healthcheck (no host
port needed), and ignores the one-shots (they only need to have run).
`WAIT_TIMEOUT` defaults to 420 s — enough for the cold OpenZaak migrate (~90 s)
plus app start.

---

## 3. `pg_isready` passes before PostGIS is ready

`pg_isready` succeeds as soon as the TCP port is open — *before* the
`postgis/postgis` image has finished running `CREATE EXTENSION postgis`. An init
container that starts migrating in that window can fail on a missing PostGIS. So
the db healthchecks add a `SELECT PostGIS_Version()` probe, making dependents wait
for the extension, not just the port.

---

## 4. `actions/upload-artifact@v4` refuses to run on Gitea

**Symptom** — the `mutation` job's `make mutation` step passes (95% score), but the
upload step right after it fails the job:

```
::error::@actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+
are not currently supported on GHES.
❌  Failure - Main https://github.com/actions/upload-artifact@v4
```

**Why** — `upload-artifact@v4` bundles `@actions/artifact` v2, which inspects the
server URL and **hard-aborts on anything that isn't `github.com`**, treating Gitea
as an unsupported GitHub Enterprise Server. The check fires regardless of whether
the Gitea server can actually store artifacts (1.24+ can). It is the *action*, not
the server, that refuses.

**Fix** — pin **`actions/upload-artifact@v3`** (and `download-artifact@v3` if ever
needed). v3 uses the older artifact protocol that Gitea implements, and has no GHES
guard. Inputs are the same (`name`, `path`, `if-no-files-found`), so it is a drop-in
swap. Do **not** bump to `@v4` until act_runner advertises github.com-compatible
artifact support.

**Second failure mode — the server's artifact backend returns 500.** Even on the
correctly-pinned `@v3`, uploads can fail with:

```
Create Artifact Container - Attempt 5 of 5 failed with error: Artifact service responded with 500
::error::Create Artifact Container failed: Artifact service responded with 500
```

This is the **Gitea server's** artifact storage failing (not the action's GHES guard),
so it is outside the repo's control. Because the `mutation` job's upload steps run with
`if: always()`, that 500 would fail the job even though the ratchet passed. **Fix:** mark
the uploads `continue-on-error: true` (issue #62). The mutation *gate* is the Stryker
ratchet — `make mutation`'s exit code fails the job on a real regression — so the report
upload is best-effort: when the server's artifact storage is restored, reports publish
again with no workflow change.

---

## 5. A runner process can't reach a service container's published port

**Symptom** — green locally, but a CI step that runs *on the runner* and talks to a
compose service over `localhost` fails. The ACL integration test's seed died with:

```
OpenZaak ready (000)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
make: *** [Makefile:114: integration] Error 1
```

OpenZaak was demonstrably up — uwsgi had been serving for ~2 minutes — yet
`curl`/`urllib` to `localhost:8000` from the runner were refused the whole time.

**Why** — the same sibling-container split as §1. Compose starts the stack via the
host daemon, so `ports: ["8000:8000"]` publishes to the *daemon host*, not to the job
container. From the runner, `localhost:8000` has nothing listening. (`make smoke`
sidesteps this by polling readiness via `docker inspect` (§2), never a service port.)

**Fix** — don't talk to service ports from the runner. Either check state via `docker
inspect` (health), or run the client **inside the compose network** so it reaches the
service by name (`http://openzaak:8000`). For a test/seed that needs the repo's own
code, deliver it via a **built image** (not a bind mount — §1), then
`docker run --network <stack>_cg …`.

**Applied** — `make integration` (ADR-0006) and `make verify-notifications` (ADR-0007)
do exactly this: they run the seed/test/driver as containers on the stack network and
reach services by **container IP** (see §6).

---

## 6. OpenZaak / NRC reject single-label hosts in URLs

**Symptom** — talking to OpenZaak or NRC by compose **service name** fails where a URL
is validated: catalogus/zaaktype filters, the zaak `zaaktype` URL, and abonnement
`callbackUrl` come back `400 "Voer een geldige URL in."` — even though the host
resolves and is reachable.

**Why** — these apps validate URLs with Django's `URLValidator`, which rejects a
**single-label** host like `openzaak` or `nrc-web` (no dot, and not `localhost`).
`localhost` passes (so it's invisible in host-port-based local runs); in-network the
reality is a service name or an IPv4 literal — and only the IP passes.

**Fix** — in-network tooling reaches OpenZaak/NRC by **container IP**
(`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'`), not
service name; the notif verify harness also registers the sink callback by IP.
(`infra/run-acl-integration.sh`, `infra/run-notification-check.sh`.)

**Related — abonnement callbacks must enforce auth.** NRC probes a callback when an
abonnement is registered and refuses it (`no-auth-on-callback-url`) unless it returns
**401** without the configured `Authorization`. The verify sink
(`infra/notification-sink.py`) enforces a bearer token for exactly this reason.

---

## 7. A job with `if: ${{ !cancelled() }}` (or `always()`) + `needs` sticks in "waiting"

**Symptom** — after upgrading to **Gitea 1.27** + **act_runner 2.0.0**, one job never
starts: the run sits in state `waiting` forever, the job has **no logs** (never
dispatched to a runner), and the other jobs finish normally. `main` stays pending/red.
Seen on the `verify-stack` job (#134).

**Why** — Gitea 1.27 reworked cancellation/aggregation: a job gated by a
**status-function `if`** (`always()` / `cancelled()` / `!cancelled()`) on top of
`needs` now routes through a new transitional **`Cancelling`** job state plus a
server↔runner **capability negotiation** ("Requires Gitea Runner 2.0.0"). On the
1.27 + 2.0.0 pairing that handshake doesn't resolve for such a job, so it's never
offered to a runner and never leaves `waiting`. Jobs with no `if`/`needs` are
unaffected. (Related upstream: go-gitea/gitea#31074, #27116, #35782.)

**Fix** — don't gate a `needs` job with a status-function `if`. Use the default
`if: success()` (i.e. omit the `if`). If you need "run even when an upstream job
fails", prefer serialising with a `concurrency` group over `needs` + `always()`.

**Also** — a run already stuck this way will **not** clear itself; force-cancel it
from the Actions UI (plain cancel can also stall on this version, #35782). Push the
workflow fix to produce a fresh run.

---

## 8. Job summaries (`$GITHUB_STEP_SUMMARY`) need Gitea ≥1.27 + runner ≥2.0

Markdown a step appends to the `$GITHUB_STEP_SUMMARY` file renders on the run page
(no artifact download). We use it for per-run reports (#136): mutation scores
(Stryker `markdown` reporter), per-service unit results (`infra/trx-summary.py` over
TRX), per-frontend results (`infra/vitest-summary.py` over each app's vitest JSON),
the verify-stack check table, and per-spec e2e results (`infra/playwright-summary.py`).

**Requirements / conventions:**

- Requires **Gitea ≥ 1.27** (stores/renders summaries) and **act_runner ≥ 2.0.0**
  (uploads them). Older pairings silently skip the upload.
- **Guard every write:** `[ -n "${GITHUB_STEP_SUMMARY:-}" ] || exit 0` — on a runner
  without support the var is unset and `>> "$GITHUB_STEP_SUMMARY"` would be an
  ambiguous-redirect error. The guard makes the step a no-op locally / on old runners.
- Use `if: always()` (step-level) on summary steps so they render even when the thing
  they report on failed. Step-level `always()` is fine on 2.0.0 — unlike the *job*-level
  status-function `if` of §7.
- Getting a report out of the e2e container: Playwright writes `playwright-report.json`
  inside the container; `infra/run-e2e-check.sh` `docker cp`s it back to the host
  (capturing the test exit code first) so the summary step can read it.

---

## 9. `if: always()` does not survive the job being killed — bound the work itself

`if: always()` makes a step run when an *earlier step failed*. It does **not** help when
the job as a whole is stopped: the run's remaining steps are simply never dispatched.

That is how #161 lost its diagnosis. `verify-stack` entered `make verify-e2e` at 09:48:17
and the job ended at 10:14:54 — 26½ minutes later, mid-suite. Every step after the e2e
shows a **0-second `failure`** stamped at that same instant:

```
14 failure  09:48:17 -> 10:14:54  Self-service e2e (Playwright, login → submit → success)
15 failure  10:14:54 -> 10:14:54  verify-stack check summary       ← if: always()
16 failure  10:14:54 -> 10:14:54  e2e spec summary                 ← if: always()
17 failure  10:14:54 -> 10:14:54  Dump container logs on failure   ← if: failure()
18 failure  10:14:54 -> 10:14:54  Tear down                        ← if: always()
```

So the per-spec summary, the container-log dump and the teardown never ran, and the job
log — which also loses whatever the killed process had buffered — ended at a single `✘`
line. A job that dies takes its own post-mortem with it.

**Read the step timings, not just the log.** `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}`
returns every step with `started_at`/`completed_at`; a row of identical zero-length
steps at the end means *killed*, not *silent*. (Job ids come from
`…/actions/runs/{run}/jobs`, and that route returns only the **latest attempt** — a
re-run hides the failed one, so keep the failing job id from the original report. Logs:
`…/actions/jobs/{id}/logs`, see also `gitea-ci-logs`.)

**Conventions that follow:**

- **Bound long-running work inside the tool**, where it can still report. Playwright's
  `globalTimeout` (`tests/e2e/playwright.config.ts`) ends the run, writes the JSON
  report and exits, so the summary and log-dump steps still get their turn. A
  `timeout-minutes` on the job would reproduce the very failure above.
- **Never let an auto-waiting action be the timeout.** Playwright actions (`fill`,
  `click`) inherit the *test* timeout, not `expect.timeout`, so a missing element costs
  the full 90 s and reports `locator.fill: Test timeout …` — the symptom. Assert the
  element visible first with its own budget and a message (`tests/e2e/keycloak-login.ts`).
- Remember `concurrency.cancel-in-progress: true` in `ci.yaml`: a new push to the same
  ref, or a re-run, kills the in-flight run the same way. Check `run_attempt` before
  concluding a job hung.