not
849bf4723b
## What & why After the Gitea 1.27 + act_runner 2.0.0 upgrade, `verify-stack` never starts: the run sits in `waiting` forever with no logs for that job, while the other five jobs pass — so `main` stays pending/red (P0). See #134. Closes #134 ### Root cause `verify-stack` was the only job gated by a status-function `if` on top of `needs`: ```yaml verify-stack: needs: [mutation] if: ${{ !cancelled() }} ``` Gitea 1.27 reworked cancellation/aggregation so that `always()`/`cancelled()`-gated `needs` jobs route through a new transitional **`Cancelling`** state + server↔runner **capability negotiation** ("Requires Gitea Runner 2.0.0"). On this 1.27 + 2.0.0 pairing that handshake doesn't resolve, so the job is never dispatched and never leaves `waiting`. Plain jobs (no `if`/`needs`) are unaffected — exactly the observed pattern. It worked pre-upgrade (old runner). ### Fix Drop the `if: ${{ !cancelled() }}`; keep `needs: [mutation]`. Default `if: success()` dispatches normally and still serialises the two memory-heavy jobs (OOM avoidance, #126). **Trade-off:** the `!cancelled()` (added in #127) let verify-stack run even when the mutation ratchet fails. Now a failing mutation skips verify-stack; the fix-and-re-push re-run exercises it, so the signal isn't lost — just deferred to the green-mutation run. If we later want both signals on one run, serialise via a `concurrency` group rather than `needs` + `always()`. Documented as §7 in `docs/runbooks/gitea-actions-gotchas.md`. ## Note on the stuck run Run 582 (the #133 merge) will **not** clear itself and must be force-cancelled from the Actions UI (plain cancel can also stall on this version, gitea#35782). This PR's own run is the first real test of the fix — if `verify-stack` dispatches and runs here, the fix holds. ## Definition of Done - [x] Linked issue (#134). - [x] Conventional Commit referencing the issue. - [ ] CI green — this PR's run is the verification (verify-stack must dispatch). - [x] Runbook updated (gotchas §7). - [ ] Closed by the merging PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #135
224 lines
11 KiB
Markdown
224 lines
11 KiB
Markdown
# Gitea Actions gotchas
|
|
|
|
How our CI (Gitea Actions on the hosted **`ubuntu-latest`** runner) differs from a
|
|
local run, and the workarounds in this repo. Referenced by `CLAUDE.md` §8.7/§15.
|
|
|
|
**One root cause sits under most of this:** the runner executes the job **inside a
|
|
container**, so when a step runs `docker compose up`, Compose starts the stack as
|
|
**sibling containers** on the host's daemon. Anything that assumes the job and
|
|
those containers share a filesystem — or a `localhost` — breaks.
|
|
|
|
| Gotcha | Fix | Lives in |
|
|
|---|---|---|
|
|
| Bind-mounted config arrives empty | `docker cp` config into external volumes | `infra/seed-config.sh` |
|
|
| `docker compose up --wait` is unsupported / flaky | poll health with `docker inspect` | `infra/wait-healthy.sh` |
|
|
| `pg_isready` passes before PostGIS is ready | add a `PostGIS_Version()` probe | the db healthchecks |
|
|
| `upload-artifact@v4` fails ("not supported on GHES") | pin `@v3` | `.gitea/workflows/ci.yaml` (`mutation` job) |
|
|
| `upload-artifact@v3` fails with "Artifact service responded with 500" | mark the upload `continue-on-error: true` (server-side; issue #62) | `.gitea/workflows/ci.yaml` (`mutation` job) |
|
|
|
|
---
|
|
|
|
## 1. Bind mounts don't reach the containers
|
|
|
|
**Symptom** — green locally, but `compose-smoke` fails with:
|
|
|
|
```
|
|
oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.
|
|
```
|
|
|
|
Migrations run fine; only the step that reads a *mounted* file fails. The same
|
|
trap hits `nrc-init`, `flowable-init`, and `keycloak`.
|
|
|
|
**Why** — a relative bind mount like `./openzaak/setup_configuration:/app/...` is
|
|
resolved by Compose to a path *inside the job container*
|
|
(`/workspace/.../setup_configuration`). The daemon then looks for that path on
|
|
*its own host*, doesn't find it, and mounts an **empty directory**. (It works on a
|
|
runner that executes jobs on the host — which is why moving to `ubuntu-latest`
|
|
exposed it.)
|
|
|
|
**Fix** — use the upstream images verbatim (no build) and stream config into
|
|
**external named volumes** with `docker cp`, which copies over the Docker API and
|
|
so works wherever the daemon runs. `infra/seed-config.sh` creates each volume,
|
|
mounts it in a throwaway helper, and copies the files in:
|
|
|
|
| Asset | Volume | Mounted at |
|
|
|---|---|---|
|
|
| OpenZaak `data.yaml` | `rr-oz-config` | `oz-init:/app/setup_configuration` |
|
|
| Keycloak realms | `rr-kc-realms` | `keycloak:/opt/keycloak/data/import` |
|
|
| `registratie.bpmn` | `rr-fl-bpmn` | `flowable-init:/work` |
|
|
|
|
The volumes are `external: true` with fixed names, so they resolve identically
|
|
under docker compose and podman-compose. `make` seeds before every `up`; `make
|
|
down` removes them. (Open Notificaties needs nothing — `nrc-init` migrates only.)
|
|
|
|
**Consequence — bare `docker compose up` can't self-seed external volumes:**
|
|
|
|
- **CI / Linux / macOS:** `make up` or `make smoke` (seed, then start).
|
|
- **No-make / Windows:** `infra/docker-compose.local.yml` — a twin stack that
|
|
**bind-mounts** the config instead. Bind mounts are fine *locally* because a
|
|
local daemon can see your working directory, so
|
|
`docker compose -f infra/docker-compose.local.yml up -d` just works.
|
|
|
|
**Why not the obvious alternatives**
|
|
|
|
- *Bake config into an image* (incl. an inline Dockerfile) — `docker compose up`
|
|
would then work unaided, but it's a build; we wanted the upstream images as-is.
|
|
- *Compose `configs:` with inline `content`* — Compose writes a client-side temp
|
|
file and bind-mounts it, hitting the exact same problem.
|
|
- *A host-executing runner* — bind mounts would work with zero seeding, but it
|
|
reintroduces a self-hosted runner and undoes the move to `ubuntu-latest`.
|
|
|
|
---
|
|
|
|
## 2. Readiness: poll health, don't use `--wait`
|
|
|
|
`docker compose up --wait` looks ideal but fails us three ways:
|
|
|
|
- **podman-compose doesn't implement it** (`unrecognized arguments: --wait`) — so
|
|
it would break local dev.
|
|
- A project-wide `--wait` **treats a one-shot exiting `0` as a failure** unless
|
|
something `depends_on` it with `service_completed_successfully`. `flowable-init`
|
|
deploys the BPMN and exits with no dependant, so `--wait` fails the moment it
|
|
does — last line `container infra-flowable-init-1 exited (0)`.
|
|
- The containerized runner **can't reach published host ports**, so an external
|
|
`curl localhost:8080/health` can't work either.
|
|
|
|
**Fix** — `infra/wait-healthy.sh` polls each durable service (`openzaak nrc-web
|
|
acl bff`, listed as `WAIT_SVCS` in the `Makefile`) with `docker ps` + `docker
|
|
inspect '{{.State.Health.Status}}'` until it reports `healthy`. It uses only
|
|
primitives both runtimes support, reads the **in-container** healthcheck (no host
|
|
port needed), and ignores the one-shots (they only need to have run).
|
|
`WAIT_TIMEOUT` defaults to 420 s — enough for the cold OpenZaak migrate (~90 s)
|
|
plus app start.
|
|
|
|
---
|
|
|
|
## 3. `pg_isready` passes before PostGIS is ready
|
|
|
|
`pg_isready` succeeds as soon as the TCP port is open — *before* the
|
|
`postgis/postgis` image has finished running `CREATE EXTENSION postgis`. An init
|
|
container that starts migrating in that window can fail on a missing PostGIS. So
|
|
the db healthchecks add a `SELECT PostGIS_Version()` probe, making dependents wait
|
|
for the extension, not just the port.
|
|
|
|
---
|
|
|
|
## 4. `actions/upload-artifact@v4` refuses to run on Gitea
|
|
|
|
**Symptom** — the `mutation` job's `make mutation` step passes (95% score), but the
|
|
upload step right after it fails the job:
|
|
|
|
```
|
|
::error::@actions/artifact v2.0.0+, upload-artifact@v4+ and download-artifact@v4+
|
|
are not currently supported on GHES.
|
|
❌ Failure - Main https://github.com/actions/upload-artifact@v4
|
|
```
|
|
|
|
**Why** — `upload-artifact@v4` bundles `@actions/artifact` v2, which inspects the
|
|
server URL and **hard-aborts on anything that isn't `github.com`**, treating Gitea
|
|
as an unsupported GitHub Enterprise Server. The check fires regardless of whether
|
|
the Gitea server can actually store artifacts (1.24+ can). It is the *action*, not
|
|
the server, that refuses.
|
|
|
|
**Fix** — pin **`actions/upload-artifact@v3`** (and `download-artifact@v3` if ever
|
|
needed). v3 uses the older artifact protocol that Gitea implements, and has no GHES
|
|
guard. Inputs are the same (`name`, `path`, `if-no-files-found`), so it is a drop-in
|
|
swap. Do **not** bump to `@v4` until act_runner advertises github.com-compatible
|
|
artifact support.
|
|
|
|
**Second failure mode — the server's artifact backend returns 500.** Even on the
|
|
correctly-pinned `@v3`, uploads can fail with:
|
|
|
|
```
|
|
Create Artifact Container - Attempt 5 of 5 failed with error: Artifact service responded with 500
|
|
::error::Create Artifact Container failed: Artifact service responded with 500
|
|
```
|
|
|
|
This is the **Gitea server's** artifact storage failing (not the action's GHES guard),
|
|
so it is outside the repo's control. Because the `mutation` job's upload steps run with
|
|
`if: always()`, that 500 would fail the job even though the ratchet passed. **Fix:** mark
|
|
the uploads `continue-on-error: true` (issue #62). The mutation *gate* is the Stryker
|
|
ratchet — `make mutation`'s exit code fails the job on a real regression — so the report
|
|
upload is best-effort: when the server's artifact storage is restored, reports publish
|
|
again with no workflow change.
|
|
|
|
---
|
|
|
|
## 5. A runner process can't reach a service container's published port
|
|
|
|
**Symptom** — green locally, but a CI step that runs *on the runner* and talks to a
|
|
compose service over `localhost` fails. The ACL integration test's seed died with:
|
|
|
|
```
|
|
OpenZaak ready (000)
|
|
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
|
|
make: *** [Makefile:114: integration] Error 1
|
|
```
|
|
|
|
OpenZaak was demonstrably up — uwsgi had been serving for ~2 minutes — yet
|
|
`curl`/`urllib` to `localhost:8000` from the runner were refused the whole time.
|
|
|
|
**Why** — the same sibling-container split as §1. Compose starts the stack via the
|
|
host daemon, so `ports: ["8000:8000"]` publishes to the *daemon host*, not to the job
|
|
container. From the runner, `localhost:8000` has nothing listening. (`make smoke`
|
|
sidesteps this by polling readiness via `docker inspect` (§2), never a service port.)
|
|
|
|
**Fix** — don't talk to service ports from the runner. Either check state via `docker
|
|
inspect` (health), or run the client **inside the compose network** so it reaches the
|
|
service by name (`http://openzaak:8000`). For a test/seed that needs the repo's own
|
|
code, deliver it via a **built image** (not a bind mount — §1), then
|
|
`docker run --network <stack>_cg …`.
|
|
|
|
**Applied** — `make integration` (ADR-0006) and `make verify-notifications` (ADR-0007)
|
|
do exactly this: they run the seed/test/driver as containers on the stack network and
|
|
reach services by **container IP** (see §6).
|
|
|
|
---
|
|
|
|
## 6. OpenZaak / NRC reject single-label hosts in URLs
|
|
|
|
**Symptom** — talking to OpenZaak or NRC by compose **service name** fails where a URL
|
|
is validated: catalogus/zaaktype filters, the zaak `zaaktype` URL, and abonnement
|
|
`callbackUrl` come back `400 "Voer een geldige URL in."` — even though the host
|
|
resolves and is reachable.
|
|
|
|
**Why** — these apps validate URLs with Django's `URLValidator`, which rejects a
|
|
**single-label** host like `openzaak` or `nrc-web` (no dot, and not `localhost`).
|
|
`localhost` passes (so it's invisible in host-port-based local runs); in-network the
|
|
reality is a service name or an IPv4 literal — and only the IP passes.
|
|
|
|
**Fix** — in-network tooling reaches OpenZaak/NRC by **container IP**
|
|
(`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'`), not
|
|
service name; the notif verify harness also registers the sink callback by IP.
|
|
(`infra/run-acl-integration.sh`, `infra/run-notification-check.sh`.)
|
|
|
|
**Related — abonnement callbacks must enforce auth.** NRC probes a callback when an
|
|
abonnement is registered and refuses it (`no-auth-on-callback-url`) unless it returns
|
|
**401** without the configured `Authorization`. The verify sink
|
|
(`infra/notification-sink.py`) enforces a bearer token for exactly this reason.
|
|
|
|
---
|
|
|
|
## 7. A job with `if: ${{ !cancelled() }}` (or `always()`) + `needs` sticks in "waiting"
|
|
|
|
**Symptom** — after upgrading to **Gitea 1.27** + **act_runner 2.0.0**, one job never
|
|
starts: the run sits in state `waiting` forever, the job has **no logs** (never
|
|
dispatched to a runner), and the other jobs finish normally. `main` stays pending/red.
|
|
Seen on the `verify-stack` job (#134).
|
|
|
|
**Why** — Gitea 1.27 reworked cancellation/aggregation: a job gated by a
|
|
**status-function `if`** (`always()` / `cancelled()` / `!cancelled()`) on top of
|
|
`needs` now routes through a new transitional **`Cancelling`** job state plus a
|
|
server↔runner **capability negotiation** ("Requires Gitea Runner 2.0.0"). On the
|
|
1.27 + 2.0.0 pairing that handshake doesn't resolve for such a job, so it's never
|
|
offered to a runner and never leaves `waiting`. Jobs with no `if`/`needs` are
|
|
unaffected. (Related upstream: go-gitea/gitea#31074, #27116, #35782.)
|
|
|
|
**Fix** — don't gate a `needs` job with a status-function `if`. Use the default
|
|
`if: success()` (i.e. omit the `if`). If you need "run even when an upstream job
|
|
fails", prefer serialising with a `concurrency` group over `needs` + `always()`.
|
|
|
|
**Also** — a run already stuck this way will **not** clear itself; force-cancel it
|
|
from the Actions UI (plain cancel can also stall on this version, #35782). Push the
|
|
workflow fix to produce a fresh run.
|