docs(infra): tighten gitea-actions-gotchas, add local compose (refs #30)
Restructure for scannability: a shared root-cause intro, a quick-reference table (gotcha → fix → where), and consistent Symptom/Why/Fix sections with tighter prose. Documents infra/docker-compose.local.yml as the no-make/Windows path and drops the now-stale "no bind mounts remain" line (the local compose uses them, which is fine locally). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,114 +1,100 @@
|
||||
# Gitea Actions gotchas
|
||||
|
||||
Known differences between Gitea Actions (our CI) and a plain local run, and the
|
||||
workarounds we adopted. Referenced by `CLAUDE.md` §8.7 and §15.
|
||||
How our CI (Gitea Actions on the hosted **`ubuntu-latest`** runner) differs from a
|
||||
local run, and the workarounds in this repo. Referenced by `CLAUDE.md` §8.7/§15.
|
||||
|
||||
## Bind mounts don't reach Compose services on the hosted runner
|
||||
**One root cause sits under most of this:** the runner executes the job **inside a
|
||||
container**, so when a step runs `docker compose up`, Compose starts the stack as
|
||||
**sibling containers** on the host's daemon. Anything that assumes the job and
|
||||
those containers share a filesystem — or a `localhost` — breaks.
|
||||
|
||||
**Symptom.** `make smoke` is green locally but the `compose-smoke` CI job fails
|
||||
with the OpenZaak init container exiting 1:
|
||||
| Gotcha | Fix | Lives in |
|
||||
|---|---|---|
|
||||
| Bind-mounted config arrives empty | `docker cp` config into external volumes | `infra/seed-config.sh` |
|
||||
| `docker compose up --wait` is unsupported / flaky | poll health with `docker inspect` | `infra/wait-healthy.sh` |
|
||||
| `pg_isready` passes before PostGIS is ready | add a `PostGIS_Version()` probe | the db healthchecks |
|
||||
|
||||
---
|
||||
|
||||
## 1. Bind mounts don't reach the containers
|
||||
|
||||
**Symptom** — green locally, but `compose-smoke` fails with:
|
||||
|
||||
```
|
||||
oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.
|
||||
```
|
||||
|
||||
Migrations run fine; only the step that reads a **mounted** file fails. The same
|
||||
class of failure hits any service that bind-mounts a workspace path —
|
||||
`nrc-init` (its `data.yaml`), `flowable-init` (the BPMN), `keycloak` (the realm
|
||||
import dir).
|
||||
Migrations run fine; only the step that reads a *mounted* file fails. The same
|
||||
trap hits `nrc-init`, `flowable-init`, and `keycloak`.
|
||||
|
||||
**Cause.** The `ubuntu-latest` runner executes the whole job **inside a
|
||||
container** (`docker.gitea.com/runner-images:ubuntu-latest`). When the job then
|
||||
runs `docker compose ... up`, Compose talks to the host's Docker daemon and
|
||||
starts the stack as **sibling containers**. A relative bind mount such as
|
||||
**Why** — a relative bind mount like `./openzaak/setup_configuration:/app/...` is
|
||||
resolved by Compose to a path *inside the job container*
|
||||
(`/workspace/.../setup_configuration`). The daemon then looks for that path on
|
||||
*its own host*, doesn't find it, and mounts an **empty directory**. (It works on a
|
||||
runner that executes jobs on the host — which is why moving to `ubuntu-latest`
|
||||
exposed it.)
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- ./openzaak/setup_configuration:/app/setup_configuration:ro
|
||||
```
|
||||
**Fix** — use the upstream images verbatim (no build) and stream config into
|
||||
**external named volumes** with `docker cp`, which copies over the Docker API and
|
||||
so works wherever the daemon runs. `infra/seed-config.sh` creates each volume,
|
||||
mounts it in a throwaway helper, and copies the files in:
|
||||
|
||||
is resolved by Compose to an absolute path **inside the job container**
|
||||
(`/workspace/eho/register-referentie/infra/openzaak/setup_configuration`). The
|
||||
daemon then looks for that path on **its own host**, doesn't find it, and
|
||||
auto-creates an **empty directory** to mount. The container starts with an empty
|
||||
mount point, so the file appears "missing".
|
||||
|
||||
This is the classic Docker-in-Docker / sibling-container bind-mount trap. It does
|
||||
not happen on a runner that executes jobs directly on the host (the previous
|
||||
self-hosted `respellion-linux` setup), which is why switching to `ubuntu-latest`
|
||||
exposed it.
|
||||
|
||||
**Fix: the upstream images are used verbatim (no build); config is streamed into
|
||||
external named volumes with `docker cp`.** `infra/seed-config.sh` creates a fixed-
|
||||
name volume per asset, runs a throwaway helper container that mounts it, and
|
||||
`docker cp`s the files in. `docker cp` streams bytes over the Docker API, so it
|
||||
works no matter where the daemon runs (including Docker-in-Docker). The services
|
||||
then mount those volumes:
|
||||
|
||||
| Asset | External volume | Mounted by → at |
|
||||
| Asset | Volume | Mounted at |
|
||||
|---|---|---|
|
||||
| OpenZaak `setup_configuration/data.yaml` | `rr-oz-config` | `oz-init` → `/app/setup_configuration` |
|
||||
| Keycloak realm exports | `rr-kc-realms` | `keycloak` → `/opt/keycloak/data/import` |
|
||||
| `workflows/registratie.bpmn` | `rr-fl-bpmn` | `flowable-init` → `/work` |
|
||||
| OpenZaak `data.yaml` | `rr-oz-config` | `oz-init:/app/setup_configuration` |
|
||||
| Keycloak realms | `rr-kc-realms` | `keycloak:/opt/keycloak/data/import` |
|
||||
| `registratie.bpmn` | `rr-fl-bpmn` | `flowable-init:/work` |
|
||||
|
||||
The volumes are declared `external: true` with fixed `name:`s so they resolve
|
||||
identically under docker compose and podman-compose. The seed step (`make` runs
|
||||
it before every `up`) recreates them fresh each time; `make down` / the
|
||||
per-service `*-down` targets remove them. Open Notificaties needs no config at all
|
||||
— `nrc-init` runs migrations only.
|
||||
The volumes are `external: true` with fixed names, so they resolve identically
|
||||
under docker compose and podman-compose. `make` seeds before every `up`; `make
|
||||
down` removes them. (Open Notificaties needs nothing — `nrc-init` migrates only.)
|
||||
|
||||
**Two hard constraints drove this design:**
|
||||
**Consequence — bare `docker compose up` can't self-seed external volumes:**
|
||||
|
||||
- Use plain `docker volume create` / `docker run` / `docker cp` — **not**
|
||||
`docker compose create`, which **podman-compose** (the local dev runtime) does
|
||||
not implement.
|
||||
- `docker cp` rather than a bind mount of the source dir, because that bind mount
|
||||
is exactly what fails on the containerized runner.
|
||||
- **CI / Linux / macOS:** `make up` or `make smoke` (seed, then start).
|
||||
- **No-make / Windows:** `infra/docker-compose.local.yml` — a twin stack that
|
||||
**bind-mounts** the config instead. Bind mounts are fine *locally* because a
|
||||
local daemon can see your working directory, so
|
||||
`docker compose -f infra/docker-compose.local.yml up -d` just works.
|
||||
|
||||
**Consequence: bare `docker compose up` no longer self-seeds** — the `external`
|
||||
volumes must be populated first, so use **`make up`** (or `make <svc>-up`), which
|
||||
seeds then starts. CI uses `make smoke`, which does the same.
|
||||
**Why not the obvious alternatives**
|
||||
|
||||
**Why not the alternatives.**
|
||||
- *Bake config into an image* (incl. an inline Dockerfile) — `docker compose up`
|
||||
would then work unaided, but it's a build; we wanted the upstream images as-is.
|
||||
- *Compose `configs:` with inline `content`* — Compose writes a client-side temp
|
||||
file and bind-mounts it, hitting the exact same problem.
|
||||
- *A host-executing runner* — bind mounts would work with zero seeding, but it
|
||||
reintroduces a self-hosted runner and undoes the move to `ubuntu-latest`.
|
||||
|
||||
- *Bake into a (possibly inline) image* — clean and portable, but it's a build;
|
||||
rejected here because the goal was to use the upstream images verbatim.
|
||||
- *Compose `configs:` with inline `content`* — Compose materialises these as a
|
||||
temp file on the **client** side and bind-mounts it → same daemon-can't-see-it
|
||||
problem.
|
||||
- *A self-hosted runner that runs jobs on the host* — bind mounts would then work
|
||||
with zero seeding, but it reintroduces a bespoke runner and undoes the move to
|
||||
the hosted `ubuntu-latest` label.
|
||||
---
|
||||
|
||||
No bind mounts of these config files remain, so the SELinux `:z`/`:Z` relabel flag
|
||||
is no longer needed anywhere in `infra/` (named volumes don't need relabeling).
|
||||
## 2. Readiness: poll health, don't use `--wait`
|
||||
|
||||
## Readiness: a portable health poll, not `docker compose up --wait`
|
||||
`docker compose up --wait` looks ideal but fails us three ways:
|
||||
|
||||
The smoke does **not** use `docker compose up --wait`, for three reasons:
|
||||
- **podman-compose doesn't implement it** (`unrecognized arguments: --wait`) — so
|
||||
it would break local dev.
|
||||
- A project-wide `--wait` **treats a one-shot exiting `0` as a failure** unless
|
||||
something `depends_on` it with `service_completed_successfully`. `flowable-init`
|
||||
deploys the BPMN and exits with no dependant, so `--wait` fails the moment it
|
||||
does — last line `container infra-flowable-init-1 exited (0)`.
|
||||
- The containerized runner **can't reach published host ports**, so an external
|
||||
`curl localhost:8080/health` can't work either.
|
||||
|
||||
- **podman-compose doesn't implement `--wait`** (`unrecognized arguments:
|
||||
--wait`), so it would break local dev.
|
||||
- A whole-project `--wait` **fails when a one-shot with no
|
||||
`service_completed_successfully` dependant exits** — `flowable-init` deploys
|
||||
the BPMN and exits 0, which `--wait` treats as the project failing (symptom:
|
||||
last compose line `container infra-flowable-init-1 exited (0)`).
|
||||
- The containerized CI runner **can't reach published host ports**, so an
|
||||
external `curl localhost:8080/health` doesn't work either.
|
||||
**Fix** — `infra/wait-healthy.sh` polls each durable service (`openzaak nrc-web
|
||||
acl bff`, listed as `WAIT_SVCS` in the `Makefile`) with `docker ps` + `docker
|
||||
inspect '{{.State.Health.Status}}'` until it reports `healthy`. It uses only
|
||||
primitives both runtimes support, reads the **in-container** healthcheck (no host
|
||||
port needed), and ignores the one-shots (they only need to have run).
|
||||
`WAIT_TIMEOUT` defaults to 420 s — enough for the cold OpenZaak migrate (~90 s)
|
||||
plus app start.
|
||||
|
||||
**Fix.** `infra/wait-healthy.sh` polls each durable, health-checked service
|
||||
(`openzaak nrc-web acl bff` — `WAIT_SVCS` in the `Makefile`) with `docker ps` +
|
||||
`docker inspect '{{.State.Health.Status}}'`, waiting for `healthy`. That uses
|
||||
only primitives both docker compose and podman-compose support, reads the
|
||||
in-container healthcheck (no host port needed), and ignores the one-shots (they
|
||||
only need to have run). `WAIT_TIMEOUT` (default 420 s) covers the cold
|
||||
OpenZaak migrate (~90 s) plus app start.
|
||||
---
|
||||
|
||||
## PostGIS readiness vs. `pg_isready`
|
||||
## 3. `pg_isready` passes before PostGIS is ready
|
||||
|
||||
`pg_isready` reports the server is accepting connections as soon as the TCP port
|
||||
is open — **before** the `postgis/postgis` image has finished running its
|
||||
`CREATE EXTENSION postgis` init scripts. An init container that starts migrating
|
||||
in that window can fail on a missing PostGIS. The db healthchecks therefore add a
|
||||
`SELECT PostGIS_Version()` probe so dependents wait for the extension, not just
|
||||
the port.
|
||||
`pg_isready` succeeds as soon as the TCP port is open — *before* the
|
||||
`postgis/postgis` image has finished running `CREATE EXTENSION postgis`. An init
|
||||
container that starts migrating in that window can fail on a missing PostGIS. So
|
||||
the db healthchecks add a `SELECT PostGIS_Version()` probe, making dependents wait
|
||||
for the extension, not just the port.
|
||||
|
||||
Reference in New Issue
Block a user