ci: unstick verify-stack on Gitea 1.27 + act_runner 2.0.0 (closes #134) (#135)
CI / lint (push) Successful in 1m34s
CI / build (push) Successful in 1m30s
CI / unit (push) Successful in 1m47s
CI / frontend (push) Successful in 3m36s
CI / mutation (push) Successful in 7m27s
CI / verify-stack (push) Canceled after 0s

## What & why

After the Gitea 1.27 + act_runner 2.0.0 upgrade, `verify-stack` never starts: the run sits in `waiting` forever with no logs for that job, while the other five jobs pass — so `main` stays pending/red (P0). See #134.

Closes #134

### Root cause

`verify-stack` was the only job gated by a status-function `if` on top of `needs`:

```yaml
verify-stack:
  needs: [mutation]
  if: ${{ !cancelled() }}
```

Gitea 1.27 reworked cancellation/aggregation so that `always()`/`cancelled()`-gated `needs` jobs route through a new transitional **`Cancelling`** state + server↔runner **capability negotiation** ("Requires Gitea Runner 2.0.0"). On this 1.27 + 2.0.0 pairing that handshake doesn't resolve, so the job is never dispatched and never leaves `waiting`. Plain jobs (no `if`/`needs`) are unaffected — exactly the observed pattern. It worked pre-upgrade (old runner).

### Fix

Drop the `if: ${{ !cancelled() }}`; keep `needs: [mutation]`. Default `if: success()` dispatches normally and still serialises the two memory-heavy jobs (OOM avoidance, #126).

**Trade-off:** the `!cancelled()` (added in #127) let verify-stack run even when the mutation ratchet fails. Now a failing mutation skips verify-stack; the fix-and-re-push re-run exercises it, so the signal isn't lost — just deferred to the green-mutation run. If we later want both signals on one run, serialise via a `concurrency` group rather than `needs` + `always()`.

Documented as §7 in `docs/runbooks/gitea-actions-gotchas.md`.

## Note on the stuck run

Run 582 (the #133 merge) will **not** clear itself and must be force-cancelled from the Actions UI (plain cancel can also stall on this version, gitea#35782). This PR's own run is the first real test of the fix — if `verify-stack` dispatches and runs here, the fix holds.

## Definition of Done

- [x] Linked issue (#134).
- [x] Conventional Commit referencing the issue.
- [ ] CI green — this PR's run is the verification (verify-stack must dispatch).
- [x] Runbook updated (gotchas §7).
- [ ] Closed by the merging PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #135
This commit was merged in pull request #135.
This commit is contained in:
not
2026-07-24 11:59:27 +00:00
parent 4698c869f3
commit 849bf4723b
2 changed files with 32 additions and 3 deletions
+7 -3
View File
@@ -144,11 +144,15 @@ jobs:
# they never co-schedule now the runner has capacity >1. A concurrent Stryker run + full-stack
# bring-up + Playwright browser on one host is what OOMs the e2e (commit d5e5fa2, #126). The
# light .NET/frontend jobs have no `needs`, so they still parallelise up to runner capacity.
# `if: !cancelled()` keeps verify-stack running even when the mutation ratchet fails (so we don't
# lose its signal) while still honouring run cancellation from the concurrency group above.
#
# No `if: ${{ !cancelled() }}` here (removed in #134): on Gitea 1.27 + act_runner 2.0.0, a job
# gated by a status-function `if` (always()/cancelled()) on top of `needs` routes through the new
# transitional "Cancelling" state + capability negotiation and never leaves `waiting` — it's never
# dispatched (gitea-actions-gotchas.md §7). Default `if: success()` dispatches normally. Cost: a
# failing mutation ratchet now skips verify-stack instead of running it anyway; the fix-and-re-push
# re-run exercises verify-stack, so we still get the signal.
verify-stack:
needs: [mutation]
if: ${{ !cancelled() }}
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
+25
View File
@@ -196,3 +196,28 @@ service name; the notif verify harness also registers the sink callback by IP.
abonnement is registered and refuses it (`no-auth-on-callback-url`) unless it returns
**401** without the configured `Authorization`. The verify sink
(`infra/notification-sink.py`) enforces a bearer token for exactly this reason.
---
## 7. A job with `if: ${{ !cancelled() }}` (or `always()`) + `needs` sticks in "waiting"
**Symptom** — after upgrading to **Gitea 1.27** + **act_runner 2.0.0**, one job never
starts: the run sits in state `waiting` forever, the job has **no logs** (never
dispatched to a runner), and the other jobs finish normally. `main` stays pending/red.
Seen on the `verify-stack` job (#134).
**Why** — Gitea 1.27 reworked cancellation/aggregation: a job gated by a
**status-function `if`** (`always()` / `cancelled()` / `!cancelled()`) on top of
`needs` now routes through a new transitional **`Cancelling`** job state plus a
server↔runner **capability negotiation** ("Requires Gitea Runner 2.0.0"). On the
1.27 + 2.0.0 pairing that handshake doesn't resolve for such a job, so it's never
offered to a runner and never leaves `waiting`. Jobs with no `if`/`needs` are
unaffected. (Related upstream: go-gitea/gitea#31074, #27116, #35782.)
**Fix** — don't gate a `needs` job with a status-function `if`. Use the default
`if: success()` (i.e. omit the `if`). If you need "run even when an upstream job
fails", prefer serialising with a `concurrency` group over `needs` + `always()`.
**Also** — a run already stuck this way will **not** clear itself; force-cancel it
from the Actions UI (plain cancel can also stall on this version, #35782). Push the
workflow fix to produce a fresh run.