ci(deploy): say why a deploy stalled, and document the cluster prerequisite (refs #175)

The first run against the lab server's VM timed out on `kubectl rollout status`
for the in-cluster registry with nothing but "timed out waiting for the
condition". The cause was three commands up the runbook: that VM was installed
from a stock Talos config, so the only node still carries the control-plane
taint and no pod can schedule — and the missing registry mirror would have
failed the image pulls right after.

`rollout status` can only ever report the symptom, so dump the whole cluster's
pods and the recent events on failure instead of `big`'s pods alone; the
scheduler's "untolerated taint" message is the answer and it lives in the
events. Runbook §9 now opens with the machine-config patch the deploy assumes,
as one applied-live patch rather than a `kubectl taint` that the controller
undoes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
not
2026-09-18 16:01:47 +02:00
co-authored by Claude Opus 5
parent fc036d53d5
commit d17e79959b
2 changed files with 34 additions and 2 deletions
+8 -2
View File
@@ -105,6 +105,12 @@ jobs:
- name: Smoke the public register - name: Smoke the public register
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
- name: Pods on failure # Cluster-wide, not just `big`: the first thing that can fail is the registry
# in its own namespace, and a scheduling problem shows up in the events, not
# in `rollout status` — which only ever says "timed out waiting".
- name: Pods and events on failure
if: failure() if: failure()
run: kubectl -n big get pods,jobs || true run: |
kubectl get pods -A -o wide || true
kubectl -n big get jobs || true
kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true
+26
View File
@@ -366,6 +366,32 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
merge gate, so the workflow deploys without re-running the checks. merge gate, so the workflow deploys without re-running the checks.
**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
gives you a node that still carries the control-plane taint and knows nothing about the
plain-HTTP registry, which the deploy hits in that order: the `registry` pod sits `Pending`
until `rollout status` times out, and once that's fixed every repo image fails to pull. Both
are one patch, applied live — no reboot:
```bash
# on the Fedora host
cat > /tmp/talos-patch.yaml <<'YAML'
cluster:
allowSchedulingOnControlPlanes: true
machine:
certSANs:
- <TALOS_VM_IP>
registries:
mirrors:
"<TALOS_VM_IP>:30500":
endpoints:
- http://<TALOS_VM_IP>:30500
YAML
talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/talos-patch.yaml
```
`allowSchedulingOnControlPlanes` rather than `kubectl taint node --all …-` because the
controller puts the taint back; the patch is what makes it stick across a reboot.
The cluster's API and registry are not exposed publicly, so the job forwards them over the The cluster's API and registry are not exposed publicly, so the job forwards them over the
same SSH hop the Gitea-runner pipeline uses: same SSH hop the Gitea-runner pipeline uses: