ci(deploy): say why a deploy stalled, and document the cluster prerequisite (refs #175)

The first run against the lab server's VM timed out on `kubectl rollout status`
for the in-cluster registry with nothing but "timed out waiting for the
condition". The cause was three commands up the runbook: that VM was installed
from a stock Talos config, so the only node still carries the control-plane
taint and no pod can schedule — and the missing registry mirror would have
failed the image pulls right after.

`rollout status` can only ever report the symptom, so dump the whole cluster's
pods and the recent events on failure instead of `big`'s pods alone; the
scheduler's "untolerated taint" message is the answer and it lives in the
events. Runbook §9 now opens with the machine-config patch the deploy assumes,
as one applied-live patch rather than a `kubectl taint` that the controller
undoes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
not
2026-09-18 16:01:47 +02:00
co-authored by Claude Opus 5
parent fc036d53d5
commit d17e79959b
2 changed files with 34 additions and 2 deletions
+8 -2
View File
@@ -105,6 +105,12 @@ jobs:
- name: Smoke the public register
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
- name: Pods on failure
# Cluster-wide, not just `big`: the first thing that can fail is the registry
# in its own namespace, and a scheduling problem shows up in the events, not
# in `rollout status` — which only ever says "timed out waiting".
- name: Pods and events on failure
if: failure()
run: kubectl -n big get pods,jobs || true
run: |
kubectl get pods -A -o wide || true
kubectl -n big get jobs || true
kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true
+26
View File
@@ -366,6 +366,32 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
merge gate, so the workflow deploys without re-running the checks.
**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
gives you a node that still carries the control-plane taint and knows nothing about the
plain-HTTP registry, which the deploy hits in that order: the `registry` pod sits `Pending`
until `rollout status` times out, and once that's fixed every repo image fails to pull. Both
are one patch, applied live — no reboot:
```bash
# on the Fedora host
cat > /tmp/talos-patch.yaml <<'YAML'
cluster:
allowSchedulingOnControlPlanes: true
machine:
certSANs:
- <TALOS_VM_IP>
registries:
mirrors:
"<TALOS_VM_IP>:30500":
endpoints:
- http://<TALOS_VM_IP>:30500
YAML
talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/talos-patch.yaml
```
`allowSchedulingOnControlPlanes` rather than `kubectl taint node --all …-` because the
controller puts the taint back; the patch is what makes it stick across a reboot.
The cluster's API and registry are not exposed publicly, so the job forwards them over the
same SSH hop the Gitea-runner pipeline uses: