From d17e79959b822fccca74d80bdeb570778bf34cd8 Mon Sep 17 00:00:00 2001 From: Niek Otten Date: Fri, 18 Sep 2026 16:01:47 +0200 Subject: [PATCH] ci(deploy): say why a deploy stalled, and document the cluster prerequisite (refs #175) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first run against the lab server's VM timed out on `kubectl rollout status` for the in-cluster registry with nothing but "timed out waiting for the condition". The cause was three commands up the runbook: that VM was installed from a stock Talos config, so the only node still carries the control-plane taint and no pod can schedule — and the missing registry mirror would have failed the image pulls right after. `rollout status` can only ever report the symptom, so dump the whole cluster's pods and the recent events on failure instead of `big`'s pods alone; the scheduler's "untolerated taint" message is the answer and it lives in the events. Runbook §9 now opens with the machine-config patch the deploy assumes, as one applied-live patch rather than a `kubectl taint` that the controller undoes. Co-Authored-By: Claude Opus 5 (1M context) --- .gitea/workflows/deploy.yaml | 10 ++++++++-- docs/runbooks/kubernetes-talos.md | 26 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.gitea/workflows/deploy.yaml b/.gitea/workflows/deploy.yaml index 8e9d5f6..818830c 100644 --- a/.gitea/workflows/deploy.yaml +++ b/.gitea/workflows/deploy.yaml @@ -105,6 +105,12 @@ jobs: - name: Smoke the public register run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register - - name: Pods on failure + # Cluster-wide, not just `big`: the first thing that can fail is the registry + # in its own namespace, and a scheduling problem shows up in the events, not + # in `rollout status` — which only ever says "timed out waiting". + - name: Pods and events on failure if: failure() - run: kubectl -n big get pods,jobs || true + run: | + kubectl get pods -A -o wide || true + kubectl -n big get jobs || true + kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true diff --git a/docs/runbooks/kubernetes-talos.md b/docs/runbooks/kubernetes-talos.md index cf73ea9..4825156 100644 --- a/docs/runbooks/kubernetes-talos.md +++ b/docs/runbooks/kubernetes-talos.md @@ -366,6 +366,32 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job` every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the merge gate, so the workflow deploys without re-running the checks. +**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config +gives you a node that still carries the control-plane taint and knows nothing about the +plain-HTTP registry, which the deploy hits in that order: the `registry` pod sits `Pending` +until `rollout status` times out, and once that's fixed every repo image fails to pull. Both +are one patch, applied live — no reboot: + +```bash +# on the Fedora host +cat > /tmp/talos-patch.yaml <<'YAML' +cluster: + allowSchedulingOnControlPlanes: true +machine: + certSANs: + - + registries: + mirrors: + ":30500": + endpoints: + - http://:30500 +YAML +talosctl -n -e patch mc --patch @/tmp/talos-patch.yaml +``` + +`allowSchedulingOnControlPlanes` rather than `kubectl taint node --all …-` because the +controller puts the taint back; the patch is what makes it stick across a reboot. + The cluster's API and registry are not exposed publicly, so the job forwards them over the same SSH hop the Gitea-runner pipeline uses: