ci(deploy): say why a deploy stalled, and document the cluster prerequisite (refs #175)
The first run against the lab server's VM timed out on `kubectl rollout status` for the in-cluster registry with nothing but "timed out waiting for the condition". The cause was three commands up the runbook: that VM was installed from a stock Talos config, so the only node still carries the control-plane taint and no pod can schedule — and the missing registry mirror would have failed the image pulls right after. `rollout status` can only ever report the symptom, so dump the whole cluster's pods and the recent events on failure instead of `big`'s pods alone; the scheduler's "untolerated taint" message is the answer and it lives in the events. Runbook §9 now opens with the machine-config patch the deploy assumes, as one applied-live patch rather than a `kubectl taint` that the controller undoes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -105,6 +105,12 @@ jobs:
|
|||||||
- name: Smoke the public register
|
- name: Smoke the public register
|
||||||
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
|
run: curl -fsS --retry 10 --retry-delay 6 --retry-all-errors http://localhost:30141/openbaar/register
|
||||||
|
|
||||||
- name: Pods on failure
|
# Cluster-wide, not just `big`: the first thing that can fail is the registry
|
||||||
|
# in its own namespace, and a scheduling problem shows up in the events, not
|
||||||
|
# in `rollout status` — which only ever says "timed out waiting".
|
||||||
|
- name: Pods and events on failure
|
||||||
if: failure()
|
if: failure()
|
||||||
run: kubectl -n big get pods,jobs || true
|
run: |
|
||||||
|
kubectl get pods -A -o wide || true
|
||||||
|
kubectl -n big get jobs || true
|
||||||
|
kubectl get events -A --sort-by=.lastTimestamp | tail -30 || true
|
||||||
|
|||||||
@@ -366,6 +366,32 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
|
|||||||
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
|
every time a PR is squash-merged to `main` (and on demand via *Run workflow*). PR CI is the
|
||||||
merge gate, so the workflow deploys without re-running the checks.
|
merge gate, so the workflow deploys without re-running the checks.
|
||||||
|
|
||||||
|
**Prerequisite: the VM must have been installed with the §1 patch.** A stock Talos config
|
||||||
|
gives you a node that still carries the control-plane taint and knows nothing about the
|
||||||
|
plain-HTTP registry, which the deploy hits in that order: the `registry` pod sits `Pending`
|
||||||
|
until `rollout status` times out, and once that's fixed every repo image fails to pull. Both
|
||||||
|
are one patch, applied live — no reboot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# on the Fedora host
|
||||||
|
cat > /tmp/talos-patch.yaml <<'YAML'
|
||||||
|
cluster:
|
||||||
|
allowSchedulingOnControlPlanes: true
|
||||||
|
machine:
|
||||||
|
certSANs:
|
||||||
|
- <TALOS_VM_IP>
|
||||||
|
registries:
|
||||||
|
mirrors:
|
||||||
|
"<TALOS_VM_IP>:30500":
|
||||||
|
endpoints:
|
||||||
|
- http://<TALOS_VM_IP>:30500
|
||||||
|
YAML
|
||||||
|
talosctl -n <TALOS_VM_IP> -e <TALOS_VM_IP> patch mc --patch @/tmp/talos-patch.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
`allowSchedulingOnControlPlanes` rather than `kubectl taint node --all …-` because the
|
||||||
|
controller puts the taint back; the patch is what makes it stick across a reboot.
|
||||||
|
|
||||||
The cluster's API and registry are not exposed publicly, so the job forwards them over the
|
The cluster's API and registry are not exposed publicly, so the job forwards them over the
|
||||||
same SSH hop the Gitea-runner pipeline uses:
|
same SSH hop the Gitea-runner pipeline uses:
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user