From c2a27952d1895cc2c9d5385e3f9d5ee16a243d21 Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Mon, 20 Jul 2026 07:39:23 +0000 Subject: [PATCH] runbook: fix section ordering and a duplicate number The dind entry landed ahead of the postgres one, and the postgres entry I added earlier was numbered 7 while 'Verify the whole loop' already was. Now 7 postgres, 8 dind, 9 verify. --- RUNBOOK.md | 92 +++++++++++++++++++++++++++--------------------------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 551117d..ba3ea96 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -153,51 +153,6 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc To stop a running job: click Cancel in the UI. -### 8. OPEN: dind is killed by its own liveness probe - -Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were -diagnosed as something else. - -The runner sits at `Init:1/2` and its dind sidecar accumulates restarts: - -``` -Liveness probe failed: command timed out: - "/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m) -Killing: Init container dind failed liveness probe -``` - -The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`. -`test -S` only asks whether a socket exists. When that cannot finish inside a second, the -node is starved rather than dind being unhealthy, and kubelet kills a working daemon. - -**Why this matters beyond the runner restarting.** Image builds failed with: - - ERROR: failed to solve: DeadlineExceeded: no active session for - -which was attributed to CPU starvation alone and addressed by dropping the runner's -`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed -dind mid-build and the buildkit session died with it. Lowering capacity reduced the load -that trips the probe, which is consistent with run #17 passing — it treated the cause of -the trigger, not the trigger. Runs #18-#21 then failed anyway. - -Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the -kill timestamps against the failed builds: - -```bash -kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events: -kubectl -n gitea get pod gitea-actions-runner-0 \ - -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}' -``` - -**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match -for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is -reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem. - -The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to -roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates -workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 — -so the precedent and the tooling are both in place. - ### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough `gitea-postgresql-0` CrashLoopBackOff with: @@ -245,7 +200,52 @@ kubectl -n longhorn-system get backups.longhorn.io -o json \ Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash recovery on start. If it cannot, restore the most recent Completed backup instead. -### 7. Verify the whole loop, not just the green checkmarks +### 8. OPEN: dind is killed by its own liveness probe + +Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were +diagnosed as something else. + +The runner sits at `Init:1/2` and its dind sidecar accumulates restarts: + +``` +Liveness probe failed: command timed out: + "/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m) +Killing: Init container dind failed liveness probe +``` + +The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`. +`test -S` only asks whether a socket exists. When that cannot finish inside a second, the +node is starved rather than dind being unhealthy, and kubelet kills a working daemon. + +**Why this matters beyond the runner restarting.** Image builds failed with: + + ERROR: failed to solve: DeadlineExceeded: no active session for + +which was attributed to CPU starvation alone and addressed by dropping the runner's +`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed +dind mid-build and the buildkit session died with it. Lowering capacity reduced the load +that trips the probe, which is consistent with run #17 passing — it treated the cause of +the trigger, not the trigger. Runs #18-#21 then failed anyway. + +Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the +kill timestamps against the failed builds: + +```bash +kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events: +kubectl -n gitea get pod gitea-actions-runner-0 \ + -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}' +``` + +**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match +for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is +reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem. + +The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to +roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates +workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 — +so the precedent and the tooling are both in place. + +### 9. Verify the whole loop, not just the green checkmarks ```bash # the digest CI pushed