runbook: record the dind liveness probe as an open issue
dind is killed by a probe whose only job is to check a socket exists, with a 1s timeout the nodes cannot always meet: 27 failures over 156 minutes. Worth recording because it probably explains build failures already attributed to something else. `DeadlineExceeded: no active session` was blamed on CPU starvation and addressed by dropping runner capacity to 1; the likelier mechanism is kubelet killing dind mid-build and taking the buildkit session with it. Capacity reduced the load that trips the probe, which fits #17 passing and #18-#21 failing anyway. Written as a hypothesis with the command to confirm it, not as a conclusion. The chart exposes no probe knobs, so the candidate fix is a Kyverno mutation in Ansible, following the existing force-best-effort-cpu precedent.
This commit is contained in:
+45
@@ -153,6 +153,51 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc
|
|||||||
|
|
||||||
To stop a running job: click Cancel in the UI.
|
To stop a running job: click Cancel in the UI.
|
||||||
|
|
||||||
|
### 8. OPEN: dind is killed by its own liveness probe
|
||||||
|
|
||||||
|
Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
|
||||||
|
diagnosed as something else.
|
||||||
|
|
||||||
|
The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:
|
||||||
|
|
||||||
|
```
|
||||||
|
Liveness probe failed: command timed out:
|
||||||
|
"/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m)
|
||||||
|
Killing: Init container dind failed liveness probe
|
||||||
|
```
|
||||||
|
|
||||||
|
The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
|
||||||
|
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
|
||||||
|
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.
|
||||||
|
|
||||||
|
**Why this matters beyond the runner restarting.** Image builds failed with:
|
||||||
|
|
||||||
|
ERROR: failed to solve: DeadlineExceeded: no active session for <id>
|
||||||
|
|
||||||
|
which was attributed to CPU starvation alone and addressed by dropping the runner's
|
||||||
|
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
|
||||||
|
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
|
||||||
|
that trips the probe, which is consistent with run #17 passing — it treated the cause of
|
||||||
|
the trigger, not the trigger. Runs #18-#21 then failed anyway.
|
||||||
|
|
||||||
|
Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
|
||||||
|
kill timestamps against the failed builds:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
|
||||||
|
kubectl -n gitea get pod gitea-actions-runner-0 \
|
||||||
|
-o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match
|
||||||
|
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
|
||||||
|
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.
|
||||||
|
|
||||||
|
The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
|
||||||
|
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
|
||||||
|
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
|
||||||
|
so the precedent and the tooling are both in place.
|
||||||
|
|
||||||
### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough
|
### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough
|
||||||
|
|
||||||
`gitea-postgresql-0` CrashLoopBackOff with:
|
`gitea-postgresql-0` CrashLoopBackOff with:
|
||||||
|
|||||||
Reference in New Issue
Block a user