runbook: fix section ordering and a duplicate number

The dind entry landed ahead of the postgres one, and the postgres entry I added
earlier was numbered 7 while 'Verify the whole loop' already was. Now 7
postgres, 8 dind, 9 verify.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 07:39:23 +00:00
parent 0dbb5af1d3
commit c2a27952d1
+46 -46
View File
@@ -153,51 +153,6 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc
To stop a running job: click Cancel in the UI. To stop a running job: click Cancel in the UI.
### 8. OPEN: dind is killed by its own liveness probe
Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.
The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:
```
Liveness probe failed: command timed out:
"/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m)
Killing: Init container dind failed liveness probe
```
The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.
**Why this matters beyond the runner restarting.** Image builds failed with:
ERROR: failed to solve: DeadlineExceeded: no active session for <id>
which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.
Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:
```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
-o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```
**The chart exposes no probe knobs**`helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.
The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.
### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough ### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough
`gitea-postgresql-0` CrashLoopBackOff with: `gitea-postgresql-0` CrashLoopBackOff with:
@@ -245,7 +200,52 @@ kubectl -n longhorn-system get backups.longhorn.io -o json \
Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash
recovery on start. If it cannot, restore the most recent Completed backup instead. recovery on start. If it cannot, restore the most recent Completed backup instead.
### 7. Verify the whole loop, not just the green checkmarks ### 8. OPEN: dind is killed by its own liveness probe
Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.
The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:
```
Liveness probe failed: command timed out:
"/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m)
Killing: Init container dind failed liveness probe
```
The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.
**Why this matters beyond the runner restarting.** Image builds failed with:
ERROR: failed to solve: DeadlineExceeded: no active session for <id>
which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.
Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:
```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
-o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```
**The chart exposes no probe knobs**`helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.
The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.
### 9. Verify the whole loop, not just the green checkmarks
```bash ```bash
# the digest CI pushed # the digest CI pushed