gitea_admin/svcforge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

runbook: fix section ordering and a duplicate number

Browse Source 
The dind entry landed ahead of the postgres one, and the postgres entry I added
earlier was numbered 7 while 'Verify the whole loop' already was. Now 7
postgres, 8 dind, 9 verify.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 07:39:23 +00:00
parent 0dbb5af1d3
commit c2a27952d1
1 changed files with 46 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RUNBOOK.md
+46 -46
View File
@@ -153,51 +153,6 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc
To stop a running job: click Cancel in the UI.
### 8. OPEN: dind is killed by its own liveness probe
Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.
The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:
```
Liveness probe failed: command timed out:
"/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m)
Killing: Init container dind failed liveness probe
```
The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.
**Why this matters beyond the runner restarting.** Image builds failed with:
ERROR: failed to solve: DeadlineExceeded: no active session for <id>
which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.
Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:
```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
-o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```
**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.
The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.
### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough
`gitea-postgresql-0` CrashLoopBackOff with:
@@ -245,7 +200,52 @@ kubectl -n longhorn-system get backups.longhorn.io -o json \
Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash
recovery on start. If it cannot, restore the most recent Completed backup instead.
### 7. Verify the whole loop, not just the green checkmarks
### 8. OPEN: dind is killed by its own liveness probe
Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.
The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:
```
Liveness probe failed: command timed out:
"/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m)
Killing: Init container dind failed liveness probe
```
The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.
**Why this matters beyond the runner restarting.** Image builds failed with:
ERROR: failed to solve: DeadlineExceeded: no active session for <id>
which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.
Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:
```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
-o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```
**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.
The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.
### 9. Verify the whole loop, not just the green checkmarks
```bash
# the digest CI pushed