diff --git a/RUNBOOK.md b/RUNBOOK.md index ba3ea96..569360f 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -245,7 +245,47 @@ roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already muta workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 — so the precedent and the tooling are both in place. -### 9. Verify the whole loop, not just the green checkmarks +### 9. The runner's three ephemeral caches, and the boot cascade they cause + +Three separate caches on this runner were container-layer only, each found the same way — +something was slow, and the cause was a cache that had never survived a restart: + +| cache | path | fixed by | +|---|---|---| +| dind image store | `/var/lib/docker` | hostPath `/var/lib/gitea-dind` | +| trivy vulnerability DB | `/root/.cache/trivy` in dind | named docker volume | +| act's action clones | `/root/.cache/act` | hostPath `/var/lib/gitea-act-cache` | + +act clones each action with **full history**, not shallow: 66.7MB/538 commits for +`astral-sh/setup-uv`, 24.4MB/222 commits for `actions/checkout`, and this workflow uses +five. After a restart that made `Set up job` an 11-minute step in which the job container +sat idle running `sleep` while the runner cloned GitHub. If a job looks hung in setup, check +the job container before blaming the network: + +```bash +C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1) +kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm +``` + +Only `sleep` means the work is in the runner, not the job. + +**The cascade this creates.** Giving dind a persistent image store made it boot slowly, +because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes +depending on node load. Two independent timeouts then fire: + +1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold + boot has still exceeded it. +2. **The runner container's own `Docker wait timeout of 5m0s`**, which is internal to the + runner image and not configurable from the chart. When dind is late, the runner exits 1 + and restarts — and that restart kills whatever job was running, which surfaces as every + step in the job failing at once with no error in the log, right after a green + `Set up job`. + +The pair self-heals: the second dind boot is fast because the store is warm, and the runner +comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and +one lost CI run. Restart the runner deliberately, not casually. + +### 10. Verify the whole loop, not just the green checkmarks ```bash # the digest CI pushed