From a8434946271a6646ddd12feb68e9874d7df67791 Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Mon, 20 Jul 2026 14:10:28 +0000 Subject: [PATCH] runbook: the runner's three ephemeral caches and the boot cascade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three caches on this runner were container-layer only, each found because something was slow: dind's image store, the trivy vuln DB, and act's action clones. All three now have real storage. act clones actions with full history, not shallow — 66.7MB/538 commits for setup-uv, 24.4MB/222 for actions/checkout, and this workflow uses five. After a restart that made `Set up job` an 11-minute step with the job container sat idle running `sleep` while the runner cloned GitHub. Includes the command to tell those two apart. Also records the cascade the dind fix creates. A persistent image store makes dockerd scan on boot — 38s, 2m13s, or over 5 minutes depending on node load — and two timeouts then fire: dind's startup probe, and the runner image's own hardcoded `Docker wait timeout of 5m0s`, which the chart cannot configure. The runner exits 1, restarts, and kills the running job, which looks like every step failing at once with no error after a green `Set up job`. It self-heals in about ten minutes at the cost of one CI run, so runner restarts are now something to do deliberately rather than casually. --- RUNBOOK.md | 42 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index ba3ea96..569360f 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -245,7 +245,47 @@ roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already muta workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 — so the precedent and the tooling are both in place. -### 9. Verify the whole loop, not just the green checkmarks +### 9. The runner's three ephemeral caches, and the boot cascade they cause + +Three separate caches on this runner were container-layer only, each found the same way — +something was slow, and the cause was a cache that had never survived a restart: + +| cache | path | fixed by | +|---|---|---| +| dind image store | `/var/lib/docker` | hostPath `/var/lib/gitea-dind` | +| trivy vulnerability DB | `/root/.cache/trivy` in dind | named docker volume | +| act's action clones | `/root/.cache/act` | hostPath `/var/lib/gitea-act-cache` | + +act clones each action with **full history**, not shallow: 66.7MB/538 commits for +`astral-sh/setup-uv`, 24.4MB/222 commits for `actions/checkout`, and this workflow uses +five. After a restart that made `Set up job` an 11-minute step in which the job container +sat idle running `sleep` while the runner cloned GitHub. If a job looks hung in setup, check +the job container before blaming the network: + +```bash +C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1) +kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm +``` + +Only `sleep` means the work is in the runner, not the job. + +**The cascade this creates.** Giving dind a persistent image store made it boot slowly, +because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes +depending on node load. Two independent timeouts then fire: + +1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold + boot has still exceeded it. +2. **The runner container's own `Docker wait timeout of 5m0s`**, which is internal to the + runner image and not configurable from the chart. When dind is late, the runner exits 1 + and restarts — and that restart kills whatever job was running, which surfaces as every + step in the job failing at once with no error in the log, right after a green + `Set up job`. + +The pair self-heals: the second dind boot is fast because the store is warm, and the runner +comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and +one lost CI run. Restart the runner deliberately, not casually. + +### 10. Verify the whole loop, not just the green checkmarks ```bash # the digest CI pushed