runbook: the runner's three ephemeral caches and the boot cascade

Three caches on this runner were container-layer only, each found because
something was slow: dind's image store, the trivy vuln DB, and act's action
clones. All three now have real storage.

act clones actions with full history, not shallow — 66.7MB/538 commits for
setup-uv, 24.4MB/222 for actions/checkout, and this workflow uses five. After a
restart that made `Set up job` an 11-minute step with the job container sat
idle running `sleep` while the runner cloned GitHub. Includes the command to
tell those two apart.

Also records the cascade the dind fix creates. A persistent image store makes
dockerd scan on boot — 38s, 2m13s, or over 5 minutes depending on node load —
and two timeouts then fire: dind's startup probe, and the runner image's own
hardcoded `Docker wait timeout of 5m0s`, which the chart cannot configure. The
runner exits 1, restarts, and kills the running job, which looks like every
step failing at once with no error after a green `Set up job`.

It self-heals in about ten minutes at the cost of one CI run, so runner
restarts are now something to do deliberately rather than casually.

RUNBOOK.md

@@ -245,7 +245,47 @@ roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already muta
 workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
 so the precedent and the tooling are both in place.
 
-### 9. Verify the whole loop, not just the green checkmarks
+### 9. The runner's three ephemeral caches, and the boot cascade they cause
+
+Three separate caches on this runner were container-layer only, each found the same way —
+something was slow, and the cause was a cache that had never survived a restart:
+
+| cache | path | fixed by |
+|---|---|---|
+| dind image store | `/var/lib/docker` | hostPath `/var/lib/gitea-dind` |
+| trivy vulnerability DB | `/root/.cache/trivy` in dind | named docker volume |
+| act's action clones | `/root/.cache/act` | hostPath `/var/lib/gitea-act-cache` |
+
+act clones each action with **full history**, not shallow: 66.7MB/538 commits for
+`astral-sh/setup-uv`, 24.4MB/222 commits for `actions/checkout`, and this workflow uses
+five. After a restart that made `Set up job` an 11-minute step in which the job container
+sat idle running `sleep` while the runner cloned GitHub. If a job looks hung in setup, check
+the job container before blaming the network:
+
+```bash
+C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1)
+kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm
+```
+
+Only `sleep` means the work is in the runner, not the job.
+
+**The cascade this creates.** Giving dind a persistent image store made it boot slowly,
+because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes
+depending on node load. Two independent timeouts then fire:
+
+1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold
+   boot has still exceeded it.
+2. **The runner container's own `Docker wait timeout of 5m0s`**, which is internal to the
+   runner image and not configurable from the chart. When dind is late, the runner exits 1
+   and restarts — and that restart kills whatever job was running, which surfaces as every
+   step in the job failing at once with no error in the log, right after a green
+   `Set up job`.
+
+The pair self-heals: the second dind boot is fast because the store is warm, and the runner
+comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and
+one lost CI run. Restart the runner deliberately, not casually.
+
+### 10. Verify the whole loop, not just the green checkmarks
 
 ```bash
 # the digest CI pushed