runbook: the 14-minute 'Set up job' failure and its cause
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
ci / bump (push) Blocked by required conditions
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
ci / bump (push) Blocked by required conditions
Every job was failing at Set up job after ~14 minutes, then the first real step died at 0s. It read like a broken action; it was an empty dind image cache re-pulling the 1.6GB act job image on every run. dind has no volume for /var/lib/docker, so the cache lives in its writable layer and dies with every pod restart. A restart-looping runner therefore never keeps one. Warming it by hand took lint from failure to success with no code change.
This commit is contained in:
+28
-1
@@ -97,7 +97,34 @@ kubectl's vendored `golang.org/x/net` and Go stdlib, inside the newest kubectl t
|
|||||||
no version cleared them. kubectl was in that image for exactly one call, and helm already
|
no version cleared them. kubectl was in that image for exactly one call, and helm already
|
||||||
does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.
|
does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.
|
||||||
|
|
||||||
### 5. Verify the whole loop, not just the green checkmarks
|
### 5. When every job fails at `Set up job` after ~14 minutes
|
||||||
|
|
||||||
|
Symptom: `Set up job` runs for 10–15 minutes and succeeds, then the first real step fails
|
||||||
|
instantly at 0s, and every downstream job is skipped. It looks like the action broke.
|
||||||
|
|
||||||
|
Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job
|
||||||
|
image (`ghcr.io/catthehacker/ubuntu:act-24.04`) before it can start. dind has no volume for
|
||||||
|
`/var/lib/docker` — the cache lives in its container writable layer and is destroyed on
|
||||||
|
every pod restart. A runner that is restart-looping therefore never keeps a cache, and each
|
||||||
|
job pays the full pull.
|
||||||
|
|
||||||
|
Check it:
|
||||||
|
```bash
|
||||||
|
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images
|
||||||
|
```
|
||||||
|
Empty output is the diagnosis. Warm it once by hand:
|
||||||
|
```bash
|
||||||
|
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
|
||||||
|
ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9
|
||||||
|
```
|
||||||
|
|
||||||
|
The durable fix is to stop the runner restarting. Its `/data` PVC is ReadWriteOnce, so
|
||||||
|
every reschedule hits `Multi-Attach error` and the pod sits in Init until Longhorn detaches
|
||||||
|
from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` for exactly
|
||||||
|
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
||||||
|
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
||||||
|
|
||||||
|
### 6. Verify the whole loop, not just the green checkmarks
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# the digest CI pushed
|
# the digest CI pushed
|
||||||
|
|||||||
Reference in New Issue
Block a user