ci: reclaim dind disk after each image build
ci / lint (push) Successful in 43s
ci / types (push) Successful in 1m29s
ci / unit (push) Successful in 59s
ci / security (push) Successful in 1m10s
ci / dockerfile (push) Successful in 24s
ci / chart (push) Successful in 9s
ci / integration (push) Successful in 1m50s
ci / image (api) (push) Successful in 4m57s
ci / image (reconciler) (push) Successful in 2m55s
ci / image (worker) (push) Successful in 2m58s
ci / bump (push) Successful in 23s

dind's /var/lib/docker is now a hostPath on node2 rather than the container's
writable layer, so it survives restarts — and nothing reclaims it. Kubelet's
image GC does not manage a nested daemon's store, so left alone it grows every
run until node2 hits disk pressure and evicts pods, which reads as a cluster
problem rather than a CI one.

The step is deliberately narrow. `docker image prune` without -a removes
dangling images only; with -a it would delete the act runner image, which no
container references between jobs, and buy back a 1.6GB re-pull on the next
run. The buildx cache is what actually grows without bound, so it is pruned by
age keeping a week, recent enough that --cache-from still hits. Named volumes
are never pruned, since that is where the trivy vuln DB lives.

always(), because a failed build still leaves layers behind, and that is when
disk is most likely to have been the reason it failed.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 04:48:48 +00:00
parent 5d7f46483e
commit 37b297bf5c
+23
View File
@@ -439,6 +439,29 @@ jobs:
docker buildx imagetools inspect "${IMAGE}:${GITHUB_SHA}" \
--format '{{.Manifest.Digest}}'
- name: reclaim dind disk
# dind's /var/lib/docker is a hostPath on node2 (see oci-k8s addons role), so
# nothing reclaims it automatically — kubelet's image GC does not manage a nested
# daemon's store. Left alone it grows every run until node2 hits disk pressure and
# starts evicting pods, which looks like a cluster problem rather than a CI one.
#
# `always()`: a failed build still leaves layers behind, and that is exactly when
# the disk is most likely to be the reason it failed.
#
# Deliberately narrow. `docker image prune` WITHOUT -a removes dangling images
# only; with -a it would delete the act runner image, which no container references
# between jobs, and buy back a 1.6GB re-pull on the very next run. The buildx cache
# is the part that actually grows without bound, so it is pruned by age and keeps a
# week — recent enough that `--cache-from` still hits on normal traffic.
#
# Named volumes are never pruned here: that is where the trivy vuln DB lives.
if: always()
run: |
docker image prune -f
docker buildx prune -af --filter until=168h
echo "--- dind disk after prune ---"
docker system df
# --- stage 11: bump the chart's digests. CI's last act. ------------------------------
bump:
runs-on: ubuntu-latest