From 37b297bf5c07dfe4dd5e020765edf8609a1188db Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Mon, 20 Jul 2026 04:48:48 +0000 Subject: [PATCH] ci: reclaim dind disk after each image build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed. --- .gitea/workflows/ci.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.gitea/workflows/ci.yaml b/.gitea/workflows/ci.yaml index 4d2ed04..f0464f8 100644 --- a/.gitea/workflows/ci.yaml +++ b/.gitea/workflows/ci.yaml @@ -439,6 +439,29 @@ jobs: docker buildx imagetools inspect "${IMAGE}:${GITHUB_SHA}" \ --format '{{.Manifest.Digest}}' + - name: reclaim dind disk + # dind's /var/lib/docker is a hostPath on node2 (see oci-k8s addons role), so + # nothing reclaims it automatically — kubelet's image GC does not manage a nested + # daemon's store. Left alone it grows every run until node2 hits disk pressure and + # starts evicting pods, which looks like a cluster problem rather than a CI one. + # + # `always()`: a failed build still leaves layers behind, and that is exactly when + # the disk is most likely to be the reason it failed. + # + # Deliberately narrow. `docker image prune` WITHOUT -a removes dangling images + # only; with -a it would delete the act runner image, which no container references + # between jobs, and buy back a 1.6GB re-pull on the very next run. The buildx cache + # is the part that actually grows without bound, so it is pruned by age and keeps a + # week — recent enough that `--cache-from` still hits on normal traffic. + # + # Named volumes are never pruned here: that is where the trivy vuln DB lives. + if: always() + run: | + docker image prune -f + docker buildx prune -af --filter until=168h + echo "--- dind disk after prune ---" + docker system df + # --- stage 11: bump the chart's digests. CI's last act. ------------------------------ bump: runs-on: ubuntu-latest