Get off Docker Hub, add a catalog values passthrough, fix the dind prune
ci / lint (push) Successful in 24s
ci / types (push) Successful in 34s
ci / unit (push) Successful in 26s
ci / security (push) Successful in 37s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 7s
ci / integration (push) Successful in 41s
ci / image (api) (push) Successful in 2m9s
ci / image (reconciler) (push) Successful in 2m1s
ci / image (worker) (push) Successful in 2m7s
ci / bump (push) Successful in 13s

Docker Hub rate-limits anonymous pulls per source IP and every node here
shares one NAT address, so a busy afternoon fails an unrelated build with
`toomanyrequests`. Nothing in this repo needs to be there.

Every base image now comes from mirror.gcr.io (python, alpine/helm,
postgres) or ghcr.io (uv, trivy). Verified digest-for-digest against
Docker Hub before switching, including the superseded postgres digest
this repo still pins, so every existing pin stays valid — same bytes,
different transport.

catalog.yaml: the three bitnami entries named `bitnamilegacy/<chart>`, a
repo alias nothing in the worker image configures, so they could never
resolve at provision time. All five entries are now `oci://` refs, which
need no `helm repo add`, and all are on latest stable:

  elasticsearch 21.3.15 -> 22.1.6     redis    20.6.2 -> 27.0.15
  postgresql    16.4.5  -> 18.8.0     podinfo  6.7.1  -> 6.14.0

Moving the chart pull is only half of it, though: a bitnami chart
defaults its own images to registry-1.docker.io. CatalogEntry gains a
`values:` dict, merged under the size's replicas and resources, so an
entry can set `global.imageRegistry` and move the image pull too. Size
wins on conflict — otherwise an entry setting replicaCount would make
every size deploy the same shape. Deep merge, because a shallow one
drops sibling keys of a shared nested map.

Bitnami charts reject a substituted registry unless
`global.security.allowInsecureImages` is set. That check is about
provenance, and the mirror serves byte-identical manifests, so it is set
deliberately and only for entries whose digests were verified.

The dind prune had `--filter until=168h` on both prunes, and it got both
cases exactly backwards. `until` reads an image's CREATED time, so it
deleted trivy every leg (a released tool image is always older than any
window) while protecting the dangling build layers it existed to remove.
Measured on node0: 21 dangling images / 5.96GB, and exactly 1 of them
older than 168h. Trivy is protected by a tag now, so the image prune
drops the filter; buildx keeps it, where age genuinely matters.

Tests: +10 unit (deep merge, precedence, no-mutation, and a guard that
fails if any catalog entry points at Docker Hub). Both new guards were
control-tested by breaking the code and watching them fail. The API test
that hardcoded `21.3.15` now reads the catalog — its subject is where
the value comes from, not what it is.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 15:32:12 +00:00
parent 58ffb9c2e0
commit 08a529fa63
10 changed files with 267 additions and 40 deletions
+18 -15
View File
@@ -103,7 +103,7 @@ jobs:
needs: [unit]
services:
postgres:
image: postgres:18@sha256:32ca0af8e77bfb8c6610c488e4691f83f972a3e9e64d3b02facf3ab111ad5500
image: mirror.gcr.io/library/postgres:18@sha256:32ca0af8e77bfb8c6610c488e4691f83f972a3e9e64d3b02facf3ab111ad5500
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: svcforge
@@ -432,7 +432,7 @@ jobs:
# place rather than accumulating versions, so the volume stays at roughly one DB,
# and the weekly prune CronJob reclaims it at the cost of one re-download.
env:
TRIVY: aquasec/trivy:0.72.0@sha256:cffe3f5161a47a6823fbd23d985795b3ed72a4c806da4c4df16266c02accdd6f
TRIVY: ghcr.io/aquasecurity/trivy:0.72.0@sha256:cffe3f5161a47a6823fbd23d985795b3ed72a4c806da4c4df16266c02accdd6f
run: |
# Pull by digest, then give it a local tag, and run the TAG.
#
@@ -485,9 +485,9 @@ jobs:
--format '{{.Manifest.Digest}}'
- name: reclaim dind disk
# dind's /var/lib/docker is a hostPath on node2 (see oci-k8s addons role), so
# dind's /var/lib/docker is a hostPath on node0 (see oci-k8s addons role), so
# nothing reclaims it automatically — kubelet's image GC does not manage a nested
# daemon's store. Left alone it grows every run until node2 hits disk pressure and
# daemon's store. Left alone it grows every run until node0 hits disk pressure and
# starts evicting pods, which looks like a cluster problem rather than a CI one.
#
# `always()`: a failed build still leaves layers behind, and that is exactly when
@@ -495,26 +495,29 @@ jobs:
#
# Deliberately narrow. `docker image prune` WITHOUT -a removes dangling images
# only; with -a it would delete the act runner image, which no container references
# between jobs, and buy back a 1.6GB re-pull on the very next run. The buildx cache
# is the part that actually grows without bound, so it is pruned by age and keeps a
# week — recent enough that `--cache-from` still hits on normal traffic.
# between jobs, and buy back a 1.6GB re-pull on the very next run.
#
# Named volumes are never pruned here: that is where the trivy vuln DB lives.
#
# "Dangling" catches more than it looks: an image pulled by digest has no tag, so it
# is dangling as soon as its container exits. That is why the trivy scan step tags
# its image — an age filter does NOT protect it, because `until` reads the image's
# created time, and a released tool image is always older than any useful window.
# Run #68 proved that: trivy was untagged and deleted in all three legs despite
# `until=168h`, and re-pulled 178MB each time.
# NO age filter on the image prune, and that is the whole point of this comment.
# `--filter until=168h` reads an image's CREATED time, so it got both cases exactly
# backwards: it deleted trivy every leg (a released tool image is always older than
# any window — run #68, 178MB re-pulled three times) while protecting the dangling
# build layers it was added to remove (they are minutes old). Measured on node0
# afterwards: 21 dangling images, 5.96GB, 19 of them created inside 25 hours, none
# of them reclaimable while the filter was there. Tagging trivy is what protects
# trivy; nothing needs to protect a dangling layer from the current build, because
# `always()` runs this after that build has already been pushed.
#
# The buildx cache keeps its age filter: a week is recent enough that `--cache-from`
# still hits on normal traffic, and that cache really does grow by age.
#
# The age filter stays anyway, for the buildx cache, which really does grow by age.
# The act runner image survives only because this step runs inside an act container,
# so the image is in use exactly while the prune runs. That is luck, not design; if
# it ever starts disappearing, tag it the same way.
if: always()
run: |
docker image prune -f --filter until=168h
docker image prune -f
docker buildx prune -af --filter until=168h
echo "--- dind disk after prune ---"
docker system df