ci: move buildx layer cache to the registry

act_runner's cache PVC is 1Gi and also holds .runner, the runner's own
registration file. --cache-to type=gha,mode=max for three images is several GB;
filling that volume breaks the runner, not just the cache. act_runner also
evicts by age with no size cap, so it fills whatever it is given.

type=registry has no such limit and lives beside the images it caches. The uv
cache still uses the runner's cache service, which is a few hundred MB.

Registry login is no longer gated to master: the build now reads and writes the
cache on every run. Pushing the release image keeps its own master-only gate.

.gitea/workflows/ci.yaml

@@ -211,7 +211,9 @@ jobs:
       - uses: docker/setup-buildx-action@c47758b77c9736f4b2ef4073d4d51994fabfe349 # v3.7.1
 
       - name: registry login
-        if: github.ref == 'refs/heads/master' && github.event_name == 'push'
+        # Not gated to master any more: the build step now reads AND writes the layer cache
+        # in the registry, so every run needs credentials. Pushing the release image is
+        # still master-only — that gate lives on the `push by digest` step, where it belongs.
         uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
         with:
           registry: ${{ env.REGISTRY }}
@@ -226,27 +228,60 @@ jobs:
         # Loaded locally, not pushed. Trivy scans this exact image next; only then does it
         # get pushed. The alternative — push, scan, and hope nobody pulled meanwhile — is
         # how a CRITICAL ends up in the registry with a green checkmark next to it.
+        #
+        # The layer cache is `type=registry`, NOT `type=gha`. Two reasons, both hard:
+        #
+        # 1. act_runner's cache server is backed by a 1Gi PVC that also holds `.runner`,
+        #    the runner's own registration file. `mode=max` stores every intermediate layer
+        #    of three images — several GB. Filling that volume does not merely lose the
+        #    cache: the runner cannot write its state and has to be re-registered by hand.
+        #    Trading "slow CI" for "broken CI" is not a trade.
+        # 2. act_runner evicts by AGE, with no size cap in its config. It will fill whatever
+        #    it is given and then wedge. The registry has no such limit and already holds
+        #    the images anyway.
+        #
+        # The uv/pip cache still uses the runner's cache service — that one is a few hundred
+        # MB and fits.
         run: |
           docker buildx build \
             -f services/${{ matrix.svc }}/Dockerfile \
             --build-arg BUILD_SHA=${{ github.sha }} \
-            --cache-from type=gha,scope=${{ matrix.svc }} \
+            --cache-from type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache \
-            --cache-to type=gha,mode=max,scope=${{ matrix.svc }} \
+            --cache-to type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache,mode=max \
             --load \
             -t svcforge/${{ matrix.svc }}:ci \
             .
 
       - name: trivy
-        uses: aquasecurity/trivy-action@18f2510ee396bbf400402947b394f2dd8c87dbb0 # v0.29.0
+        # Run trivy directly rather than via aquasecurity/trivy-action, for the same reason
-        with:
+        # gitleaks is run directly above: the command is the documented one, pinned by
-          image-ref: svcforge/${{ matrix.svc }}:ci
+        # digest, with no nested action resolution.
-          severity: HIGH,CRITICAL
+        #
-          # A CVE with no fix available is not something this PR can act on; failing on it
+        # It is also the only thing that works here. trivy-action internally does
-          # only teaches people to add ignore entries. Rebuilding on a new base image picks
+        # `uses: aquasecurity/setup-trivy@v0.2.2`, and that tag no longer exists upstream —
-          # the fix up the day it exists.
+        # the earliest published tag today is v0.2.6. The runner clones it and fails with
-          ignore-unfixed: true
+        # "Unable to resolve v0.2.2: reference not found". A third-party action pinned by
-          exit-code: "1"
+        # SHA still resolves ITS OWN dependencies by mutable tag, so pinning the outer
-          format: table
+        # action bought nothing.
+        #
+        # --ignore-unfixed: a CVE with no fix available is not something this PR can act
+        # on, and failing on it only teaches people to add ignore entries. Rebuilding on a
+        # new base image picks the fix up the day it exists.
+        #
+        # The mounted cache dir keeps the vuln DB (~50MB) across the three matrix legs on
+        # this runner instead of re-downloading it for each.
+        run: |
+          docker run --rm \
+            -v /var/run/docker.sock:/var/run/docker.sock \
+            -v "$PWD/.trivycache:/root/.cache/trivy" \
+            aquasec/trivy:0.58.1@sha256:ab70a02200597efa04748f210f793936eb647cbcdb0ea69cc30b226d6f5a22c7 \
+            image \
+              --severity HIGH,CRITICAL \
+              --ignore-unfixed \
+              --exit-code 1 \
+              --format table \
+              --no-progress \
+              svcforge/${{ matrix.svc }}:ci
 
       - name: push by digest
         if: github.ref == 'refs/heads/master' && github.event_name == 'push'
@@ -262,7 +297,7 @@ jobs:
           docker buildx build \
             -f services/${{ matrix.svc }}/Dockerfile \
             --build-arg BUILD_SHA=${{ github.sha }} \
-            --cache-from type=gha,scope=${{ matrix.svc }} \
+            --cache-from type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache \
             --push \
             -t "${IMAGE}:${GITHUB_SHA}" \
             .

RUNBOOK.md

@@ -12,6 +12,89 @@ alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'
 
 ---
 
+## Setting up CI/CD from scratch
+
+What it takes to get this repo building in Gitea Actions, including every trap that cost
+real time. Do it in this order; each step fails loudly if the one before it was skipped.
+
+### 1. The repo and its secrets
+
+```bash
+GITEA=https://gitea.oci-oci.duckdns.org
+USER=gitea_admin        # the registry namespaces packages by OWNER, so this is IMAGE_NS too
+
+curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
+  -H 'content-type: application/json' \
+  -d '{"name":"svcforge","default_branch":"master","private":false}'
+
+# A PAT with exactly three scopes. Not admin.
+curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
+  -H 'content-type: application/json' \
+  -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'
+```
+
+Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API):
+
+| Secret | Value | Why it exists |
+|---|---|---|
+| `REGISTRY_USER` | `gitea_admin` | |
+| `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. |
+| `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. Note what it is *not*: a kubeconfig. CI's maximum blast radius is a bad commit. |
+
+### 2. The runner needs a cache server, and it fails SOFT without one
+
+`cache: enabled: false` in the act_runner config is the default, and it does not fail the
+build. It prints:
+
+```
+Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.
+```
+
+…and carries on, re-downloading every wheel on every run, forever. **A cache that is off
+looks exactly like a cache that is always cold.** Enable it in
+`oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it):
+
+```yaml
+cache:
+  enabled: true
+  dir: /data/cache     # the runner's PVC, so it survives a restart
+  host: ""             # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
+  port: 8088
+```
+
+```bash
+cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea
+```
+
+Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to
+type=gha` are both no-ops.
+
+### 3. Traps that are specific to Gitea, not GitHub
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. |
+| `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. |
+| The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. |
+| Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. |
+
+### 4. Verify the whole loop, not just the green checkmarks
+
+```bash
+# the digest CI pushed
+docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
+  --format '{{.Manifest.Digest}}'
+# the digest the chart deploys — these must be equal
+grep -A2 'api:' deploy/chart/values.yaml
+# what ArgoCD actually synced
+kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'
+```
+
+If those three disagree, the deploy is not what CI tested, and every other guarantee in
+this document is void.
+
+---
+
 ## Measured numbers
 
 From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks