ci: reclaim dind disk after each image build

dind's /var/lib/docker is now a hostPath on node2 rather than the container's
writable layer, so it survives restarts — and nothing reclaims it. Kubelet's
image GC does not manage a nested daemon's store, so left alone it grows every
run until node2 hits disk pressure and evicts pods, which reads as a cluster
problem rather than a CI one.

The step is deliberately narrow. `docker image prune` without -a removes
dangling images only; with -a it would delete the act runner image, which no
container references between jobs, and buy back a 1.6GB re-pull on the next
run. The buildx cache is what actually grows without bound, so it is pruned by
age keeping a week, recent enough that --cache-from still hits. Named volumes
are never pruned, since that is where the trivy vuln DB lives.

always(), because a failed build still leaves layers behind, and that is when
disk is most likely to have been the reason it failed.

.gitea/workflows/ci.yaml

@@ -439,6 +439,29 @@ jobs:
           docker buildx imagetools inspect "${IMAGE}:${GITHUB_SHA}" \
             --format '{{.Manifest.Digest}}'
 
+      - name: reclaim dind disk
+        # dind's /var/lib/docker is a hostPath on node2 (see oci-k8s addons role), so
+        # nothing reclaims it automatically — kubelet's image GC does not manage a nested
+        # daemon's store. Left alone it grows every run until node2 hits disk pressure and
+        # starts evicting pods, which looks like a cluster problem rather than a CI one.
+        #
+        # `always()`: a failed build still leaves layers behind, and that is exactly when
+        # the disk is most likely to be the reason it failed.
+        #
+        # Deliberately narrow. `docker image prune` WITHOUT -a removes dangling images
+        # only; with -a it would delete the act runner image, which no container references
+        # between jobs, and buy back a 1.6GB re-pull on the very next run. The buildx cache
+        # is the part that actually grows without bound, so it is pruned by age and keeps a
+        # week — recent enough that `--cache-from` still hits on normal traffic.
+        #
+        # Named volumes are never pruned here: that is where the trivy vuln DB lives.
+        if: always()
+        run: |
+          docker image prune -f
+          docker buildx prune -af --filter until=168h
+          echo "--- dind disk after prune ---"
+          docker system df
+
   # --- stage 11: bump the chart's digests. CI's last act. ------------------------------
   bump:
     runs-on: ubuntu-latest

RUNBOOK.md

@@ -124,7 +124,26 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
 that reason. A dedicated PVC for the image cache would survive restarts outright, but on
 this cluster that volume faulted and blocked the runner, so it is deliberately not used.
 
-### 6. Verify the whole loop, not just the green checkmarks
+### 6. Restarting the runner orphans its in-flight jobs
+
+Any change to the act_runner StatefulSet recreates the pod, and every job it was running is
+left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks
+the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
+15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
+
+Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both
+the run id and the run index. Delete the run instead, which does exist and takes the **run
+index** (`#14`), not the database id (`51`):
+
+```bash
+curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \
+  "$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14"     # -> 204
+```
+
+The queued run starts within seconds. The two ids are easy to confuse and the wrong one
+returns 404 rather than touching the wrong run, which is the safe direction.
+
+### 7. Verify the whole loop, not just the green checkmarks
 
 ```bash
 # the digest CI pushed