Compare commits

..

2 Commits

Author SHA1 Message Date
Nguyen Minh Phuc e8b6116a58 ci: age-filter the image prune so digest-pulled images survive
ci / lint (push) Successful in 43s
ci / types (push) Successful in 1m14s
ci / unit (push) Successful in 45s
ci / security (push) Successful in 1m24s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 11s
ci / integration (push) Successful in 1m40s
ci / image (api) (push) Successful in 2m32s
ci / image (reconciler) (push) Successful in 3m0s
ci / image (worker) (push) Failing after 49s
ci / bump (push) Has been skipped
Run #17 showed `docker image prune -f` deleting the trivy image, which buys
back a 157MB pull on every subsequent run. An image pulled by digest carries no
tag, so it is dangling the moment its container exits, and bare dangling-only
pruning is not as narrow as it reads.

The act runner image survived the same prune only by accident: this step runs
inside an act container, so that image happens to be in use exactly while the
prune executes. That is luck, not design, and it would not hold if the step
ever moved.

An until=168h filter on both prunes keeps a week of recently used layers,
matching the buildx cache window already in place.

Verified in run #17 before this change: trivy logged "Need to update DB" zero
times, so the named-volume DB cache works, and registry digests equal the
chart's for all three images.
2026-07-20 05:17:13 +00:00
Nguyen Minh Phuc 5f18f9eeeb runbook: correct the claim that runner restarts orphan jobs
The previous entry stated that restarting act_runner leaves every in-flight job
orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the
new pod and finished normally, while run #14 really was left stuck. Both
outcomes happen, so in_progress after a restart is ambiguous and the runner
logs are what settle it.

Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere
in its swagger, and DELETE on a run returned 204 against a live run without
stopping it. The UI button is the only way to cancel.
2026-07-20 05:17:13 +00:00
2 changed files with 30 additions and 13 deletions
+8 -1
View File
@@ -455,9 +455,16 @@ jobs:
# week — recent enough that `--cache-from` still hits on normal traffic. # week — recent enough that `--cache-from` still hits on normal traffic.
# #
# Named volumes are never pruned here: that is where the trivy vuln DB lives. # Named volumes are never pruned here: that is where the trivy vuln DB lives.
#
# Both prunes carry an age filter, because "dangling" catches more than it looks.
# An image pulled by digest has no tag, so it is dangling the moment its container
# exits. A bare `docker image prune -f` therefore deleted the trivy image on every
# run and bought back a 157MB pull on the next one. The act runner image survived
# that only by accident: this step executes inside an act container, so the image
# is in use exactly while the prune runs. Relying on that is not a design.
if: always() if: always()
run: | run: |
docker image prune -f docker image prune -f --filter until=168h
docker buildx prune -af --filter until=168h docker buildx prune -af --filter until=168h
echo "--- dind disk after prune ---" echo "--- dind disk after prune ---"
docker system df docker system df
+22 -12
View File
@@ -124,24 +124,34 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
that reason. A dedicated PVC for the image cache would survive restarts outright, but on that reason. A dedicated PVC for the image cache would survive restarts outright, but on
this cluster that volume faulted and blocked the runner, so it is deliberately not used. this cluster that volume faulted and blocked the runner, so it is deliberately not used.
### 6. Restarting the runner orphans its in-flight jobs ### 6. Stopping a run, and reading a restarted runner correctly
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks outcomes have been observed on this cluster:
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both - run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
the run id and the run index. Delete the run instead, which does exist and takes the **run 15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
index** (`#14`), not the database id (`51`): - run #16 had its remaining jobs re-dispatched to the new pod and carried on normally
So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
actually working before concluding anything, or you will diagnose a healthy run as a zombie:
```bash ```bash
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \ kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
``` ```
The queued run starts within seconds. The two ids are easy to confuse and the wrong one Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
returns 404 rather than touching the wrong run, which is the safe direction. run sitting in `waiting` behind a live one is correct, not a block.
**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
so an API token cannot drive it, and cancelling from a script is simply not available.
`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.
To stop a running job: click Cancel in the UI.
### 7. Verify the whole loop, not just the green checkmarks ### 7. Verify the whole loop, not just the green checkmarks