Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e8b6116a58 | |||
| 5f18f9eeeb |
@@ -455,9 +455,16 @@ jobs:
|
|||||||
# week — recent enough that `--cache-from` still hits on normal traffic.
|
# week — recent enough that `--cache-from` still hits on normal traffic.
|
||||||
#
|
#
|
||||||
# Named volumes are never pruned here: that is where the trivy vuln DB lives.
|
# Named volumes are never pruned here: that is where the trivy vuln DB lives.
|
||||||
|
#
|
||||||
|
# Both prunes carry an age filter, because "dangling" catches more than it looks.
|
||||||
|
# An image pulled by digest has no tag, so it is dangling the moment its container
|
||||||
|
# exits. A bare `docker image prune -f` therefore deleted the trivy image on every
|
||||||
|
# run and bought back a 157MB pull on the next one. The act runner image survived
|
||||||
|
# that only by accident: this step executes inside an act container, so the image
|
||||||
|
# is in use exactly while the prune runs. Relying on that is not a design.
|
||||||
if: always()
|
if: always()
|
||||||
run: |
|
run: |
|
||||||
docker image prune -f
|
docker image prune -f --filter until=168h
|
||||||
docker buildx prune -af --filter until=168h
|
docker buildx prune -af --filter until=168h
|
||||||
echo "--- dind disk after prune ---"
|
echo "--- dind disk after prune ---"
|
||||||
docker system df
|
docker system df
|
||||||
|
|||||||
+22
-12
@@ -124,24 +124,34 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
|
|||||||
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
||||||
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
||||||
|
|
||||||
### 6. Restarting the runner orphans its in-flight jobs
|
### 6. Stopping a run, and reading a restarted runner correctly
|
||||||
|
|
||||||
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is
|
Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
|
||||||
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks
|
outcomes have been observed on this cluster:
|
||||||
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
|
|
||||||
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
|
|
||||||
|
|
||||||
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both
|
- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
|
||||||
the run id and the run index. Delete the run instead, which does exist and takes the **run
|
15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
|
||||||
index** (`#14`), not the database id (`51`):
|
- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally
|
||||||
|
|
||||||
|
So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
|
||||||
|
actually working before concluding anything, or you will diagnose a healthy run as a zombie:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \
|
kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
|
||||||
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The queued run starts within seconds. The two ids are easy to confuse and the wrong one
|
Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
|
||||||
returns 404 rather than touching the wrong run, which is the safe direction.
|
run sitting in `waiting` behind a live one is correct, not a block.
|
||||||
|
|
||||||
|
**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
|
||||||
|
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
|
||||||
|
so an API token cannot drive it, and cancelling from a script is simply not available.
|
||||||
|
|
||||||
|
`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
|
||||||
|
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
|
||||||
|
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.
|
||||||
|
|
||||||
|
To stop a running job: click Cancel in the UI.
|
||||||
|
|
||||||
### 7. Verify the whole loop, not just the green checkmarks
|
### 7. Verify the whole loop, not just the green checkmarks
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user