runbook: correct the claim that runner restarts orphan jobs

The previous entry stated that restarting act_runner leaves every in-flight job
orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the
new pod and finished normally, while run #14 really was left stuck. Both
outcomes happen, so in_progress after a restart is ambiguous and the runner
logs are what settle it.

Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere
in its swagger, and DELETE on a run returned 204 against a live run without
stopping it. The UI button is the only way to cancel.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 04:56:07 +00:00
parent 1a36f43c69
commit 5f18f9eeeb
+22 -12
View File
@@ -124,24 +124,34 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
### 6. Restarting the runner orphans its in-flight jobs
### 6. Stopping a run, and reading a restarted runner correctly
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
outcomes have been observed on this cluster:
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both
the run id and the run index. Delete the run instead, which does exist and takes the **run
index** (`#14`), not the database id (`51`):
- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally
So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
actually working before concluding anything, or you will diagnose a healthy run as a zombie:
```bash
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
```
The queued run starts within seconds. The two ids are easy to confuse and the wrong one
returns 404 rather than touching the wrong run, which is the safe direction.
Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
run sitting in `waiting` behind a live one is correct, not a block.
**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
so an API token cannot drive it, and cancelling from a script is simply not available.
`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.
To stop a running job: click Cancel in the UI.
### 7. Verify the whole loop, not just the green checkmarks