gitea_admin/svcforge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

runbook: correct the claim that runner restarts orphan jobs

Browse Source 
The previous entry stated that restarting act_runner leaves every in-flight job
orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the
new pod and finished normally, while run #14 really was left stuck. Both
outcomes happen, so in_progress after a restart is ambiguous and the runner
logs are what settle it.

Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere
in its swagger, and DELETE on a run returned 204 against a live run without
stopping it. The UI button is the only way to cancel.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 04:56:07 +00:00
parent 1a36f43c69
commit 5f18f9eeeb
1 changed files with 22 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RUNBOOK.md
+22 -12
View File
@@ -124,24 +124,34 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
that reason. A dedicated PVC for the image cache would survive restarts outright, but on that reason. A dedicated PVC for the image cache would survive restarts outright, but on
this cluster that volume faulted and blocked the runner, so it is deliberately not used. this cluster that volume faulted and blocked the runner, so it is deliberately not used.
### 6. Restarting the runner orphans its in-flight jobs ### 6. Stopping a run, and reading a restarted runner correctly
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks outcomes have been observed on this cluster:
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both - run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
the run id and the run index. Delete the run instead, which does exist and takes the **run 15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
index** (`#14`), not the database id (`51`): - run #16 had its remaining jobs re-dispatched to the new pod and carried on normally
So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
actually working before concluding anything, or you will diagnose a healthy run as a zombie:
```bash ```bash
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \ kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
``` ```
The queued run starts within seconds. The two ids are easy to confuse and the wrong one Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
returns 404 rather than touching the wrong run, which is the safe direction. run sitting in `waiting` behind a live one is correct, not a block.
**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
so an API token cannot drive it, and cancelling from a script is simply not available.
`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.
To stop a running job: click Cancel in the UI.
### 7. Verify the whole loop, not just the green checkmarks ### 7. Verify the whole loop, not just the green checkmarks