From 5f18f9eeebff247556f0e7fb9900cfeaca4fd48c Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Mon, 20 Jul 2026 04:56:07 +0000 Subject: [PATCH] runbook: correct the claim that runner restarts orphan jobs The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel. --- RUNBOOK.md | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 67bfce0..2cb6cc1 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -124,24 +124,34 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` that reason. A dedicated PVC for the image cache would survive restarts outright, but on this cluster that volume faulted and blocked the runner, so it is deliberately not used. -### 6. Restarting the runner orphans its in-flight jobs +### 6. Stopping a run, and reading a restarted runner correctly -Any change to the act_runner StatefulSet recreates the pod, and every job it was running is -left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks -the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at -15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it. +Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both +outcomes have been observed on this cluster: -Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both -the run id and the run index. Delete the run instead, which does exist and takes the **run -index** (`#14`), not the database id (`51`): +- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at + 15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for +- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally + +So `in_progress` after a restart is ambiguous on its own. Check whether the runner is +actually working before concluding anything, or you will diagnose a healthy run as a zombie: ```bash -curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \ - "$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204 +kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4 ``` -The queued run starts within seconds. The two ids are easy to confuse and the wrong one -returns 404 rather than touching the wrong run, which is the safe direction. +Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later +run sitting in `waiting` behind a live one is correct, not a block. + +**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching +`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route, +so an API token cannot drive it, and cancelling from a script is simply not available. + +`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database +id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping +it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel. + +To stop a running job: click Cancel in the UI. ### 7. Verify the whole loop, not just the green checkmarks