From 5d7f46483ebed044e08662815f9395d7c24b5a20 Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Mon, 20 Jul 2026 04:41:15 +0000 Subject: [PATCH] runbook: recovering the queue after a runner restart Restarting act_runner leaves its in-flight jobs in_progress with nothing behind them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run index, not the database id. --- RUNBOOK.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index b311850..67bfce0 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -124,7 +124,26 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` that reason. A dedicated PVC for the image cache would survive restarts outright, but on this cluster that volume faulted and blocked the runner, so it is deliberately not used. -### 6. Verify the whole loop, not just the green checkmarks +### 6. Restarting the runner orphans its in-flight jobs + +Any change to the act_runner StatefulSet recreates the pod, and every job it was running is +left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks +the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at +15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it. + +Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both +the run id and the run index. Delete the run instead, which does exist and takes the **run +index** (`#14`), not the database id (`51`): + +```bash +curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \ + "$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204 +``` + +The queued run starts within seconds. The two ids are easy to confuse and the wrong one +returns 404 rather than touching the wrong run, which is the safe direction. + +### 7. Verify the whole loop, not just the green checkmarks ```bash # the digest CI pushed