runbook: recovering the queue after a runner restart

Restarting act_runner leaves its in-flight jobs in_progress with nothing behind
them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no
cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run
index, not the database id.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 04:41:15 +00:00
parent b02d4e85c6
commit 5d7f46483e
+20 -1
View File
@@ -124,7 +124,26 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
that reason. A dedicated PVC for the image cache would survive restarts outright, but on that reason. A dedicated PVC for the image cache would survive restarts outright, but on
this cluster that volume faulted and blocked the runner, so it is deliberately not used. this cluster that volume faulted and blocked the runner, so it is deliberately not used.
### 6. Verify the whole loop, not just the green checkmarks ### 6. Restarting the runner orphans its in-flight jobs
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both
the run id and the run index. Delete the run instead, which does exist and takes the **run
index** (`#14`), not the database id (`51`):
```bash
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
```
The queued run starts within seconds. The two ids are easy to confuse and the wrong one
returns 404 rather than touching the wrong run, which is the safe direction.
### 7. Verify the whole loop, not just the green checkmarks
```bash ```bash
# the digest CI pushed # the digest CI pushed