runbook: recovering the queue after a runner restart
Restarting act_runner leaves its in-flight jobs in_progress with nothing behind
them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no
cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run
index, not the database id.
This commit is contained in:
+20
-1
@@ -124,7 +124,26 @@ from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml`
|
|||||||
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
||||||
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
||||||
|
|
||||||
### 6. Verify the whole loop, not just the green checkmarks
|
### 6. Restarting the runner orphans its in-flight jobs
|
||||||
|
|
||||||
|
Any change to the act_runner StatefulSet recreates the pod, and every job it was running is
|
||||||
|
left `in_progress` forever with nothing behind it. At `capacity: 1` a single orphan blocks
|
||||||
|
the whole queue: the next run sits in `waiting` indefinitely. One was still `in_progress` at
|
||||||
|
15m45s, so do not wait for `ZOMBIE_TASK_TIMEOUT` to rescue it.
|
||||||
|
|
||||||
|
Gitea 1.26 has no cancel endpoint — `POST .../actions/runs/{id}/cancel` returns 404 for both
|
||||||
|
the run id and the run index. Delete the run instead, which does exist and takes the **run
|
||||||
|
index** (`#14`), not the database id (`51`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: token $T" \
|
||||||
|
"$G/api/v1/repos/gitea_admin/svcforge/actions/runs/14" # -> 204
|
||||||
|
```
|
||||||
|
|
||||||
|
The queued run starts within seconds. The two ids are easy to confuse and the wrong one
|
||||||
|
returns 404 rather than touching the wrong run, which is the safe direction.
|
||||||
|
|
||||||
|
### 7. Verify the whole loop, not just the green checkmarks
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# the digest CI pushed
|
# the digest CI pushed
|
||||||
|
|||||||
Reference in New Issue
Block a user