Nguyen Minh Phuc
5f18f9eeeb
The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel.
351 lines
16 KiB
Markdown
351 lines
16 KiB
Markdown
# svcforge runbook
|
||
|
||
Four entries. Each starts from an alert firing and ends at either a fix or an escalation.
|
||
Every command is copy-pasteable; none of them require thinking at 3am, which is the point.
|
||
|
||
Set these first:
|
||
|
||
```bash
|
||
set -a; . ~/.config/svcforge/secrets.env; set +a # SVCFORGE_PG_DSN_SESSION for psql
|
||
alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'
|
||
```
|
||
|
||
---
|
||
|
||
## Setting up CI/CD from scratch
|
||
|
||
What it takes to get this repo building in Gitea Actions, including every trap that cost
|
||
real time. Do it in this order; each step fails loudly if the one before it was skipped.
|
||
|
||
### 1. The repo and its secrets
|
||
|
||
```bash
|
||
GITEA=https://gitea.oci-oci.duckdns.org
|
||
USER=gitea_admin # the registry namespaces packages by OWNER, so this is IMAGE_NS too
|
||
|
||
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
|
||
-H 'content-type: application/json' \
|
||
-d '{"name":"svcforge","default_branch":"master","private":false}'
|
||
|
||
# A PAT with exactly three scopes. Not admin.
|
||
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
|
||
-H 'content-type: application/json' \
|
||
-d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'
|
||
```
|
||
|
||
Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API):
|
||
|
||
| Secret | Value | Why it exists |
|
||
|---|---|---|
|
||
| `REGISTRY_USER` | `gitea_admin` | |
|
||
| `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. |
|
||
| `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit. |
|
||
|
||
### 2. The runner needs a cache server, and it fails SOFT without one
|
||
|
||
`cache: enabled: false` in the act_runner config is the default, and it does not fail the
|
||
build. It prints:
|
||
|
||
```
|
||
Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.
|
||
```
|
||
|
||
…and carries on, re-downloading every wheel on every run, forever. **A cache that is off
|
||
looks exactly like a cache that is always cold.** Enable it in
|
||
`oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it):
|
||
|
||
```yaml
|
||
cache:
|
||
enabled: true
|
||
dir: /data/cache # the runner's PVC, so it survives a restart
|
||
host: "" # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
|
||
port: 8088
|
||
```
|
||
|
||
```bash
|
||
cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea
|
||
```
|
||
|
||
Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to
|
||
type=gha` are both no-ops.
|
||
|
||
### 3. Traps that are specific to Gitea, not GitHub
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---|---|---|
|
||
| `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. |
|
||
| `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. |
|
||
| The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. |
|
||
| Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. |
|
||
|
||
### 4. Keeping trivy green
|
||
|
||
The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that
|
||
passed yesterday fails today without a single line of code changing. Two rules keep it sane.
|
||
|
||
**Bump the version, do not add an ignore.** The worker image went 39 findings (2 CRITICAL)
|
||
→ 18 → 5 → 0 across three fixes, each a version bump or a removal:
|
||
|
||
| Change | Result |
|
||
|---|---|
|
||
| `alpine/helm` 3.16.2 → 3.21.3, `kubectl` 1.31.2 → 1.35.3 | 39 → 18, both CRITICALs cleared |
|
||
| `kubectl` 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1) | 18 → 5 |
|
||
| dropped kubectl entirely — `helm --create-namespace` replaced `kubectl apply` | **5 → 0** |
|
||
|
||
**The cheapest CVE is the binary you do not ship.** The last five findings lived in
|
||
kubectl's vendored `golang.org/x/net` and Go stdlib, inside the newest kubectl that exists —
|
||
no version cleared them. kubectl was in that image for exactly one call, and helm already
|
||
does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.
|
||
|
||
### 5. When every job fails at `Set up job` after ~14 minutes
|
||
|
||
Symptom: `Set up job` runs for 10–15 minutes and succeeds, then the first real step fails
|
||
instantly at 0s, and every downstream job is skipped. It looks like the action broke.
|
||
|
||
Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job
|
||
image (`ghcr.io/catthehacker/ubuntu:act-24.04`) before it can start. dind has no volume for
|
||
`/var/lib/docker` — the cache lives in its container writable layer and is destroyed on
|
||
every pod restart. A runner that is restart-looping therefore never keeps a cache, and each
|
||
job pays the full pull.
|
||
|
||
Check it:
|
||
```bash
|
||
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images
|
||
```
|
||
Empty output is the diagnosis. Warm it once by hand:
|
||
```bash
|
||
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
|
||
ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9
|
||
```
|
||
|
||
The durable fix is to stop the runner restarting. Its `/data` PVC is ReadWriteOnce, so
|
||
every reschedule hits `Multi-Attach error` and the pod sits in Init until Longhorn detaches
|
||
from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` for exactly
|
||
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
|
||
this cluster that volume faulted and blocked the runner, so it is deliberately not used.
|
||
|
||
### 6. Stopping a run, and reading a restarted runner correctly
|
||
|
||
Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
|
||
outcomes have been observed on this cluster:
|
||
|
||
- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
|
||
15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
|
||
- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally
|
||
|
||
So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
|
||
actually working before concluding anything, or you will diagnose a healthy run as a zombie:
|
||
|
||
```bash
|
||
kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
|
||
```
|
||
|
||
Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
|
||
run sitting in `waiting` behind a live one is correct, not a block.
|
||
|
||
**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
|
||
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
|
||
so an API token cannot drive it, and cancelling from a script is simply not available.
|
||
|
||
`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
|
||
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
|
||
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.
|
||
|
||
To stop a running job: click Cancel in the UI.
|
||
|
||
### 7. Verify the whole loop, not just the green checkmarks
|
||
|
||
```bash
|
||
# the digest CI pushed
|
||
docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
|
||
--format '{{.Manifest.Digest}}'
|
||
# the digest the chart deploys — these must be equal
|
||
grep -A2 'api:' deploy/chart/values.yaml
|
||
# what ArgoCD actually synced
|
||
kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'
|
||
```
|
||
|
||
If those three disagree, the deploy is not what CI tested, and every other guarantee in
|
||
this document is void.
|
||
|
||
---
|
||
|
||
## Measured numbers
|
||
|
||
From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks
|
||
per run. **These came off local Postgres on the same box, with a sub-millisecond round trip.
|
||
Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach** —
|
||
the shape is what transfers, not the absolute numbers.
|
||
|
||
| replicas | pool max_size | worker concurrency | connections used | drain (200 tasks) | throughput |
|
||
|---:|---:|---:|---:|---:|---:|
|
||
| 1 | 5 | 4 | 5 | 4.3s | 46.9 task/s |
|
||
| 2 | 5 | 4 | 10 | 4.2s | 47.6 task/s |
|
||
| 4 | 5 | 4 | 20 | 2.2s | 92.3 task/s |
|
||
| 2 | 20 | 16 | 40 | 2.1s | 97.1 task/s |
|
||
|
||
Three things this says:
|
||
|
||
1. **Going from 1 replica to 2 bought nothing** (46.9 → 47.6). Throughput here is bounded by
|
||
per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated
|
||
semaphore is the most common wrong fix for a slow queue.
|
||
2. **Concurrency is the knob that moved it** — 4 replicas (20 connections) and 2 replicas at
|
||
concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the
|
||
same throughput for twice the connections, which on the free tier is the worse trade.
|
||
3. **The budget is `(api + worker replicas) x max_size`**, and it is spent whether or not the
|
||
connections are busy. The bottom row costs 40 connections for a 2% gain over the row above
|
||
it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count.
|
||
|
||
Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding
|
||
it against the actual pooler is the experiment worth running: raise `replicas x max_size`
|
||
until claims slow and `PoolTimeout` appears, and write the number here.
|
||
|
||
---
|
||
|
||
## Queue stuck
|
||
|
||
**Alert:** `SvcforgeQueueDepthRising`
|
||
|
||
**Diagnose.** Start here, always:
|
||
|
||
```bash
|
||
sfsql -c "select state, count(*) from tasks group by 1;"
|
||
```
|
||
|
||
Then split the three causes apart — they look identical from the alert and need opposite fixes:
|
||
|
||
```bash
|
||
# Stuck leases: rows 'running' with a locked_at that never advances.
|
||
sfsql -c "select kind, locked_by, locked_at, last_error from tasks
|
||
where state='running' order by locked_at limit 10;"
|
||
|
||
# No workers: is anything actually consuming?
|
||
kubectl get pods -l app=worker -o wide
|
||
kubectl logs -l app=worker --tail=20 --prefix
|
||
|
||
# run_after in the future: backoff has parked everything.
|
||
sfsql -c "select count(*) from tasks where state='queued' and run_after > now();"
|
||
```
|
||
|
||
| What you see | Cause | Fix |
|
||
|---|---|---|
|
||
| `running` rows, `locked_at` older than 5m, no worker pods hold those IDs | Workers died mid-task | None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it. |
|
||
| Zero worker pods, or all `CrashLoopBackOff` | No consumer | Fix the workers. `kubectl describe pod -l app=worker`. |
|
||
| Everything `queued` with `run_after` far in the future | Backoff, i.e. tasks are failing and retrying | This is not a queue problem. Go to **Provision failing**. |
|
||
| `queued` rows with `run_after <= now()` and healthy workers | Real: claim is not returning rows | Check pooler connection budget (see **Supabase full**). |
|
||
|
||
**Never** hand-edit `state='running'` back to `'queued'`. The lease does that, and doing it
|
||
by hand while the worker is actually alive gives you two workers on one task — the exact
|
||
thing the whole design prevents.
|
||
|
||
**Escalate** if workers are healthy, leases are fresh, and depth still grows: that is a
|
||
claim-query or pooler bug, not an ops problem.
|
||
|
||
---
|
||
|
||
## Provision failing
|
||
|
||
**Alert:** `SvcforgeTaskFailed` (tasks reaching the dead-letter state), or `SvcforgeProvisionSlow`
|
||
(they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way).
|
||
|
||
**Diagnose:**
|
||
|
||
```bash
|
||
sfsql -c "select id, service_type, chart_version, error from instances where state='failed';"
|
||
sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;"
|
||
|
||
NS=tenant-<team>
|
||
helm list -n "$NS"
|
||
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20
|
||
```
|
||
|
||
| `error` looks like | Cause | Fix |
|
||
|---|---|---|
|
||
| `chart "..." version "..." not found` | Bad pin in `catalog.yaml` | Correct the version, commit. The next `upgrade`/`provision` picks it up. |
|
||
| `timed out waiting for the condition` | Cluster capacity — the chart installed but pods never became ready | `kubectl describe pod -n $NS`. Usually `Insufficient cpu/memory` or a PVC pending on Longhorn. |
|
||
| `Error: ... forbidden: User "system:serviceaccount:svcforge:..."` | RBAC | The worker's ClusterRole is missing a verb. Chart change, not a manual `kubectl edit`. |
|
||
| `ImagePullBackOff` in events | Registry auth or a gated image | Prefer `bitnamilegacy/*` images, which pull anonymously. |
|
||
|
||
After fixing the cause, tasks that already dead-lettered do **not** retry themselves. Requeue
|
||
deliberately:
|
||
|
||
```bash
|
||
sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null
|
||
where id = <task_id>;"
|
||
```
|
||
|
||
**Escalate** if `error` is empty on a failed instance — that means the failure path itself
|
||
lost the message.
|
||
|
||
---
|
||
|
||
## Orphaned release
|
||
|
||
**Alert:** `SvcforgeReconcilerStale` — the reconciler has not completed a loop recently, so
|
||
drift is no longer being *detected* at all. Drift itself is reported in the reconciler's logs
|
||
and metrics rather than paged on, because it is usually benign and always needs a human to
|
||
judge. A stale reconciler is the real emergency: nothing is watching.
|
||
|
||
The control loop **never auto-deletes a release.** That is deliberate: a bug in the drift
|
||
check that deletes things is unrecoverable, and one that only reports is a Tuesday.
|
||
|
||
**Diagnose:**
|
||
|
||
```bash
|
||
helm list -A -o json | jq -r '.[].name' | sort > /tmp/real
|
||
sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want
|
||
|
||
comm -23 /tmp/real /tmp/want # in the cluster, not in the DB -> orphan
|
||
comm -13 /tmp/real /tmp/want # in the DB, not in the cluster -> missing
|
||
```
|
||
|
||
| Direction | Meaning | Action |
|
||
|---|---|---|
|
||
| Orphan (cluster only) | A deprovision half-finished, or someone ran `helm install` by hand | Confirm the tenant is gone, then `helm uninstall <name> -n <ns>` **by hand**, and write down that you did. |
|
||
| Missing (DB only) | Someone deleted a release out from under us | Requeue a `provision` task for that instance. It is idempotent; it will rebuild. |
|
||
|
||
**Escalate** before uninstalling anything you did not personally trace to a deleted instance.
|
||
A wrong `helm uninstall` here deletes a tenant's data.
|
||
|
||
---
|
||
|
||
## Supabase full
|
||
|
||
**Alert:** none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages
|
||
you before you hit it; you find out when writes start failing. Until someone adds a size rule,
|
||
this entry is driven by the calendar, not by an alert. Check it monthly:
|
||
|
||
```bash
|
||
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
|
||
```
|
||
|
||
**Diagnose:**
|
||
|
||
```bash
|
||
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
|
||
sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables
|
||
order by pg_total_relation_size(relid) desc limit 5;"
|
||
sfsql -c "select count(*) from pg_stat_activity;"
|
||
```
|
||
|
||
It is almost always `tasks`. Every provision, upgrade and verify leaves a row forever.
|
||
|
||
```bash
|
||
sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';"
|
||
sfsql -c "vacuum (analyze) tasks;"
|
||
```
|
||
|
||
`vacuum` alone reclaims space **for reuse by Postgres**, but does not return it to the
|
||
filesystem — so `pg_database_size` may barely move. That is expected and fine; the space is
|
||
free for new rows. `vacuum full` does return it, takes an `ACCESS EXCLUSIVE` lock, and will
|
||
stall every worker for its duration. Only do it in a window, and only if you actually need
|
||
the bytes back.
|
||
|
||
If `count(*) from pg_stat_activity` is near the pooler's ceiling, the cause is arithmetic, not
|
||
load: `worker_replicas × pool_max_size + api_replicas × pool_max_size`. Lower `max_size` or
|
||
replicas. **Replica count is a database-capacity decision here**, which is unusual and worth
|
||
remembering.
|
||
|
||
**Escalate** if size is growing with `tasks` already pruned — that means `instances` is
|
||
growing, i.e. tenants are real, i.e. the free tier is the wrong tier.
|