svcforge/RUNBOOK.md

svcforge runbook

Four entries. Each starts from an alert firing and ends at either a fix or an escalation. Every command is copy-pasteable; none of them require thinking at 3am, which is the point.

Set these first:

set -a; . ~/.config/svcforge/secrets.env; set +a   # SVCFORGE_PG_DSN_SESSION for psql
alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'

---

Setting up CI/CD from scratch

What it takes to get this repo building in Gitea Actions, including every trap that cost real time. Do it in this order; each step fails loudly if the one before it was skipped.

1. The repo and its secrets

GITEA=https://gitea.oci-oci.duckdns.org
USER=gitea_admin        # the registry namespaces packages by OWNER, so this is IMAGE_NS too

curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge","default_branch":"master","private":false}'

# A PAT with exactly three scopes. Not admin.
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'

Then set three repo Actions secrets (Settings → Actions → Secrets, or the API):

Secret	Value	Why it exists
REGISTRY_USER	gitea_admin
REGISTRY_TOKEN	the PAT	The auto-injected GITEA_TOKEN is rejected by the package registry with a 401. This is the single most common reason a first pipeline fails at docker push.
CI_BOT_TOKEN	the PAT	Used only by the bump job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit.

2. The runner needs a cache server, and it fails SOFT without one

cache: enabled: false in the act_runner config is the default, and it does not fail the build. It prints:

Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.

…and carries on, re-downloading every wheel on every run, forever. A cache that is off looks exactly like a cache that is always cold. Enable it in oci-k8s/k8s/roles/addons/tasks/main.yml (Ansible owns this — never kubectl edit it):

cache:
  enabled: true
  dir: /data/cache     # the runner's PVC, so it survives a restart
  host: ""             # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
  port: 8088

cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea

Without this, astral-sh/setup-uv's enable-cache: true and buildx's --cache-to type=gha are both no-ops.

3. Traps that are specific to Gitea, not GitHub

Symptom	Cause	Fix
Unable to resolve v0.2.2: reference not found	A third-party action pinned by SHA still resolves its own dependencies by mutable tag. trivy-action@v0.29.0 does uses: setup-trivy@v0.2.2, and that tag was removed upstream.	Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing.
docker push → 401	Used GITEA_TOKEN.	Use a PAT with write:package.
The pipeline retriggers itself forever	The bump job commits to the repo it is triggered by.	[skip ci] in the commit message (Gitea honours it), and a git diff --quiet guard so an unchanged digest commits nothing.
Service container unreachable at localhost	Jobs run inside a container, so a service is reached by its service name, not localhost.	postgres:5432, not localhost:5432.

4. Keeping trivy green

The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that passed yesterday fails today without a single line of code changing. Two rules keep it sane.

Bump the version, do not add an ignore. The worker image went 39 findings (2 CRITICAL) → 18 → 5 → 0 across three fixes, each a version bump or a removal:

Change	Result
alpine/helm 3.16.2 → 3.21.3, kubectl 1.31.2 → 1.35.3	39 → 18, both CRITICALs cleared
kubectl 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1)	18 → 5
dropped kubectl entirely — helm --create-namespace replaced kubectl apply	5 → 0

The cheapest CVE is the binary you do not ship. The last five findings lived in kubectl's vendored golang.org/x/net and Go stdlib, inside the newest kubectl that exists — no version cleared them. kubectl was in that image for exactly one call, and helm already does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.

5. When every job fails at Set up job after ~14 minutes

Symptom: Set up job runs for 10–15 minutes and succeeds, then the first real step fails instantly at 0s, and every downstream job is skipped. It looks like the action broke.

Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job image (ghcr.io/catthehacker/ubuntu:act-24.04) before it can start. dind has no volume for /var/lib/docker — the cache lives in its container writable layer and is destroyed on every pod restart. A runner that is restart-looping therefore never keeps a cache, and each job pays the full pull.

Check it:

kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images

Empty output is the diagnosis. Warm it once by hand:

kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
  ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9

The durable fix is a persistent image store, which the runner now has: /var/lib/docker is a hostPath on node0 (see oci-k8s/.../addons/tasks/main.yml), so the act image survives a restart and is not re-pulled. The runner is pinned to node0, not node2 — node2 is a single-core control-plane node whose pod network was measured 21x slower under its own load, which starved every clone and pull. Its /data PVC is NFS ReadWriteMany, so a reschedule attaches immediately with no Multi-Attach wait. Entries 9 and 10 cover the caches and the node move in full.

6. Stopping a run, and reading a restarted runner correctly

Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both outcomes have been observed on this cluster:

• run #14 was left with two jobs in_progress and nothing behind them, still stuck at 15m45s, so ZOMBIE_TASK_TIMEOUT is not a rescue you can wait for
• run #16 had its remaining jobs re-dispatched to the new pod and carried on normally

So in_progress after a restart is ambiguous on its own. Check whether the runner is actually working before concluding anything, or you will diagnose a healthy run as a zombie:

kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4

Recent NewParallelExecutor lines mean it is executing, not stuck. At capacity: 1 a later run sitting in waiting behind a live one is correct, not a block.

Gitea 1.26 has no cancel endpoint at all — the full swagger contains no path matching cancel. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route, so an API token cannot drive it, and cancelling from a script is simply not available.

DELETE .../actions/runs/{index} exists and takes the run index (14), not the database id (51). It cleared the stuck #14, but returned 204 against the live #16 without stopping it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.

To stop a running job: click Cancel in the UI.

7. Gitea postgres: Input/output error, and when scale 0/1 is not enough

gitea-postgresql-0 CrashLoopBackOff with:

mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error

and the Gitea API returning 500, so every CI run dies at checkout with Failed to connect to gitea-http:3000. The "Initializing PostgreSQL database" line above that error is alarming and is not what it looks like: the data is fine, the mount is broken, so the container sees an empty directory.

The documented recovery — scale to 0, wait for detached, scale back to 1 — was not enough here. The volume came back detached/faulted and simply refused to attach, so the pod sat in ContainerCreating. auto-salvage: true does not help: salvage happens during attach, and a faulted volume never gets that far, so it cannot rescue itself.

What the volume was actually saying:

V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}')
kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}'   # detached/faulted
kubectl -n longhorn-system get replicas.longhorn.io -o json \
  | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv'

The replica carries a failedAt timestamp, and that alone is what keeps the volume faulted. Clearing it is the salvage:

kubectl -n longhorn-system patch replicas.longhorn.io <replica-name> --type merge \
  -p '{"spec":{"failedAt":"","lastFailedAt":""}}'

The volume went attached/healthy and postgres reached 1/1 within 40 seconds, with the repo, its size and the whole CI run history intact.

Check the backups before patching anything, because this cluster runs Longhorn at one replica — there is no second copy to fall back on, only the nightly backup:

kubectl -n longhorn-system get backups.longhorn.io -o json \
  | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv'

Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash recovery on start. If it cannot, restore the most recent Completed backup instead.

8. OPEN: dind is killed by its own liveness probe

Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were diagnosed as something else.

The runner sits at Init:1/2 and its dind sidecar accumulates restarts:

Liveness probe failed: command timed out:
  "/usr/bin/test -S /var/run/docker.sock" timed out after 1s   (x27 over 156m)
Killing: Init container dind failed liveness probe

The probe is hardcoded at timeoutSeconds: 1, failureThreshold: 3, periodSeconds: 10. test -S only asks whether a socket exists. When that cannot finish inside a second, the node is starved rather than dind being unhealthy, and kubelet kills a working daemon.

Why this matters beyond the runner restarting. Image builds failed with:

ERROR: failed to solve: DeadlineExceeded: no active session for <id>

which was attributed to CPU starvation alone and addressed by dropping the runner's capacity to 1. Starvation is real, but the mechanism is more likely that kubelet killed dind mid-build and the buildkit session died with it. Lowering capacity reduced the load that trips the probe, which is consistent with run #17 passing — it treated the cause of the trigger, not the trigger. Runs #18-#21 then failed anyway.

Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the kill timestamps against the failed builds:

kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
  -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'

The chart exposes no probe knobs — helm show values gitea-charts/actions has no match for probe. So this cannot be fixed the way capacity was, and a kubectl patch is reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.

The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to roughly timeoutSeconds: 5 and failureThreshold: 6. This cluster already mutates workloads that way — see force-best-effort-cpu, which rewrites every CPU request to 0 — so the precedent and the tooling are both in place.

9. The runner's three ephemeral caches, and the boot cascade they cause

Three separate caches on this runner were container-layer only, each found the same way — something was slow, and the cause was a cache that had never survived a restart:

cache	path	fixed by
dind image store	/var/lib/docker	hostPath /var/lib/gitea-dind
trivy vulnerability DB	/root/.cache/trivy in dind	named docker volume
act's action clones	/root/.cache/act	hostPath /var/lib/gitea-act-cache

act clones each action with full history, not shallow: 66.7MB/538 commits for astral-sh/setup-uv, 24.4MB/222 commits for actions/checkout, and this workflow uses five. After a restart that made Set up job an 11-minute step in which the job container sat idle running sleep while the runner cloned GitHub. If a job looks hung in setup, check the job container before blaming the network:

C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1)
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm

Only sleep means the work is in the runner, not the job.

The cascade this creates. Giving dind a persistent image store made it boot slowly, because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes depending on node load. Two independent timeouts then fire:

1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold boot has still exceeded it.
2. The runner container's own Docker wait timeout of 5m0s, which is internal to the runner image and not configurable from the chart. When dind is late, the runner exits 1 and restarts — and that restart kills whatever job was running, which surfaces as every step in the job failing at once with no error in the log, right after a green Set up job.

The pair self-heals: the second dind boot is fast because the store is warm, and the runner comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and one lost CI run. Restart the runner deliberately, not casually.

10. Verify the whole loop, not just the green checkmarks

# the digest CI pushed
docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
  --format '{{.Manifest.Digest}}'
# the digest the chart deploys — these must be equal
grep -A2 'api:' deploy/chart/values.yaml
# what ArgoCD actually synced
kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'

If those three disagree, the deploy is not what CI tested, and every other guarantee in this document is void.

---

Measured numbers

From scripts/load.py + in-process workers on a FakeProvisioner (delay=0.05s), 200 tasks per run. These came off local Postgres on the same box, with a sub-millisecond round trip. Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach — the shape is what transfers, not the absolute numbers.

replicas	pool max_size	worker concurrency	connections used	drain (200 tasks)	throughput
1	5	4	5	4.3s	46.9 task/s
2	5	4	10	4.2s	47.6 task/s
4	5	4	20	2.2s	92.3 task/s
2	20	16	40	2.1s	97.1 task/s

Three things this says:

1. Going from 1 replica to 2 bought nothing (46.9 → 47.6). Throughput here is bounded by per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated semaphore is the most common wrong fix for a slow queue.
2. Concurrency is the knob that moved it — 4 replicas (20 connections) and 2 replicas at concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the same throughput for twice the connections, which on the free tier is the worse trade.
3. The budget is (api + worker replicas) x max_size, and it is spent whether or not the connections are busy. The bottom row costs 40 connections for a 2% gain over the row above it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count.

Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding it against the actual pooler is the experiment worth running: raise replicas x max_size until claims slow and PoolTimeout appears, and write the number here.

---

Queue stuck

Alert: SvcforgeQueueDepthRising

Diagnose. Start here, always:

sfsql -c "select state, count(*) from tasks group by 1;"

Then split the three causes apart — they look identical from the alert and need opposite fixes:

# Stuck leases: rows 'running' with a locked_at that never advances.
sfsql -c "select kind, locked_by, locked_at, last_error from tasks
          where state='running' order by locked_at limit 10;"

# No workers: is anything actually consuming?
kubectl get pods -l app=worker -o wide
kubectl logs -l app=worker --tail=20 --prefix

# run_after in the future: backoff has parked everything.
sfsql -c "select count(*) from tasks where state='queued' and run_after > now();"

What you see	Cause	Fix
running rows, locked_at older than 5m, no worker pods hold those IDs	Workers died mid-task	None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it.
Zero worker pods, or all CrashLoopBackOff	No consumer	Fix the workers. kubectl describe pod -l app=worker.
Everything queued with run_after far in the future	Backoff, i.e. tasks are failing and retrying	This is not a queue problem. Go to Provision failing.
queued rows with run_after <= now() and healthy workers	Real: claim is not returning rows	Check pooler connection budget (see Supabase full).

Never hand-edit state='running' back to 'queued'. The lease does that, and doing it by hand while the worker is actually alive gives you two workers on one task — the exact thing the whole design prevents.

Escalate if workers are healthy, leases are fresh, and depth still grows: that is a claim-query or pooler bug, not an ops problem.

---

Provision failing

Alert: SvcforgeTaskFailed (tasks reaching the dead-letter state), or SvcforgeProvisionSlow (they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way).

Diagnose:

sfsql -c "select id, service_type, chart_version, error from instances where state='failed';"
sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;"

NS=tenant-<team>
helm list -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20

error looks like	Cause	Fix
chart "..." version "..." not found	Bad pin in catalog.yaml	Correct the version, commit. The next upgrade/provision picks it up.
timed out waiting for the condition	Cluster capacity — the chart installed but pods never became ready	kubectl describe pod -n $NS. Usually Insufficient cpu/memory or a PVC pending on Longhorn.
Error: ... forbidden: User "system:serviceaccount:svcforge:..."	RBAC	The worker's ClusterRole is missing a verb. Chart change, not a manual kubectl edit.
ImagePullBackOff in events	Registry auth or a gated image	Prefer bitnamilegacy/* images, which pull anonymously.

After fixing the cause, tasks that already dead-lettered do not retry themselves. Requeue deliberately:

sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null
          where id = <task_id>;"

Escalate if error is empty on a failed instance — that means the failure path itself lost the message.

---

Orphaned release

Alert: SvcforgeReconcilerStale — the reconciler has not completed a loop recently, so drift is no longer being detected at all. Drift itself is reported in the reconciler's logs and metrics rather than paged on, because it is usually benign and always needs a human to judge. A stale reconciler is the real emergency: nothing is watching.

The control loop never auto-deletes a release. That is deliberate: a bug in the drift check that deletes things is unrecoverable, and one that only reports is a Tuesday.

Diagnose:

helm list -A -o json | jq -r '.[].name' | sort > /tmp/real
sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want

comm -23 /tmp/real /tmp/want   # in the cluster, not in the DB  -> orphan
comm -13 /tmp/real /tmp/want   # in the DB, not in the cluster  -> missing

Direction	Meaning	Action
Orphan (cluster only)	A deprovision half-finished, or someone ran helm install by hand	Confirm the tenant is gone, then helm uninstall <name> -n <ns> by hand, and write down that you did.
Missing (DB only)	Someone deleted a release out from under us	Requeue a provision task for that instance. It is idempotent; it will rebuild.

Escalate before uninstalling anything you did not personally trace to a deleted instance. A wrong helm uninstall here deletes a tenant's data.

---

Supabase full

Alert: none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages you before you hit it; you find out when writes start failing. Until someone adds a size rule, this entry is driven by the calendar, not by an alert. Check it monthly:

sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"

Diagnose:

sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables
          order by pg_total_relation_size(relid) desc limit 5;"
sfsql -c "select count(*) from pg_stat_activity;"

It is almost always tasks. Every provision, upgrade and verify leaves a row forever.

sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';"
sfsql -c "vacuum (analyze) tasks;"

vacuum alone reclaims space for reuse by Postgres, but does not return it to the filesystem — so pg_database_size may barely move. That is expected and fine; the space is free for new rows. vacuum full does return it, takes an ACCESS EXCLUSIVE lock, and will stall every worker for its duration. Only do it in a window, and only if you actually need the bytes back.

If count(*) from pg_stat_activity is near the pooler's ceiling, the cause is arithmetic, not load: worker_replicas × pool_max_size + api_replicas × pool_max_size. Lower max_size or replicas. Replica count is a database-capacity decision here, which is unusual and worth remembering.

Escalate if size is growing with tasks already pruned — that means instances is growing, i.e. tenants are real, i.e. the free tier is the wrong tier.