svcforge/RUNBOOK.md

svcforge runbook

Four entries. Each starts from an alert firing and ends at either a fix or an escalation. Every command is copy-pasteable; none of them require thinking at 3am, which is the point.

Set these first:

set -a; . ~/.config/svcforge/secrets.env; set +a   # SVCFORGE_PG_DSN_SESSION for psql
alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'

---

Setting up CI/CD from scratch

What it takes to get this repo building in Gitea Actions, including every trap that cost real time. Do it in this order; each step fails loudly if the one before it was skipped.

1. The repo and its secrets

GITEA=https://gitea.oci-oci.duckdns.org
USER=gitea_admin        # the registry namespaces packages by OWNER, so this is IMAGE_NS too

curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge","default_branch":"master","private":false}'

# A PAT with exactly three scopes. Not admin.
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'

Then set three repo Actions secrets (Settings → Actions → Secrets, or the API):

Secret	Value	Why it exists
REGISTRY_USER	gitea_admin
REGISTRY_TOKEN	the PAT	The auto-injected GITEA_TOKEN is rejected by the package registry with a 401. This is the single most common reason a first pipeline fails at docker push.
CI_BOT_TOKEN	the PAT	Used only by the bump job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit.

2. The runner needs a cache server, and it fails SOFT without one

cache: enabled: false in the act_runner config is the default, and it does not fail the build. It prints:

Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.

…and carries on, re-downloading every wheel on every run, forever. A cache that is off looks exactly like a cache that is always cold. Enable it in oci-k8s/k8s/roles/addons/tasks/main.yml (Ansible owns this — never kubectl edit it):

cache:
  enabled: true
  dir: /data/cache     # the runner's PVC, so it survives a restart
  host: ""             # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
  port: 8088

cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea

Without this, astral-sh/setup-uv's enable-cache: true and buildx's --cache-to type=gha are both no-ops.

3. Traps that are specific to Gitea, not GitHub

Symptom	Cause	Fix
Unable to resolve v0.2.2: reference not found	A third-party action pinned by SHA still resolves its own dependencies by mutable tag. trivy-action@v0.29.0 does uses: setup-trivy@v0.2.2, and that tag was removed upstream.	Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing.
docker push → 401	Used GITEA_TOKEN.	Use a PAT with write:package.
The pipeline retriggers itself forever	The bump job commits to the repo it is triggered by.	[skip ci] in the commit message (Gitea honours it), and a git diff --quiet guard so an unchanged digest commits nothing.
Service container unreachable at localhost	Jobs run inside a container, so a service is reached by its service name, not localhost.	postgres:5432, not localhost:5432.

4. Keeping trivy green

The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that passed yesterday fails today without a single line of code changing. Two rules keep it sane.

Bump the version, do not add an ignore. The worker image went 39 findings (2 CRITICAL) → 18 → 5 → 0 across three fixes, each a version bump or a removal:

Change	Result
alpine/helm 3.16.2 → 3.21.3, kubectl 1.31.2 → 1.35.3	39 → 18, both CRITICALs cleared
kubectl 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1)	18 → 5
dropped kubectl entirely — helm --create-namespace replaced kubectl apply	5 → 0

The cheapest CVE is the binary you do not ship. The last five findings lived in kubectl's vendored golang.org/x/net and Go stdlib, inside the newest kubectl that exists — no version cleared them. kubectl was in that image for exactly one call, and helm already does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.

5. When every job fails at Set up job after ~14 minutes

Symptom: Set up job runs for 10–15 minutes and succeeds, then the first real step fails instantly at 0s, and every downstream job is skipped. It looks like the action broke.

Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job image (ghcr.io/catthehacker/ubuntu:act-24.04) before it can start. dind has no volume for /var/lib/docker — the cache lives in its container writable layer and is destroyed on every pod restart. A runner that is restart-looping therefore never keeps a cache, and each job pays the full pull.

Check it:

kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images

Empty output is the diagnosis. Warm it once by hand:

kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
  ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9

The durable fix is to stop the runner restarting. Its /data PVC is ReadWriteOnce, so every reschedule hits Multi-Attach error and the pod sits in Init until Longhorn detaches from the old node. It is pinned to node2 in oci-k8s/.../addons/tasks/main.yml for exactly that reason. A dedicated PVC for the image cache would survive restarts outright, but on this cluster that volume faulted and blocked the runner, so it is deliberately not used.

6. Verify the whole loop, not just the green checkmarks

# the digest CI pushed
docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
  --format '{{.Manifest.Digest}}'
# the digest the chart deploys — these must be equal
grep -A2 'api:' deploy/chart/values.yaml
# what ArgoCD actually synced
kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'

If those three disagree, the deploy is not what CI tested, and every other guarantee in this document is void.

---

Measured numbers

From scripts/load.py + in-process workers on a FakeProvisioner (delay=0.05s), 200 tasks per run. These came off local Postgres on the same box, with a sub-millisecond round trip. Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach — the shape is what transfers, not the absolute numbers.

replicas	pool max_size	worker concurrency	connections used	drain (200 tasks)	throughput
1	5	4	5	4.3s	46.9 task/s
2	5	4	10	4.2s	47.6 task/s
4	5	4	20	2.2s	92.3 task/s
2	20	16	40	2.1s	97.1 task/s

Three things this says:

1. Going from 1 replica to 2 bought nothing (46.9 → 47.6). Throughput here is bounded by per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated semaphore is the most common wrong fix for a slow queue.
2. Concurrency is the knob that moved it — 4 replicas (20 connections) and 2 replicas at concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the same throughput for twice the connections, which on the free tier is the worse trade.
3. The budget is (api + worker replicas) x max_size, and it is spent whether or not the connections are busy. The bottom row costs 40 connections for a 2% gain over the row above it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count.

Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding it against the actual pooler is the experiment worth running: raise replicas x max_size until claims slow and PoolTimeout appears, and write the number here.

---

Queue stuck

Alert: SvcforgeQueueDepthRising

Diagnose. Start here, always:

sfsql -c "select state, count(*) from tasks group by 1;"

Then split the three causes apart — they look identical from the alert and need opposite fixes:

# Stuck leases: rows 'running' with a locked_at that never advances.
sfsql -c "select kind, locked_by, locked_at, last_error from tasks
          where state='running' order by locked_at limit 10;"

# No workers: is anything actually consuming?
kubectl get pods -l app=worker -o wide
kubectl logs -l app=worker --tail=20 --prefix

# run_after in the future: backoff has parked everything.
sfsql -c "select count(*) from tasks where state='queued' and run_after > now();"

What you see	Cause	Fix
running rows, locked_at older than 5m, no worker pods hold those IDs	Workers died mid-task	None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it.
Zero worker pods, or all CrashLoopBackOff	No consumer	Fix the workers. kubectl describe pod -l app=worker.
Everything queued with run_after far in the future	Backoff, i.e. tasks are failing and retrying	This is not a queue problem. Go to Provision failing.
queued rows with run_after <= now() and healthy workers	Real: claim is not returning rows	Check pooler connection budget (see Supabase full).

Never hand-edit state='running' back to 'queued'. The lease does that, and doing it by hand while the worker is actually alive gives you two workers on one task — the exact thing the whole design prevents.

Escalate if workers are healthy, leases are fresh, and depth still grows: that is a claim-query or pooler bug, not an ops problem.

---

Provision failing

Alert: SvcforgeTaskFailed (tasks reaching the dead-letter state), or SvcforgeProvisionSlow (they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way).

Diagnose:

sfsql -c "select id, service_type, chart_version, error from instances where state='failed';"
sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;"

NS=tenant-<team>
helm list -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20

error looks like	Cause	Fix
chart "..." version "..." not found	Bad pin in catalog.yaml	Correct the version, commit. The next upgrade/provision picks it up.
timed out waiting for the condition	Cluster capacity — the chart installed but pods never became ready	kubectl describe pod -n $NS. Usually Insufficient cpu/memory or a PVC pending on Longhorn.
Error: ... forbidden: User "system:serviceaccount:svcforge:..."	RBAC	The worker's ClusterRole is missing a verb. Chart change, not a manual kubectl edit.
ImagePullBackOff in events	Registry auth or a gated image	Prefer bitnamilegacy/* images, which pull anonymously.

After fixing the cause, tasks that already dead-lettered do not retry themselves. Requeue deliberately:

sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null
          where id = <task_id>;"

Escalate if error is empty on a failed instance — that means the failure path itself lost the message.

---

Orphaned release

Alert: SvcforgeReconcilerStale — the reconciler has not completed a loop recently, so drift is no longer being detected at all. Drift itself is reported in the reconciler's logs and metrics rather than paged on, because it is usually benign and always needs a human to judge. A stale reconciler is the real emergency: nothing is watching.

The control loop never auto-deletes a release. That is deliberate: a bug in the drift check that deletes things is unrecoverable, and one that only reports is a Tuesday.

Diagnose:

helm list -A -o json | jq -r '.[].name' | sort > /tmp/real
sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want

comm -23 /tmp/real /tmp/want   # in the cluster, not in the DB  -> orphan
comm -13 /tmp/real /tmp/want   # in the DB, not in the cluster  -> missing

Direction	Meaning	Action
Orphan (cluster only)	A deprovision half-finished, or someone ran helm install by hand	Confirm the tenant is gone, then helm uninstall <name> -n <ns> by hand, and write down that you did.
Missing (DB only)	Someone deleted a release out from under us	Requeue a provision task for that instance. It is idempotent; it will rebuild.

Escalate before uninstalling anything you did not personally trace to a deleted instance. A wrong helm uninstall here deletes a tenant's data.

---

Supabase full

Alert: none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages you before you hit it; you find out when writes start failing. Until someone adds a size rule, this entry is driven by the calendar, not by an alert. Check it monthly:

sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"

Diagnose:

sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables
          order by pg_total_relation_size(relid) desc limit 5;"
sfsql -c "select count(*) from pg_stat_activity;"

It is almost always tasks. Every provision, upgrade and verify leaves a row forever.

sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';"
sfsql -c "vacuum (analyze) tasks;"

vacuum alone reclaims space for reuse by Postgres, but does not return it to the filesystem — so pg_database_size may barely move. That is expected and fine; the space is free for new rows. vacuum full does return it, takes an ACCESS EXCLUSIVE lock, and will stall every worker for its duration. Only do it in a window, and only if you actually need the bytes back.

If count(*) from pg_stat_activity is near the pooler's ceiling, the cause is arithmetic, not load: worker_replicas × pool_max_size + api_replicas × pool_max_size. Lower max_size or replicas. Replica count is a database-capacity decision here, which is unusual and worth remembering.

Escalate if size is growing with tasks already pruned — that means instances is growing, i.e. tenants are real, i.e. the free tier is the wrong tier.