svcforge/RUNBOOK.md

# svcforge runbook

Four entries. Each starts from an alert firing and ends at either a fix or an escalation.
Every command is copy-pasteable; none of them require thinking at 3am, which is the point.

Set these first:

```bash
set -a; . ~/.config/svcforge/secrets.env; set +a   # SVCFORGE_PG_DSN_SESSION for psql
alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'
```

---

## Setting up CI/CD from scratch

What it takes to get this repo building in Gitea Actions, including every trap that cost
real time. Do it in this order; each step fails loudly if the one before it was skipped.

### 1. The repo and its secrets

```bash
GITEA=https://gitea.oci-oci.duckdns.org
USER=gitea_admin        # the registry namespaces packages by OWNER, so this is IMAGE_NS too

curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge","default_branch":"master","private":false}'

# A PAT with exactly three scopes. Not admin.
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'
```

Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API):

| Secret | Value | Why it exists |
|---|---|---|
| `REGISTRY_USER` | `gitea_admin` | |
| `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. |
| `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit. |

### 2. The runner needs a cache server, and it fails SOFT without one

`cache: enabled: false` in the act_runner config is the default, and it does not fail the
build. It prints:

```
Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.
```

…and carries on, re-downloading every wheel on every run, forever. **A cache that is off
looks exactly like a cache that is always cold.** Enable it in
`oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it):

```yaml
cache:
  enabled: true
  dir: /data/cache     # the runner's PVC, so it survives a restart
  host: ""             # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
  port: 8088
```

```bash
cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea
```

Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to
type=gha` are both no-ops.

### 3. Traps that are specific to Gitea, not GitHub

| Symptom | Cause | Fix |
|---|---|---|
| `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. |
| `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. |
| The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. |
| Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. |

### 4. Keeping trivy green

The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that
passed yesterday fails today without a single line of code changing. Two rules keep it sane.

**Bump the version, do not add an ignore.** The worker image went 39 findings (2 CRITICAL)
→ 18 → 5 → 0 across three fixes, each a version bump or a removal:

| Change | Result |
|---|---|
| `alpine/helm` 3.16.2 → 3.21.3, `kubectl` 1.31.2 → 1.35.3 | 39 → 18, both CRITICALs cleared |
| `kubectl` 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1) | 18 → 5 |
| dropped kubectl entirely — `helm --create-namespace` replaced `kubectl apply` | **5 → 0** |

**The cheapest CVE is the binary you do not ship.** The last five findings lived in
kubectl's vendored `golang.org/x/net` and Go stdlib, inside the newest kubectl that exists —
no version cleared them. kubectl was in that image for exactly one call, and helm already
does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.

### 5. When every job fails at `Set up job` after ~14 minutes

Symptom: `Set up job` runs for 10–15 minutes and succeeds, then the first real step fails
instantly at 0s, and every downstream job is skipped. It looks like the action broke.

Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job
image (`ghcr.io/catthehacker/ubuntu:act-24.04`) before it can start. dind has no volume for
`/var/lib/docker` — the cache lives in its container writable layer and is destroyed on
every pod restart. A runner that is restart-looping therefore never keeps a cache, and each
job pays the full pull.

Check it:
```bash
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images
```
Empty output is the diagnosis. Warm it once by hand:
```bash
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
  ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9
```

The durable fix is to stop the runner restarting. Its `/data` PVC is ReadWriteOnce, so
every reschedule hits `Multi-Attach error` and the pod sits in Init until Longhorn detaches
from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` for exactly
that reason. A dedicated PVC for the image cache would survive restarts outright, but on
this cluster that volume faulted and blocked the runner, so it is deliberately not used.

### 6. Stopping a run, and reading a restarted runner correctly

Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
outcomes have been observed on this cluster:

- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
  15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally

So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
actually working before concluding anything, or you will diagnose a healthy run as a zombie:

```bash
kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
```

Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
run sitting in `waiting` behind a live one is correct, not a block.

**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
so an API token cannot drive it, and cancelling from a script is simply not available.

`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.

To stop a running job: click Cancel in the UI.

### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough

`gitea-postgresql-0` CrashLoopBackOff with:

    mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error

and the Gitea API returning 500, so every CI run dies at checkout with
`Failed to connect to gitea-http:3000`. The "Initializing PostgreSQL database" line above
that error is alarming and is not what it looks like: the data is fine, the mount is broken,
so the container sees an empty directory.

The documented recovery — scale to 0, wait for `detached`, scale back to 1 — was **not
enough** here. The volume came back `detached/faulted` and simply refused to attach, so the
pod sat in ContainerCreating. `auto-salvage: true` does not help: salvage happens during
attach, and a faulted volume never gets that far, so it cannot rescue itself.

What the volume was actually saying:

```bash
V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}')
kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}'   # detached/faulted
kubectl -n longhorn-system get replicas.longhorn.io -o json \
  | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv'
```

The replica carries a `failedAt` timestamp, and that alone is what keeps the volume faulted.
Clearing it is the salvage:

```bash
kubectl -n longhorn-system patch replicas.longhorn.io <replica-name> --type merge \
  -p '{"spec":{"failedAt":"","lastFailedAt":""}}'
```

The volume went `attached/healthy` and postgres reached 1/1 within 40 seconds, with the repo,
its size and the whole CI run history intact.

**Check the backups before patching anything**, because this cluster runs Longhorn at one
replica — there is no second copy to fall back on, only the nightly backup:

```bash
kubectl -n longhorn-system get backups.longhorn.io -o json \
  | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv'
```

Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash
recovery on start. If it cannot, restore the most recent Completed backup instead.

### 8. OPEN: dind is killed by its own liveness probe

Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.

The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:

```
Liveness probe failed: command timed out:
  "/usr/bin/test -S /var/run/docker.sock" timed out after 1s   (x27 over 156m)
Killing: Init container dind failed liveness probe
```

The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.

**Why this matters beyond the runner restarting.** Image builds failed with:

    ERROR: failed to solve: DeadlineExceeded: no active session for <id>

which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.

Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:

```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
  -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```

**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.

The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.

### 9. Verify the whole loop, not just the green checkmarks

```bash
# the digest CI pushed
docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
  --format '{{.Manifest.Digest}}'
# the digest the chart deploys — these must be equal
grep -A2 'api:' deploy/chart/values.yaml
# what ArgoCD actually synced
kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'
```

If those three disagree, the deploy is not what CI tested, and every other guarantee in
this document is void.

---

## Measured numbers

From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks
per run. **These came off local Postgres on the same box, with a sub-millisecond round trip.
Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach** —
the shape is what transfers, not the absolute numbers.

| replicas | pool max_size | worker concurrency | connections used | drain (200 tasks) | throughput |
|---:|---:|---:|---:|---:|---:|
| 1 | 5 | 4 | 5 | 4.3s | 46.9 task/s |
| 2 | 5 | 4 | 10 | 4.2s | 47.6 task/s |
| 4 | 5 | 4 | 20 | 2.2s | 92.3 task/s |
| 2 | 20 | 16 | 40 | 2.1s | 97.1 task/s |

Three things this says:

1. **Going from 1 replica to 2 bought nothing** (46.9 → 47.6). Throughput here is bounded by
   per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated
   semaphore is the most common wrong fix for a slow queue.
2. **Concurrency is the knob that moved it** — 4 replicas (20 connections) and 2 replicas at
   concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the
   same throughput for twice the connections, which on the free tier is the worse trade.
3. **The budget is `(api + worker replicas) x max_size`**, and it is spent whether or not the
   connections are busy. The bottom row costs 40 connections for a 2% gain over the row above
   it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count.

Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding
it against the actual pooler is the experiment worth running: raise `replicas x max_size`
until claims slow and `PoolTimeout` appears, and write the number here.

---

## Queue stuck

**Alert:** `SvcforgeQueueDepthRising`

**Diagnose.** Start here, always:

```bash
sfsql -c "select state, count(*) from tasks group by 1;"
```

Then split the three causes apart — they look identical from the alert and need opposite fixes:

```bash
# Stuck leases: rows 'running' with a locked_at that never advances.
sfsql -c "select kind, locked_by, locked_at, last_error from tasks
          where state='running' order by locked_at limit 10;"

# No workers: is anything actually consuming?
kubectl get pods -l app=worker -o wide
kubectl logs -l app=worker --tail=20 --prefix

# run_after in the future: backoff has parked everything.
sfsql -c "select count(*) from tasks where state='queued' and run_after > now();"
```

| What you see | Cause | Fix |
|---|---|---|
| `running` rows, `locked_at` older than 5m, no worker pods hold those IDs | Workers died mid-task | None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it. |
| Zero worker pods, or all `CrashLoopBackOff` | No consumer | Fix the workers. `kubectl describe pod -l app=worker`. |
| Everything `queued` with `run_after` far in the future | Backoff, i.e. tasks are failing and retrying | This is not a queue problem. Go to **Provision failing**. |
| `queued` rows with `run_after <= now()` and healthy workers | Real: claim is not returning rows | Check pooler connection budget (see **Supabase full**). |

**Never** hand-edit `state='running'` back to `'queued'`. The lease does that, and doing it
by hand while the worker is actually alive gives you two workers on one task — the exact
thing the whole design prevents.

**Escalate** if workers are healthy, leases are fresh, and depth still grows: that is a
claim-query or pooler bug, not an ops problem.

---

## Provision failing

**Alert:** `SvcforgeTaskFailed` (tasks reaching the dead-letter state), or `SvcforgeProvisionSlow`
(they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way).

**Diagnose:**

```bash
sfsql -c "select id, service_type, chart_version, error from instances where state='failed';"
sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;"

NS=tenant-<team>
helm list -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20
```

| `error` looks like | Cause | Fix |
|---|---|---|
| `chart "..." version "..." not found` | Bad pin in `catalog.yaml` | Correct the version, commit. The next `upgrade`/`provision` picks it up. |
| `timed out waiting for the condition` | Cluster capacity — the chart installed but pods never became ready | `kubectl describe pod -n $NS`. Usually `Insufficient cpu/memory` or a PVC pending on Longhorn. |
| `Error: ... forbidden: User "system:serviceaccount:svcforge:..."` | RBAC | The worker's ClusterRole is missing a verb. Chart change, not a manual `kubectl edit`. |
| `ImagePullBackOff` in events | Registry auth or a gated image | Prefer `bitnamilegacy/*` images, which pull anonymously. |

After fixing the cause, tasks that already dead-lettered do **not** retry themselves. Requeue
deliberately:

```bash
sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null
          where id = <task_id>;"
```

**Escalate** if `error` is empty on a failed instance — that means the failure path itself
lost the message.

---

## Orphaned release

**Alert:** `SvcforgeReconcilerStale` — the reconciler has not completed a loop recently, so
drift is no longer being *detected* at all. Drift itself is reported in the reconciler's logs
and metrics rather than paged on, because it is usually benign and always needs a human to
judge. A stale reconciler is the real emergency: nothing is watching.

The control loop **never auto-deletes a release.** That is deliberate: a bug in the drift
check that deletes things is unrecoverable, and one that only reports is a Tuesday.

**Diagnose:**

```bash
helm list -A -o json | jq -r '.[].name' | sort > /tmp/real
sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want

comm -23 /tmp/real /tmp/want   # in the cluster, not in the DB  -> orphan
comm -13 /tmp/real /tmp/want   # in the DB, not in the cluster  -> missing
```

| Direction | Meaning | Action |
|---|---|---|
| Orphan (cluster only) | A deprovision half-finished, or someone ran `helm install` by hand | Confirm the tenant is gone, then `helm uninstall <name> -n <ns>` **by hand**, and write down that you did. |
| Missing (DB only) | Someone deleted a release out from under us | Requeue a `provision` task for that instance. It is idempotent; it will rebuild. |

**Escalate** before uninstalling anything you did not personally trace to a deleted instance.
A wrong `helm uninstall` here deletes a tenant's data.

---

## Supabase full

**Alert:** none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages
you before you hit it; you find out when writes start failing. Until someone adds a size rule,
this entry is driven by the calendar, not by an alert. Check it monthly:

```bash
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
```

**Diagnose:**

```bash
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables
          order by pg_total_relation_size(relid) desc limit 5;"
sfsql -c "select count(*) from pg_stat_activity;"
```

It is almost always `tasks`. Every provision, upgrade and verify leaves a row forever.

```bash
sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';"
sfsql -c "vacuum (analyze) tasks;"
```

`vacuum` alone reclaims space **for reuse by Postgres**, but does not return it to the
filesystem — so `pg_database_size` may barely move. That is expected and fine; the space is
free for new rows. `vacuum full` does return it, takes an `ACCESS EXCLUSIVE` lock, and will
stall every worker for its duration. Only do it in a window, and only if you actually need
the bytes back.

If `count(*) from pg_stat_activity` is near the pooler's ceiling, the cause is arithmetic, not
load: `worker_replicas × pool_max_size + api_replicas × pool_max_size`. Lower `max_size` or
replicas. **Replica count is a database-capacity decision here**, which is unusual and worth
remembering.

**Escalate** if size is growing with `tasks` already pruned — that means `instances` is
growing, i.e. tenants are real, i.e. the free tier is the wrong tier.