svcforge/RUNBOOK.md

# svcforge runbook

Four entries. Each starts from an alert firing and ends at either a fix or an escalation.
Every command is copy-pasteable; none of them require thinking at 3am, which is the point.

Set these first:

```bash
set -a; . ~/.config/svcforge/secrets.env; set +a   # SVCFORGE_PG_DSN_SESSION for psql
alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"'
```

---

## Setting up CI/CD from scratch

What it takes to get this repo building in Gitea Actions, including every trap that cost
real time. Do it in this order; each step fails loudly if the one before it was skipped.

### 1. The repo and its secrets

```bash
GITEA=https://gitea.oci-oci.duckdns.org
USER=gitea_admin        # the registry namespaces packages by OWNER, so this is IMAGE_NS too

curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge","default_branch":"master","private":false}'

# A PAT with exactly three scopes. Not admin.
curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \
  -H 'content-type: application/json' \
  -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}'
```

Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API):

| Secret | Value | Why it exists |
|---|---|---|
| `REGISTRY_USER` | `gitea_admin` | |
| `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. |
| `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit. |

### 2. The runner needs a cache server, and it fails SOFT without one

`cache: enabled: false` in the act_runner config is the default, and it does not fail the
build. It prints:

```
Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache.
```

…and carries on, re-downloading every wheel on every run, forever. **A cache that is off
looks exactly like a cache that is always cold.** Enable it in
`oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it):

```yaml
cache:
  enabled: true
  dir: /data/cache     # the runner's PVC, so it survives a restart
  host: ""             # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container
  port: 8088
```

```bash
cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea
```

Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to
type=gha` are both no-ops.

### 3. Traps that are specific to Gitea, not GitHub

| Symptom | Cause | Fix |
|---|---|---|
| `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. |
| `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. |
| The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. |
| Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. |

### 4. Keeping trivy green

The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that
passed yesterday fails today without a single line of code changing. Two rules keep it sane.

**Bump the version, do not add an ignore.** The worker image went 39 findings (2 CRITICAL)
→ 18 → 5 → 0 across three fixes, each a version bump or a removal:

| Change | Result |
|---|---|
| `alpine/helm` 3.16.2 → 3.21.3, `kubectl` 1.31.2 → 1.35.3 | 39 → 18, both CRITICALs cleared |
| `kubectl` 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1) | 18 → 5 |
| dropped kubectl entirely — `helm --create-namespace` replaced `kubectl apply` | **5 → 0** |

**The cheapest CVE is the binary you do not ship.** The last five findings lived in
kubectl's vendored `golang.org/x/net` and Go stdlib, inside the newest kubectl that exists —
no version cleared them. kubectl was in that image for exactly one call, and helm already
does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter.

### 5. When every job fails at `Set up job` after ~14 minutes

Symptom: `Set up job` runs for 10–15 minutes and succeeds, then the first real step fails
instantly at 0s, and every downstream job is skipped. It looks like the action broke.

Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job
image (`ghcr.io/catthehacker/ubuntu:act-24.04`) before it can start. dind has no volume for
`/var/lib/docker` — the cache lives in its container writable layer and is destroyed on
every pod restart. A runner that is restart-looping therefore never keeps a cache, and each
job pays the full pull.

Check it:
```bash
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images
```
Empty output is the diagnosis. Warm it once by hand:
```bash
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
  ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9
```

The durable fix is a persistent image store, which the runner now has: `/var/lib/docker`
is a hostPath on node0 (see `oci-k8s/.../addons/tasks/main.yml`), so the act image survives
a restart and is not re-pulled. The runner is pinned to **node0**, not node2 — node2 is a
single-core control-plane node whose pod network was measured 21x slower under its own
load, which starved every clone and pull. Its `/data` PVC is NFS ReadWriteMany, so a
reschedule attaches immediately with no `Multi-Attach` wait. Entries 9 and 10 cover the
caches and the node move in full.

### 6. Stopping a run, and reading a restarted runner correctly

Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both
outcomes have been observed on this cluster:

- run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at
  15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for
- run #16 had its remaining jobs re-dispatched to the new pod and carried on normally

So `in_progress` after a restart is ambiguous on its own. Check whether the runner is
actually working before concluding anything, or you will diagnose a healthy run as a zombie:

```bash
kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4
```

Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later
run sitting in `waiting` behind a live one is correct, not a block.

**Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching
`cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route,
so an API token cannot drive it, and cancelling from a script is simply not available.

`DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database
id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping
it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel.

To stop a running job: click Cancel in the UI.

### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough

`gitea-postgresql-0` CrashLoopBackOff with:

    mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error

and the Gitea API returning 500, so every CI run dies at checkout with
`Failed to connect to gitea-http:3000`. The "Initializing PostgreSQL database" line above
that error is alarming and is not what it looks like: the data is fine, the mount is broken,
so the container sees an empty directory.

The documented recovery — scale to 0, wait for `detached`, scale back to 1 — was **not
enough** here. The volume came back `detached/faulted` and simply refused to attach, so the
pod sat in ContainerCreating. `auto-salvage: true` does not help: salvage happens during
attach, and a faulted volume never gets that far, so it cannot rescue itself.

What the volume was actually saying:

```bash
V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}')
kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}'   # detached/faulted
kubectl -n longhorn-system get replicas.longhorn.io -o json \
  | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv'
```

The replica carries a `failedAt` timestamp, and that alone is what keeps the volume faulted.
Clearing it is the salvage:

```bash
kubectl -n longhorn-system patch replicas.longhorn.io <replica-name> --type merge \
  -p '{"spec":{"failedAt":"","lastFailedAt":""}}'
```

The volume went `attached/healthy` and postgres reached 1/1 within 40 seconds, with the repo,
its size and the whole CI run history intact.

**Check the backups before patching anything**, because this cluster runs Longhorn at one
replica — there is no second copy to fall back on, only the nightly backup:

```bash
kubectl -n longhorn-system get backups.longhorn.io -o json \
  | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv'
```

Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash
recovery on start. If it cannot, restore the most recent Completed backup instead.

### 8. OPEN: dind is killed by its own liveness probe

Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were
diagnosed as something else.

The runner sits at `Init:1/2` and its dind sidecar accumulates restarts:

```
Liveness probe failed: command timed out:
  "/usr/bin/test -S /var/run/docker.sock" timed out after 1s   (x27 over 156m)
Killing: Init container dind failed liveness probe
```

The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`.
`test -S` only asks whether a socket exists. When that cannot finish inside a second, the
node is starved rather than dind being unhealthy, and kubelet kills a working daemon.

**Why this matters beyond the runner restarting.** Image builds failed with:

    ERROR: failed to solve: DeadlineExceeded: no active session for <id>

which was attributed to CPU starvation alone and addressed by dropping the runner's
`capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed
dind mid-build and the buildkit session died with it. Lowering capacity reduced the load
that trips the probe, which is consistent with run #17 passing — it treated the cause of
the trigger, not the trigger. Runs #18-#21 then failed anyway.

Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the
kill timestamps against the failed builds:

```bash
kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events:
kubectl -n gitea get pod gitea-actions-runner-0 \
  -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}'
```

**The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match
for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is
reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem.

The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to
roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates
workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 —
so the precedent and the tooling are both in place.

### 9. The runner's three ephemeral caches, and the boot cascade they cause

Three separate caches on this runner were container-layer only, each found the same way —
something was slow, and the cause was a cache that had never survived a restart:

| cache | path | fixed by |
|---|---|---|
| dind image store | `/var/lib/docker` | hostPath `/var/lib/gitea-dind` |
| trivy vulnerability DB | `/root/.cache/trivy` in dind | named docker volume |
| act's action clones | `/root/.cache/act` | hostPath `/var/lib/gitea-act-cache` |

act clones each action with **full history**, not shallow: 66.7MB/538 commits for
`astral-sh/setup-uv`, 24.4MB/222 commits for `actions/checkout`, and this workflow uses
five. After a restart that made `Set up job` an 11-minute step in which the job container
sat idle running `sleep` while the runner cloned GitHub. If a job looks hung in setup, check
the job container before blaming the network:

```bash
C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1)
kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm
```

Only `sleep` means the work is in the runner, not the job.

**The cascade this creates.** Giving dind a persistent image store made it boot slowly,
because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes
depending on node load. Two independent timeouts then fire:

1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold
   boot has still exceeded it.
2. **The runner container's own `Docker wait timeout of 5m0s`**, which is internal to the
   runner image and not configurable from the chart. When dind is late, the runner exits 1
   and restarts — and that restart kills whatever job was running, which surfaces as every
   step in the job failing at once with no error in the log, right after a green
   `Set up job`.

The pair self-heals: the second dind boot is fast because the store is warm, and the runner
comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and
one lost CI run. Restart the runner deliberately, not casually.

### 10. Verify the whole loop, not just the green checkmarks

```bash
# the digest CI pushed
docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api:<sha> \
  --format '{{.Manifest.Digest}}'
# the digest the chart deploys — these must be equal
grep -A2 'api:' deploy/chart/values.yaml
# what ArgoCD actually synced
kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}'
```

If those three disagree, the deploy is not what CI tested, and every other guarantee in
this document is void.

---

## Measured numbers

From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks
per run. **These came off local Postgres on the same box, with a sub-millisecond round trip.
Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach** —
the shape is what transfers, not the absolute numbers.

| replicas | pool max_size | worker concurrency | connections used | drain (200 tasks) | throughput |
|---:|---:|---:|---:|---:|---:|
| 1 | 5 | 4 | 5 | 4.3s | 46.9 task/s |
| 2 | 5 | 4 | 10 | 4.2s | 47.6 task/s |
| 4 | 5 | 4 | 20 | 2.2s | 92.3 task/s |
| 2 | 20 | 16 | 40 | 2.1s | 97.1 task/s |

Three things this says:

1. **Going from 1 replica to 2 bought nothing** (46.9 → 47.6). Throughput here is bounded by
   per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated
   semaphore is the most common wrong fix for a slow queue.
2. **Concurrency is the knob that moved it** — 4 replicas (20 connections) and 2 replicas at
   concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the
   same throughput for twice the connections, which on the free tier is the worse trade.
3. **The budget is `(api + worker replicas) x max_size`**, and it is spent whether or not the
   connections are busy. The bottom row costs 40 connections for a 2% gain over the row above
   it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count.

Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding
it against the actual pooler is the experiment worth running: raise `replicas x max_size`
until claims slow and `PoolTimeout` appears, and write the number here.

---

## Queue stuck

**Alert:** `SvcforgeQueueDepthRising`

**Diagnose.** Start here, always:

```bash
sfsql -c "select state, count(*) from tasks group by 1;"
```

Then split the three causes apart — they look identical from the alert and need opposite fixes:

```bash
# Stuck leases: rows 'running' with a locked_at that never advances.
sfsql -c "select kind, locked_by, locked_at, last_error from tasks
          where state='running' order by locked_at limit 10;"

# No workers: is anything actually consuming?
kubectl get pods -l app=worker -o wide
kubectl logs -l app=worker --tail=20 --prefix

# run_after in the future: backoff has parked everything.
sfsql -c "select count(*) from tasks where state='queued' and run_after > now();"
```

| What you see | Cause | Fix |
|---|---|---|
| `running` rows, `locked_at` older than 5m, no worker pods hold those IDs | Workers died mid-task | None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it. |
| Zero worker pods, or all `CrashLoopBackOff` | No consumer | Fix the workers. `kubectl describe pod -l app=worker`. |
| Everything `queued` with `run_after` far in the future | Backoff, i.e. tasks are failing and retrying | This is not a queue problem. Go to **Provision failing**. |
| `queued` rows with `run_after <= now()` and healthy workers | Real: claim is not returning rows | Check pooler connection budget (see **Supabase full**). |

**Never** hand-edit `state='running'` back to `'queued'`. The lease does that, and doing it
by hand while the worker is actually alive gives you two workers on one task — the exact
thing the whole design prevents.

**Escalate** if workers are healthy, leases are fresh, and depth still grows: that is a
claim-query or pooler bug, not an ops problem.

---

## Provision failing

**Alert:** `SvcforgeTaskFailed` (tasks reaching the dead-letter state), or `SvcforgeProvisionSlow`
(they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way).

**Diagnose:**

```bash
sfsql -c "select id, service_type, chart_version, error from instances where state='failed';"
sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;"

NS=tenant-<team>
helm list -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20
```

| `error` looks like | Cause | Fix |
|---|---|---|
| `chart "..." version "..." not found` | Bad pin in `catalog.yaml` | Correct the version, commit. The next `upgrade`/`provision` picks it up. |
| `timed out waiting for the condition` | Cluster capacity — the chart installed but pods never became ready | `kubectl describe pod -n $NS`. Usually `Insufficient cpu/memory` or a PVC pending on Longhorn. |
| `Error: ... forbidden: User "system:serviceaccount:svcforge:..."` | RBAC | The worker's ClusterRole is missing a verb. Chart change, not a manual `kubectl edit`. |
| `ImagePullBackOff` in events | Registry auth or a gated image | Prefer `bitnamilegacy/*` images, which pull anonymously. |

After fixing the cause, tasks that already dead-lettered do **not** retry themselves. Requeue
deliberately:

```bash
sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null
          where id = <task_id>;"
```

**Escalate** if `error` is empty on a failed instance — that means the failure path itself
lost the message.

---

## Orphaned release

**Alert:** `SvcforgeReconcilerStale` — the reconciler has not completed a loop recently, so
drift is no longer being *detected* at all. Drift itself is reported in the reconciler's logs
and metrics rather than paged on, because it is usually benign and always needs a human to
judge. A stale reconciler is the real emergency: nothing is watching.

The control loop **never auto-deletes a release.** That is deliberate: a bug in the drift
check that deletes things is unrecoverable, and one that only reports is a Tuesday.

**Diagnose:**

```bash
helm list -A -o json | jq -r '.[].name' | sort > /tmp/real
sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want

comm -23 /tmp/real /tmp/want   # in the cluster, not in the DB  -> orphan
comm -13 /tmp/real /tmp/want   # in the DB, not in the cluster  -> missing
```

| Direction | Meaning | Action |
|---|---|---|
| Orphan (cluster only) | A deprovision half-finished, or someone ran `helm install` by hand | Confirm the tenant is gone, then `helm uninstall <name> -n <ns>` **by hand**, and write down that you did. |
| Missing (DB only) | Someone deleted a release out from under us | Requeue a `provision` task for that instance. It is idempotent; it will rebuild. |

**Escalate** before uninstalling anything you did not personally trace to a deleted instance.
A wrong `helm uninstall` here deletes a tenant's data.

---

## Supabase full

**Alert:** none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages
you before you hit it; you find out when writes start failing. Until someone adds a size rule,
this entry is driven by the calendar, not by an alert. Check it monthly:

```bash
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
```

**Diagnose:**

```bash
sfsql -c "select pg_size_pretty(pg_database_size(current_database()));"
sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables
          order by pg_total_relation_size(relid) desc limit 5;"
sfsql -c "select count(*) from pg_stat_activity;"
```

It is almost always `tasks`. Every provision, upgrade and verify leaves a row forever.

```bash
sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';"
sfsql -c "vacuum (analyze) tasks;"
```

`vacuum` alone reclaims space **for reuse by Postgres**, but does not return it to the
filesystem — so `pg_database_size` may barely move. That is expected and fine; the space is
free for new rows. `vacuum full` does return it, takes an `ACCESS EXCLUSIVE` lock, and will
stall every worker for its duration. Only do it in a window, and only if you actually need
the bytes back.

If `count(*) from pg_stat_activity` is near the pooler's ceiling, the cause is arithmetic, not
load: `worker_replicas × pool_max_size + api_replicas × pool_max_size`. Lower `max_size` or
replicas. **Replica count is a database-capacity decision here**, which is unusual and worth
remembering.

**Escalate** if size is growing with `tasks` already pruned — that means `instances` is
growing, i.e. tenants are real, i.e. the free tier is the wrong tier.

---

## Deploy stuck: ArgoCD says Synced at an old commit

**Symptom:** CI is green and the bump commit is on `master`, but the running pods are on the
previous digest. `kubectl -n argocd get application svcforge` says **`Synced`** — at a
revision several commits behind. Nothing looks broken, which is what makes this expensive.

**Diagnose.** Compare what ArgoCD thinks it synced against what `master` actually is:

```bash
kubectl -n argocd get application svcforge \
  -o jsonpath='{.status.sync.revision}{"  reconciledAt="}{.status.reconciledAt}{"\n"}'
git -C ~/workspace/svcforge-reference log --oneline origin/master -1
```

**`reconciledAt` alone does not tell you.** ArgoCD writes that field only when the computed
status *changes*, so on a cluster where everything is Synced and nothing is deploying it can
sit still while the controller is fine. It is evidence only when paired with a
`sync.revision` that is *behind `master`* — which is exactly the case you are in if you are
reading this section.

**One metric answers it, and only one.** The controller observes
`argocd_app_reconcile_count` once per completed app reconciliation. Flat means it is doing
nothing:

```bash
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- --post-data='query=sum(increase(argocd_app_reconcile_count[10m]))' \
  http://localhost:9090/api/v1/query
```

**Two metrics that look like they answer it and do not.** Both were tried on 2026-07-22
against a controller that had reconciled nothing for 82 minutes, and both read healthy:

| metric | reading at the time | why it lies |
|---|---|---|
| `argocd_redis_request_total` | 45 reads / 15m, climbing | something in the process still touches the cache when nothing is reconciling — it measures "the pod is running", which `up` already covers |
| `workqueue_unfinished_work_seconds{name="app_reconciliation_queue"}` | 0 | it only counts work already *in* the queue, and the queue is empty. Nothing is stuck; nothing is being enqueued |

Keep the second one anyway — it catches a genuinely stuck queue item, which is a different
failure. Just never read a zero from it as "healthy".

**What is actually wrong: the periodic git poll does not run.** Read the controller's own
metrics on an idle cluster:

```bash
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \
  | grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"'
```

Measured 2026-07-22, across two separate controller pods, on a fully healthy API server:

```
03:37:03  controller starts, refreshes all 3 apps   adds_total = 3
03:38:31  adds_total = 3
03:41:33  adds_total = 3
03:44:34  adds_total = 3
03:47:35  adds_total = 3      <- expiry is 2m0s, jitter 60s
```

`workqueue_depth 0`, `longest_running_processor_seconds 0`, `adds_total` frozen. Nothing is
*blocked* — nothing is being **enqueued**. The controller logs its own schedule at startup
(`appResyncPeriod=2m0s, appResyncJitter=1m0s`) and `argocd-cm` carries the matching
`timeout.reconciliation: 120s`, so the setting is read and then never acted on.

Refreshes still happen from two other paths, which is what makes this so easy to
misread as working:

| path | fires when | observed |
|---|---|---|
| startup | controller (re)starts | 3 apps refreshed within ~2s of ready |
| cluster events | a watched resource changes | 68 adds during one svcforge rollout, then flat the moment the cluster went quiet |
| periodic poll | every 2m ± 60s | **never** |

**The consequence is the thing to take away: a commit that changes only the repo is never
noticed.** Every "auto-sync" observed on 2026-07-22 happened within seconds of a controller
restart, i.e. it was the startup refresh, not the poll. Do not read a successful deploy
straight after a restart as evidence that polling works.

**A webhook now covers for it** (added 2026-07-22, `oci-k8s` `--tags argocd,gitea`). Gitea
POSTs every push to `https://argocd.oci-oci.duckdns.org/api/webhook`, so a commit refreshes
ArgoCD in under a second instead of waiting for a poll that never comes.

It is registered as Gitea's **`gogs`** type, which looks wrong and is not: ArgoCD's webhook
handler dispatches on the `X-Gogs-Event` header and ships no Gitea parser. Gitea forked from
Gogs and still emits that wire format on request. The shared secret lives in
`argocd-secret` as `webhook.gogs.secret` and in Ansible as `argocd_webhook_secret`; both
sides must match or every delivery fails signature validation *silently*, which looks
identical to having no webhook at all.

Check a delivery when a push does not deploy:

```bash
# Gitea's own record of the last attempt, including the response ArgoCD gave
curl -s -u "$USER:$PASS" \
  https://gitea.oci-oci.duckdns.org/api/v1/repos/gitea_admin/svcforge/hooks | jq '.[].id'

# ArgoCD's side
kubectl -n argocd logs deploy/argocd-server --tail=200 | grep -i webhook
```

`Unknown webhook event` means the hook type is wrong (it must be `gogs`). A 400 on signature
means the secrets have drifted — re-run `03_install_addons.yml --tags argocd,gitea`, which
rewrites both ends from the same variable.

Manual nudge, still valid if the webhook is ever down:

```bash
kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=normal --overwrite
```

Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22
wedge before the metrics were checked:

```bash
# Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.
kubectl -n argocd logs statefulset/argocd-application-controller --tail=8000 \
  | grep -oE 'time="[0-9-]+T[0-9]{2}' | sort | uniq -c | tail
```

A flat `Goroutines=NNN` across hours in those heartbeat lines means blocked goroutines, not
an idle controller.

**Cause seen here (2026-07-21).** Not ArgoCD config — `timeout.reconciliation` was 120s the
whole time. The controller's server-side dry-run applies go through the cluster's admission
webhooks, and Kyverno's mutate webhook was `failurePolicy: Fail`. Kyverno restarts under
this cluster's memory pressure, and each restart is a window where that webhook is
unreachable, so the applies blocked and the controller wedged for **11 hours** — reconciling
zero apps while still reporting `Synced`. Fixed in `oci-k8s` by setting `failurePolicy:
Ignore` on both ClusterPolicies; see the comment there.

**Unstick it now:**

```bash
# 1. Force a re-poll. If the revision advances, polling was the only problem.
kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=hard --overwrite

# 2. If it does not advance within ~60s, the controller is wedged. Restart it —
#    ArgoCD holds no state of its own; everything is in the cluster and in git.
kubectl -n argocd rollout restart statefulset/argocd-application-controller
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=180s
```

**Then check the actual chain, because `Synced` is not the same as `deployed`:**

```bash
git show origin/master:deploy/chart/values.yaml | grep -A2 -E '^\s+(api|worker|reconciler):'
kubectl -n svcforge get pods \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort -u
```

The digests must match. If the sync stalls with a Job stuck `Complete` but never deleted,
it is holding `argocd.argoproj.io/hook-finalizer` — see below.

**Related: the migrate Job deadlock.** A PreSync hook Job that finished but keeps the
finalizer blocks the sync forever:

```bash
kubectl -n svcforge get job svcforge-migrate -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl -n svcforge patch job svcforge-migrate --type=merge -p '{"metadata":{"finalizers":null}}'
```

The Application goes `Synced` within seconds of the patch.

<!-- ci: exercising the runner CPU limit and the webhook end to end, 2026-07-22 -->