# svcforge runbook Four entries. Each starts from an alert firing and ends at either a fix or an escalation. Every command is copy-pasteable; none of them require thinking at 3am, which is the point. Set these first: ```bash set -a; . ~/.config/svcforge/secrets.env; set +a # SVCFORGE_PG_DSN_SESSION for psql alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"' ``` --- ## Setting up CI/CD from scratch What it takes to get this repo building in Gitea Actions, including every trap that cost real time. Do it in this order; each step fails loudly if the one before it was skipped. ### 1. The repo and its secrets ```bash GITEA=https://gitea.oci-oci.duckdns.org USER=gitea_admin # the registry namespaces packages by OWNER, so this is IMAGE_NS too curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \ -H 'content-type: application/json' \ -d '{"name":"svcforge","default_branch":"master","private":false}' # A PAT with exactly three scopes. Not admin. curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \ -H 'content-type: application/json' \ -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}' ``` Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API): | Secret | Value | Why it exists | |---|---|---| | `REGISTRY_USER` | `gitea_admin` | | | `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. | | `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. It is deliberately not a kubeconfig, so CI's maximum blast radius is a bad commit. | ### 2. The runner needs a cache server, and it fails SOFT without one `cache: enabled: false` in the act_runner config is the default, and it does not fail the build. It prints: ``` Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache. ``` …and carries on, re-downloading every wheel on every run, forever. **A cache that is off looks exactly like a cache that is always cold.** Enable it in `oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it): ```yaml cache: enabled: true dir: /data/cache # the runner's PVC, so it survives a restart host: "" # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container port: 8088 ``` ```bash cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea ``` Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to type=gha` are both no-ops. ### 3. Traps that are specific to Gitea, not GitHub | Symptom | Cause | Fix | |---|---|---| | `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. | | `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. | | The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. | | Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. | ### 4. Keeping trivy green The image gate is a moving target: trivy's vulnerability DB updates daily, so an image that passed yesterday fails today without a single line of code changing. Two rules keep it sane. **Bump the version, do not add an ignore.** The worker image went 39 findings (2 CRITICAL) → 18 → 5 → 0 across three fixes, each a version bump or a removal: | Change | Result | |---|---| | `alpine/helm` 3.16.2 → 3.21.3, `kubectl` 1.31.2 → 1.35.3 | 39 → 18, both CRITICALs cleared | | `kubectl` 1.35.3 → 1.36.2 (k8s 1.35.x vendors spdystream 0.5.0; the fix is 0.5.1) | 18 → 5 | | dropped kubectl entirely — `helm --create-namespace` replaced `kubectl apply` | **5 → 0** | **The cheapest CVE is the binary you do not ship.** The last five findings lived in kubectl's vendored `golang.org/x/net` and Go stdlib, inside the newest kubectl that exists — no version cleared them. kubectl was in that image for exactly one call, and helm already does the same thing with a flag. Removing it removed the CVEs, a binary, and an adapter. ### 5. When every job fails at `Set up job` after ~14 minutes Symptom: `Set up job` runs for 10–15 minutes and succeeds, then the first real step fails instantly at 0s, and every downstream job is skipped. It looks like the action broke. Cause: the dind sidecar's image cache is empty, so each run re-pulls the ~1.6GB act job image (`ghcr.io/catthehacker/ubuntu:act-24.04`) before it can start. dind has no volume for `/var/lib/docker` — the cache lives in its container writable layer and is destroyed on every pod restart. A runner that is restart-looping therefore never keeps a cache, and each job pays the full pull. Check it: ```bash kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker images ``` Empty output is the diagnosis. Warm it once by hand: ```bash kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \ ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9 ``` The durable fix is a persistent image store, which the runner now has: `/var/lib/docker` is a hostPath on node0 (see `oci-k8s/.../addons/tasks/main.yml`), so the act image survives a restart and is not re-pulled. The runner is pinned to **node0**, not node2 — node2 is a single-core control-plane node whose pod network was measured 21x slower under its own load, which starved every clone and pull. Its `/data` PVC is NFS ReadWriteMany, so a reschedule attaches immediately with no `Multi-Attach` wait. Entries 9 and 10 cover the caches and the node move in full. ### 6. Stopping a run, and reading a restarted runner correctly Restarting the act_runner StatefulSet does not reliably orphan its in-flight jobs. Both outcomes have been observed on this cluster: - run #14 was left with two jobs `in_progress` and nothing behind them, still stuck at 15m45s, so `ZOMBIE_TASK_TIMEOUT` is not a rescue you can wait for - run #16 had its remaining jobs re-dispatched to the new pod and carried on normally So `in_progress` after a restart is ambiguous on its own. Check whether the runner is actually working before concluding anything, or you will diagnose a healthy run as a zombie: ```bash kubectl -n gitea logs gitea-actions-runner-0 -c runner --tail=4 ``` Recent `NewParallelExecutor` lines mean it is executing, not stuck. At `capacity: 1` a later run sitting in `waiting` behind a live one is correct, not a block. **Gitea 1.26 has no cancel endpoint at all** — the full swagger contains no path matching `cancel`. The red "Cancel workflow run" button in the web UI is a CSRF-protected web route, so an API token cannot drive it, and cancelling from a script is simply not available. `DELETE .../actions/runs/{index}` exists and takes the **run index** (`14`), not the database id (`51`). It cleared the stuck #14, but returned 204 against the live #16 without stopping it and then 404 on retry. Treat it as a way to remove a finished run, not a cancel. To stop a running job: click Cancel in the UI. ### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough `gitea-postgresql-0` CrashLoopBackOff with: mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error and the Gitea API returning 500, so every CI run dies at checkout with `Failed to connect to gitea-http:3000`. The "Initializing PostgreSQL database" line above that error is alarming and is not what it looks like: the data is fine, the mount is broken, so the container sees an empty directory. The documented recovery — scale to 0, wait for `detached`, scale back to 1 — was **not enough** here. The volume came back `detached/faulted` and simply refused to attach, so the pod sat in ContainerCreating. `auto-salvage: true` does not help: salvage happens during attach, and a faulted volume never gets that far, so it cannot rescue itself. What the volume was actually saying: ```bash V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}') kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}' # detached/faulted kubectl -n longhorn-system get replicas.longhorn.io -o json \ | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv' ``` The replica carries a `failedAt` timestamp, and that alone is what keeps the volume faulted. Clearing it is the salvage: ```bash kubectl -n longhorn-system patch replicas.longhorn.io --type merge \ -p '{"spec":{"failedAt":"","lastFailedAt":""}}' ``` The volume went `attached/healthy` and postgres reached 1/1 within 40 seconds, with the repo, its size and the whole CI run history intact. **Check the backups before patching anything**, because this cluster runs Longhorn at one replica — there is no second copy to fall back on, only the nightly backup: ```bash kubectl -n longhorn-system get backups.longhorn.io -o json \ | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv' ``` Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash recovery on start. If it cannot, restore the most recent Completed backup instead. ### 8. OPEN: dind is killed by its own liveness probe Unresolved as of 2026-07-20. Recorded because it probably explains build failures that were diagnosed as something else. The runner sits at `Init:1/2` and its dind sidecar accumulates restarts: ``` Liveness probe failed: command timed out: "/usr/bin/test -S /var/run/docker.sock" timed out after 1s (x27 over 156m) Killing: Init container dind failed liveness probe ``` The probe is hardcoded at `timeoutSeconds: 1`, `failureThreshold: 3`, `periodSeconds: 10`. `test -S` only asks whether a socket exists. When that cannot finish inside a second, the node is starved rather than dind being unhealthy, and kubelet kills a working daemon. **Why this matters beyond the runner restarting.** Image builds failed with: ERROR: failed to solve: DeadlineExceeded: no active session for which was attributed to CPU starvation alone and addressed by dropping the runner's `capacity` to 1. Starvation is real, but the mechanism is more likely that kubelet killed dind mid-build and the buildkit session died with it. Lowering capacity reduced the load that trips the probe, which is consistent with run #17 passing — it treated the cause of the trigger, not the trigger. Runs #18-#21 then failed anyway. Treat this as a strong hypothesis, not a settled one. Confirming it means correlating the kill timestamps against the failed builds: ```bash kubectl -n gitea describe pod gitea-actions-runner-0 | grep -A10 Events: kubectl -n gitea get pod gitea-actions-runner-0 \ -o jsonpath='{.status.initContainerStatuses[?(@.name=="dind")].lastState.terminated}' ``` **The chart exposes no probe knobs** — `helm show values gitea-charts/actions` has no match for `probe`. So this cannot be fixed the way `capacity` was, and a `kubectl patch` is reverted by the next Ansible run. Same shape as the longhorn-csi-plugin probe problem. The candidate fix is a Kyverno mutating policy authored in Ansible, relaxing the probe to roughly `timeoutSeconds: 5` and `failureThreshold: 6`. This cluster already mutates workloads that way — see `force-best-effort-cpu`, which rewrites every CPU request to 0 — so the precedent and the tooling are both in place. ### 9. The runner's three ephemeral caches, and the boot cascade they cause Three separate caches on this runner were container-layer only, each found the same way — something was slow, and the cause was a cache that had never survived a restart: | cache | path | fixed by | |---|---|---| | dind image store | `/var/lib/docker` | hostPath `/var/lib/gitea-dind` | | trivy vulnerability DB | `/root/.cache/trivy` in dind | named docker volume | | act's action clones | `/root/.cache/act` | hostPath `/var/lib/gitea-act-cache` | act clones each action with **full history**, not shallow: 66.7MB/538 commits for `astral-sh/setup-uv`, 24.4MB/222 commits for `actions/checkout`, and this workflow uses five. After a restart that made `Set up job` an 11-minute step in which the job container sat idle running `sleep` while the runner cloned GitHub. If a job looks hung in setup, check the job container before blaming the network: ```bash C=$(kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker ps -q | head -1) kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker exec "$C" ps -eo pid,etime,comm ``` Only `sleep` means the work is in the runner, not the job. **The cascade this creates.** Giving dind a persistent image store made it boot slowly, because dockerd scans that store on startup — measured at 38s, 2m13s, and over 5 minutes depending on node load. Two independent timeouts then fire: 1. dind's startup probe. Widened to 5 minutes by the Kyverno policy in oci-k8s, and a cold boot has still exceeded it. 2. **The runner container's own `Docker wait timeout of 5m0s`**, which is internal to the runner image and not configurable from the chart. When dind is late, the runner exits 1 and restarts — and that restart kills whatever job was running, which surfaces as every step in the job failing at once with no error in the log, right after a green `Set up job`. The pair self-heals: the second dind boot is fast because the store is warm, and the runner comes up behind it. The cost is roughly ten minutes of thrash after any runner restart, and one lost CI run. Restart the runner deliberately, not casually. ### 10. Verify the whole loop, not just the green checkmarks ```bash # the digest CI pushed docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api: \ --format '{{.Manifest.Digest}}' # the digest the chart deploys — these must be equal grep -A2 'api:' deploy/chart/values.yaml # what ArgoCD actually synced kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}' ``` If those three disagree, the deploy is not what CI tested, and every other guarantee in this document is void. --- ## Measured numbers From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks per run. **These came off local Postgres on the same box, with a sub-millisecond round trip. Supabase's pooler is ~5ms away, so treat these as a ceiling the real thing will not reach** — the shape is what transfers, not the absolute numbers. | replicas | pool max_size | worker concurrency | connections used | drain (200 tasks) | throughput | |---:|---:|---:|---:|---:|---:| | 1 | 5 | 4 | 5 | 4.3s | 46.9 task/s | | 2 | 5 | 4 | 10 | 4.2s | 47.6 task/s | | 4 | 5 | 4 | 20 | 2.2s | 92.3 task/s | | 2 | 20 | 16 | 40 | 2.1s | 97.1 task/s | Three things this says: 1. **Going from 1 replica to 2 bought nothing** (46.9 → 47.6). Throughput here is bounded by per-worker concurrency (the semaphore), not by replica count. Adding pods to a saturated semaphore is the most common wrong fix for a slow queue. 2. **Concurrency is the knob that moved it** — 4 replicas (20 connections) and 2 replicas at concurrency 16 (40 connections) land in the same place, ~92-97 task/s. The second buys the same throughput for twice the connections, which on the free tier is the worse trade. 3. **The budget is `(api + worker replicas) x max_size`**, and it is spent whether or not the connections are busy. The bottom row costs 40 connections for a 2% gain over the row above it. On Supabase free tier, that arithmetic — not throughput — is what decides replica count. Nothing failed at any setting, so the real connection ceiling was never hit locally. Finding it against the actual pooler is the experiment worth running: raise `replicas x max_size` until claims slow and `PoolTimeout` appears, and write the number here. --- ## Queue stuck **Alert:** `SvcforgeQueueDepthRising` **Diagnose.** Start here, always: ```bash sfsql -c "select state, count(*) from tasks group by 1;" ``` Then split the three causes apart — they look identical from the alert and need opposite fixes: ```bash # Stuck leases: rows 'running' with a locked_at that never advances. sfsql -c "select kind, locked_by, locked_at, last_error from tasks where state='running' order by locked_at limit 10;" # No workers: is anything actually consuming? kubectl get pods -l app=worker -o wide kubectl logs -l app=worker --tail=20 --prefix # run_after in the future: backoff has parked everything. sfsql -c "select count(*) from tasks where state='queued' and run_after > now();" ``` | What you see | Cause | Fix | |---|---|---| | `running` rows, `locked_at` older than 5m, no worker pods hold those IDs | Workers died mid-task | None. The reconciler resets expired leases within 60s. If it does not, the reconciler is down — check it. | | Zero worker pods, or all `CrashLoopBackOff` | No consumer | Fix the workers. `kubectl describe pod -l app=worker`. | | Everything `queued` with `run_after` far in the future | Backoff, i.e. tasks are failing and retrying | This is not a queue problem. Go to **Provision failing**. | | `queued` rows with `run_after <= now()` and healthy workers | Real: claim is not returning rows | Check pooler connection budget (see **Supabase full**). | **Never** hand-edit `state='running'` back to `'queued'`. The lease does that, and doing it by hand while the worker is actually alive gives you two workers on one task — the exact thing the whole design prevents. **Escalate** if workers are healthy, leases are fresh, and depth still grows: that is a claim-query or pooler bug, not an ops problem. --- ## Provision failing **Alert:** `SvcforgeTaskFailed` (tasks reaching the dead-letter state), or `SvcforgeProvisionSlow` (they still succeed, but the p95 has drifted out — usually cluster capacity, diagnosed the same way). **Diagnose:** ```bash sfsql -c "select id, service_type, chart_version, error from instances where state='failed';" sfsql -c "select id, kind, attempts, last_error from tasks where state='failed' order by id desc limit 10;" NS=tenant- helm list -n "$NS" kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20 ``` | `error` looks like | Cause | Fix | |---|---|---| | `chart "..." version "..." not found` | Bad pin in `catalog.yaml` | Correct the version, commit. The next `upgrade`/`provision` picks it up. | | `timed out waiting for the condition` | Cluster capacity — the chart installed but pods never became ready | `kubectl describe pod -n $NS`. Usually `Insufficient cpu/memory` or a PVC pending on Longhorn. | | `Error: ... forbidden: User "system:serviceaccount:svcforge:..."` | RBAC | The worker's ClusterRole is missing a verb. Chart change, not a manual `kubectl edit`. | | `ImagePullBackOff` in events | Registry auth or a gated image | Prefer `bitnamilegacy/*` images, which pull anonymously. | After fixing the cause, tasks that already dead-lettered do **not** retry themselves. Requeue deliberately: ```bash sfsql -c "update tasks set state='queued', attempts=0, run_after=now(), last_error=null where id = ;" ``` **Escalate** if `error` is empty on a failed instance — that means the failure path itself lost the message. --- ## Orphaned release **Alert:** `SvcforgeReconcilerStale` — the reconciler has not completed a loop recently, so drift is no longer being *detected* at all. Drift itself is reported in the reconciler's logs and metrics rather than paged on, because it is usually benign and always needs a human to judge. A stale reconciler is the real emergency: nothing is watching. The control loop **never auto-deletes a release.** That is deliberate: a bug in the drift check that deletes things is unrecoverable, and one that only reports is a Tuesday. **Diagnose:** ```bash helm list -A -o json | jq -r '.[].name' | sort > /tmp/real sfsql -tAc "select release_name from instances where state in ('ready','provisioning');" | sort > /tmp/want comm -23 /tmp/real /tmp/want # in the cluster, not in the DB -> orphan comm -13 /tmp/real /tmp/want # in the DB, not in the cluster -> missing ``` | Direction | Meaning | Action | |---|---|---| | Orphan (cluster only) | A deprovision half-finished, or someone ran `helm install` by hand | Confirm the tenant is gone, then `helm uninstall -n ` **by hand**, and write down that you did. | | Missing (DB only) | Someone deleted a release out from under us | Requeue a `provision` task for that instance. It is idempotent; it will rebuild. | **Escalate** before uninstalling anything you did not personally trace to a deleted instance. A wrong `helm uninstall` here deletes a tenant's data. --- ## Supabase full **Alert:** none — and that is a gap, not a decision. The free tier is 0.5 GB and nothing pages you before you hit it; you find out when writes start failing. Until someone adds a size rule, this entry is driven by the calendar, not by an alert. Check it monthly: ```bash sfsql -c "select pg_size_pretty(pg_database_size(current_database()));" ``` **Diagnose:** ```bash sfsql -c "select pg_size_pretty(pg_database_size(current_database()));" sfsql -c "select relname, pg_size_pretty(pg_total_relation_size(relid)) from pg_catalog.pg_statio_user_tables order by pg_total_relation_size(relid) desc limit 5;" sfsql -c "select count(*) from pg_stat_activity;" ``` It is almost always `tasks`. Every provision, upgrade and verify leaves a row forever. ```bash sfsql -c "delete from tasks where state='done' and created_at < now() - interval '7 days';" sfsql -c "vacuum (analyze) tasks;" ``` `vacuum` alone reclaims space **for reuse by Postgres**, but does not return it to the filesystem — so `pg_database_size` may barely move. That is expected and fine; the space is free for new rows. `vacuum full` does return it, takes an `ACCESS EXCLUSIVE` lock, and will stall every worker for its duration. Only do it in a window, and only if you actually need the bytes back. If `count(*) from pg_stat_activity` is near the pooler's ceiling, the cause is arithmetic, not load: `worker_replicas × pool_max_size + api_replicas × pool_max_size`. Lower `max_size` or replicas. **Replica count is a database-capacity decision here**, which is unusual and worth remembering. **Escalate** if size is growing with `tasks` already pruned — that means `instances` is growing, i.e. tenants are real, i.e. the free tier is the wrong tier. --- ## Deploy stuck: ArgoCD says Synced at an old commit **Symptom:** CI is green and the bump commit is on `master`, but the running pods are on the previous digest. `kubectl -n argocd get application svcforge` says **`Synced`** — at a revision several commits behind. Nothing looks broken, which is what makes this expensive. **Diagnose.** Compare what ArgoCD thinks it synced against what `master` actually is: ```bash kubectl -n argocd get application svcforge \ -o jsonpath='{.status.sync.revision}{" reconciledAt="}{.status.reconciledAt}{"\n"}' git -C ~/workspace/svcforge-reference log --oneline origin/master -1 ``` **`reconciledAt` alone does not tell you.** ArgoCD writes that field only when the computed status *changes*, so on a cluster where everything is Synced and nothing is deploying it can sit still while the controller is fine. It is evidence only when paired with a `sync.revision` that is *behind `master`* — which is exactly the case you are in if you are reading this section. **One metric answers it, and only one.** The controller observes `argocd_app_reconcile_count` once per completed app reconciliation. Flat means it is doing nothing: ```bash kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ wget -qO- --post-data='query=sum(increase(argocd_app_reconcile_count[10m]))' \ http://localhost:9090/api/v1/query ``` **Two metrics that look like they answer it and do not.** Both were tried on 2026-07-22 against a controller that had reconciled nothing for 82 minutes, and both read healthy: | metric | reading at the time | why it lies | |---|---|---| | `argocd_redis_request_total` | 45 reads / 15m, climbing | something in the process still touches the cache when nothing is reconciling — it measures "the pod is running", which `up` already covers | | `workqueue_unfinished_work_seconds{name="app_reconciliation_queue"}` | 0 | it only counts work already *in* the queue, and the queue is empty. Nothing is stuck; nothing is being enqueued | Keep the second one anyway — it catches a genuinely stuck queue item, which is a different failure. Just never read a zero from it as "healthy". **What is actually wrong: the periodic git poll does not run.** Read the controller's own metrics on an idle cluster: ```bash kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \ | grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"' ``` Measured 2026-07-22, across two separate controller pods, on a fully healthy API server: ``` 03:37:03 controller starts, refreshes all 3 apps adds_total = 3 03:38:31 adds_total = 3 03:41:33 adds_total = 3 03:44:34 adds_total = 3 03:47:35 adds_total = 3 <- expiry is 2m0s, jitter 60s ``` `workqueue_depth 0`, `longest_running_processor_seconds 0`, `adds_total` frozen. Nothing is *blocked* — nothing is being **enqueued**. The controller logs its own schedule at startup (`appResyncPeriod=2m0s, appResyncJitter=1m0s`) and `argocd-cm` carries the matching `timeout.reconciliation: 120s`, so the setting is read and then never acted on. Refreshes still happen from two other paths, which is what makes this so easy to misread as working: | path | fires when | observed | |---|---|---| | startup | controller (re)starts | 3 apps refreshed within ~2s of ready | | cluster events | a watched resource changes | 68 adds during one svcforge rollout, then flat the moment the cluster went quiet | | periodic poll | every 2m ± 60s | **never** | **The consequence is the thing to take away: a commit that changes only the repo is never noticed.** Every "auto-sync" observed on 2026-07-22 happened within seconds of a controller restart, i.e. it was the startup refresh, not the poll. Do not read a successful deploy straight after a restart as evidence that polling works. **A webhook now covers for it** (added 2026-07-22, `oci-k8s` `--tags argocd,gitea`). Gitea POSTs every push to `https://argocd.oci-oci.duckdns.org/api/webhook`, so a commit refreshes ArgoCD in under a second instead of waiting for a poll that never comes. It is registered as Gitea's **`gogs`** type, which looks wrong and is not: ArgoCD's webhook handler dispatches on the `X-Gogs-Event` header and ships no Gitea parser. Gitea forked from Gogs and still emits that wire format on request. The shared secret lives in `argocd-secret` as `webhook.gogs.secret` and in Ansible as `argocd_webhook_secret`; both sides must match or every delivery fails signature validation *silently*, which looks identical to having no webhook at all. Check a delivery when a push does not deploy: ```bash # Gitea's own record of the last attempt, including the response ArgoCD gave curl -s -u "$USER:$PASS" \ https://gitea.oci-oci.duckdns.org/api/v1/repos/gitea_admin/svcforge/hooks | jq '.[].id' # ArgoCD's side kubectl -n argocd logs deploy/argocd-server --tail=200 | grep -i webhook ``` `Unknown webhook event` means the hook type is wrong (it must be `gogs`). A 400 on signature means the secrets have drifted — re-run `03_install_addons.yml --tags argocd,gitea`, which rewrites both ends from the same variable. Manual nudge, still valid if the webhook is ever down: ```bash kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=normal --overwrite ``` Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22 wedge before the metrics were checked: ```bash # Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat. kubectl -n argocd logs statefulset/argocd-application-controller --tail=8000 \ | grep -oE 'time="[0-9-]+T[0-9]{2}' | sort | uniq -c | tail ``` A flat `Goroutines=NNN` across hours in those heartbeat lines means blocked goroutines, not an idle controller. **Cause seen here (2026-07-21).** Not ArgoCD config — `timeout.reconciliation` was 120s the whole time. The controller's server-side dry-run applies go through the cluster's admission webhooks, and Kyverno's mutate webhook was `failurePolicy: Fail`. Kyverno restarts under this cluster's memory pressure, and each restart is a window where that webhook is unreachable, so the applies blocked and the controller wedged for **11 hours** — reconciling zero apps while still reporting `Synced`. Fixed in `oci-k8s` by setting `failurePolicy: Ignore` on both ClusterPolicies; see the comment there. **Unstick it now:** ```bash # 1. Force a re-poll. If the revision advances, polling was the only problem. kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=hard --overwrite # 2. If it does not advance within ~60s, the controller is wedged. Restart it — # ArgoCD holds no state of its own; everything is in the cluster and in git. kubectl -n argocd rollout restart statefulset/argocd-application-controller kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=180s ``` **Then check the actual chain, because `Synced` is not the same as `deployed`:** ```bash git show origin/master:deploy/chart/values.yaml | grep -A2 -E '^\s+(api|worker|reconciler):' kubectl -n svcforge get pods \ -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort -u ``` The digests must match. If the sync stalls with a Job stuck `Complete` but never deleted, it is holding `argocd.argoproj.io/hook-finalizer` — see below. **Related: the migrate Job deadlock.** A PreSync hook Job that finished but keeps the finalizer blocks the sync forever: ```bash kubectl -n svcforge get job svcforge-migrate -o jsonpath='{.metadata.finalizers}{"\n"}' kubectl -n svcforge patch job svcforge-migrate --type=merge -p '{"metadata":{"finalizers":null}}' ``` The Application goes `Synced` within seconds of the patch.