runbook: ArgoCD synced-but-stale, and how to tell a wedged controller from a slow poll

Cost 11 hours yesterday and looked like nothing was wrong, because the
Application reported Synced the whole time — at a revision five commits
behind. Records the diagnosis that actually works (log-lines-per-hour, and
a flat goroutine count meaning blocked rather than idle), the Kyverno
failurePolicy root cause, and the two escalating fixes. Folds in the
hook-finalizer deadlock, which has now happened four times and was only
written down in chat.

RUNBOOK.md

@@ -482,3 +482,71 @@ remembering.
 
 **Escalate** if size is growing with `tasks` already pruned — that means `instances` is
 growing, i.e. tenants are real, i.e. the free tier is the wrong tier.
+
+---
+
+## Deploy stuck: ArgoCD says Synced at an old commit
+
+**Symptom:** CI is green and the bump commit is on `master`, but the running pods are on the
+previous digest. `kubectl -n argocd get application svcforge` says **`Synced`** — at a
+revision several commits behind. Nothing looks broken, which is what makes this expensive.
+
+**Diagnose.** Compare what ArgoCD thinks it synced against what `master` actually is:
+
+```bash
+kubectl -n argocd get application svcforge \
+  -o jsonpath='{.status.sync.revision}{"  reconciledAt="}{.status.reconciledAt}{"\n"}'
+git -C ~/workspace/svcforge-reference log --oneline origin/master -1
+```
+
+If the revision is stale, ask whether the controller is doing *anything*:
+
+```bash
+# Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.
+kubectl -n argocd logs statefulset/argocd-application-controller --tail=8000 \
+  | grep -oE 'time="[0-9-]+T[0-9]{2}' | sort | uniq -c | tail
+```
+
+A flat `Goroutines=NNN` across hours in those heartbeat lines means blocked goroutines, not
+an idle controller.
+
+**Cause seen here (2026-07-21).** Not ArgoCD config — `timeout.reconciliation` was 120s the
+whole time. The controller's server-side dry-run applies go through the cluster's admission
+webhooks, and Kyverno's mutate webhook was `failurePolicy: Fail`. Kyverno restarts under
+this cluster's memory pressure, and each restart is a window where that webhook is
+unreachable, so the applies blocked and the controller wedged for **11 hours** — reconciling
+zero apps while still reporting `Synced`. Fixed in `oci-k8s` by setting `failurePolicy:
+Ignore` on both ClusterPolicies; see the comment there.
+
+**Unstick it now:**
+
+```bash
+# 1. Force a re-poll. If the revision advances, polling was the only problem.
+kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=hard --overwrite
+
+# 2. If it does not advance within ~60s, the controller is wedged. Restart it —
+#    ArgoCD holds no state of its own; everything is in the cluster and in git.
+kubectl -n argocd rollout restart statefulset/argocd-application-controller
+kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=180s
+```
+
+**Then check the actual chain, because `Synced` is not the same as `deployed`:**
+
+```bash
+git show origin/master:deploy/chart/values.yaml | grep -A2 -E '^\s+(api|worker|reconciler):'
+kubectl -n svcforge get pods \
+  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort -u
+```
+
+The digests must match. If the sync stalls with a Job stuck `Complete` but never deleted,
+it is holding `argocd.argoproj.io/hook-finalizer` — see below.
+
+**Related: the migrate Job deadlock.** A PreSync hook Job that finished but keeps the
+finalizer blocks the sync forever:
+
+```bash
+kubectl -n svcforge get job svcforge-migrate -o jsonpath='{.metadata.finalizers}{"\n"}'
+kubectl -n svcforge patch job svcforge-migrate --type=merge -p '{"metadata":{"finalizers":null}}'
+```
+
+The Application goes `Synced` within seconds of the patch.