diff --git a/RUNBOOK.md b/RUNBOOK.md index c2d6a9f..1aad2e2 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -482,3 +482,71 @@ remembering. **Escalate** if size is growing with `tasks` already pruned — that means `instances` is growing, i.e. tenants are real, i.e. the free tier is the wrong tier. + +--- + +## Deploy stuck: ArgoCD says Synced at an old commit + +**Symptom:** CI is green and the bump commit is on `master`, but the running pods are on the +previous digest. `kubectl -n argocd get application svcforge` says **`Synced`** — at a +revision several commits behind. Nothing looks broken, which is what makes this expensive. + +**Diagnose.** Compare what ArgoCD thinks it synced against what `master` actually is: + +```bash +kubectl -n argocd get application svcforge \ + -o jsonpath='{.status.sync.revision}{" reconciledAt="}{.status.reconciledAt}{"\n"}' +git -C ~/workspace/svcforge-reference log --oneline origin/master -1 +``` + +If the revision is stale, ask whether the controller is doing *anything*: + +```bash +# Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat. +kubectl -n argocd logs statefulset/argocd-application-controller --tail=8000 \ + | grep -oE 'time="[0-9-]+T[0-9]{2}' | sort | uniq -c | tail +``` + +A flat `Goroutines=NNN` across hours in those heartbeat lines means blocked goroutines, not +an idle controller. + +**Cause seen here (2026-07-21).** Not ArgoCD config — `timeout.reconciliation` was 120s the +whole time. The controller's server-side dry-run applies go through the cluster's admission +webhooks, and Kyverno's mutate webhook was `failurePolicy: Fail`. Kyverno restarts under +this cluster's memory pressure, and each restart is a window where that webhook is +unreachable, so the applies blocked and the controller wedged for **11 hours** — reconciling +zero apps while still reporting `Synced`. Fixed in `oci-k8s` by setting `failurePolicy: +Ignore` on both ClusterPolicies; see the comment there. + +**Unstick it now:** + +```bash +# 1. Force a re-poll. If the revision advances, polling was the only problem. +kubectl -n argocd annotate application svcforge argocd.argoproj.io/refresh=hard --overwrite + +# 2. If it does not advance within ~60s, the controller is wedged. Restart it — +# ArgoCD holds no state of its own; everything is in the cluster and in git. +kubectl -n argocd rollout restart statefulset/argocd-application-controller +kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=180s +``` + +**Then check the actual chain, because `Synced` is not the same as `deployed`:** + +```bash +git show origin/master:deploy/chart/values.yaml | grep -A2 -E '^\s+(api|worker|reconciler):' +kubectl -n svcforge get pods \ + -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort -u +``` + +The digests must match. If the sync stalls with a Job stuck `Complete` but never deleted, +it is holding `argocd.argoproj.io/hook-finalizer` — see below. + +**Related: the migrate Job deadlock.** A PreSync hook Job that finished but keeps the +finalizer blocks the sync forever: + +```bash +kubectl -n svcforge get job svcforge-migrate -o jsonpath='{.metadata.finalizers}{"\n"}' +kubectl -n svcforge patch job svcforge-migrate --type=merge -p '{"metadata":{"finalizers":null}}' +``` + +The Application goes `Synced` within seconds of the patch.