runbook: name the ArgoCD wedge instead of describing its symptoms

Reproduced three times on 2026-07-22, and the controller's own workqueue metrics say what it is: workqueue_depth 0 workqueue_longest_running_processor_seconds 0 workqueue_adds_total 68 <- frozen Empty queue, idle processors, no new adds. Nothing is blocked -- nothing is being enqueued. The periodic app-resync timer stops firing, so no Application is ever queued for refresh again. Every occurrence followed an API server disruption, with the matching informer line in the logs: watch ended with error, http2: client connection lost, on *v1alpha1.Application. The informer re-lists; the resync does not resume. That is why the controller reads healthy from every angle except reconcile count, and why only a restart clears it. On a single-control-plane cluster anything that kills kube-apiserver -- including memory reclaim stalling /livez past its 80s threshold -- can silently stop all GitOps.
2026-07-22 03:37:22 +00:00
parent 173acc8612
commit 1b56cda231
1 changed files with 35 additions and 0 deletions
@@ -527,6 +527,41 @@ apps, and both read healthy:
 Keep the second one anyway — it catches a genuinely stuck queue item, which is a different
 failure. Just never read a zero from it as "healthy".

+**What the wedge actually is.** Read the controller's own metrics during one:
+
+```bash
+kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+  wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \
+  | grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"'
+```
+
+Measured 2026-07-22 on a controller that had reconciled nothing for 15 minutes:
+
+```
+workqueue_depth                          0
+workqueue_longest_running_processor_seconds  0
+workqueue_adds_total                     68   <- frozen
+```
+
+Empty queue, idle processors, and no new adds. Nothing is *blocked* — nothing is being
+**enqueued**. The periodic app-resync timer (`appResyncPeriod=2m0s, appResyncJitter=1m0s`,
+logged at controller startup) has stopped firing, so no Application is ever queued for
+refresh again.
+
+The trigger is an Application-informer watch break. Every occurrence followed an API server
+disruption, and the logs carry the matching line:
+
+```
+Warning: watch ended with error ... reflector=...reflector.go:290 type=*v1alpha1.Application
+  err="unable to decode an event from the watch stream: http2: client connection lost"
+```
+
+The informer re-lists, but the resync does not resume. That is why the controller looks
+alive from every angle except the one that matters, and why nothing short of a restart
+clears it. Anything that kills kube-apiserver — on a single-control-plane cluster, that
+includes memory reclaim stalling `/livez` past its 80s threshold — can silently stop all
+GitOps until someone notices.
+
 Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22
 wedge before the metrics were checked: