From 1b56cda23194a1331bd07561c0e85592b88fff53 Mon Sep 17 00:00:00 2001 From: Nguyen Minh Phuc Date: Wed, 22 Jul 2026 03:37:22 +0000 Subject: [PATCH] runbook: name the ArgoCD wedge instead of describing its symptoms Reproduced three times on 2026-07-22, and the controller's own workqueue metrics say what it is: workqueue_depth 0 workqueue_longest_running_processor_seconds 0 workqueue_adds_total 68 <- frozen Empty queue, idle processors, no new adds. Nothing is blocked -- nothing is being enqueued. The periodic app-resync timer stops firing, so no Application is ever queued for refresh again. Every occurrence followed an API server disruption, with the matching informer line in the logs: watch ended with error, http2: client connection lost, on *v1alpha1.Application. The informer re-lists; the resync does not resume. That is why the controller reads healthy from every angle except reconcile count, and why only a restart clears it. On a single-control-plane cluster anything that kills kube-apiserver -- including memory reclaim stalling /livez past its 80s threshold -- can silently stop all GitOps. --- RUNBOOK.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/RUNBOOK.md b/RUNBOOK.md index 61e15ec..0f44c8d 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -527,6 +527,41 @@ apps, and both read healthy: Keep the second one anyway — it catches a genuinely stuck queue item, which is a different failure. Just never read a zero from it as "healthy". +**What the wedge actually is.** Read the controller's own metrics during one: + +```bash +kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \ + | grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"' +``` + +Measured 2026-07-22 on a controller that had reconciled nothing for 15 minutes: + +``` +workqueue_depth 0 +workqueue_longest_running_processor_seconds 0 +workqueue_adds_total 68 <- frozen +``` + +Empty queue, idle processors, and no new adds. Nothing is *blocked* — nothing is being +**enqueued**. The periodic app-resync timer (`appResyncPeriod=2m0s, appResyncJitter=1m0s`, +logged at controller startup) has stopped firing, so no Application is ever queued for +refresh again. + +The trigger is an Application-informer watch break. Every occurrence followed an API server +disruption, and the logs carry the matching line: + +``` +Warning: watch ended with error ... reflector=...reflector.go:290 type=*v1alpha1.Application + err="unable to decode an event from the watch stream: http2: client connection lost" +``` + +The informer re-lists, but the resync does not resume. That is why the controller looks +alive from every angle except the one that matters, and why nothing short of a restart +clears it. Anything that kills kube-apiserver — on a single-control-plane cluster, that +includes memory reclaim stalling `/livez` past its 80s threshold — can silently stop all +GitOps until someone notices. + Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22 wedge before the metrics were checked: