diff --git a/RUNBOOK.md b/RUNBOOK.md index 61e15ec..0f44c8d 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -527,6 +527,41 @@ apps, and both read healthy: Keep the second one anyway — it catches a genuinely stuck queue item, which is a different failure. Just never read a zero from it as "healthy". +**What the wedge actually is.** Read the controller's own metrics during one: + +```bash +kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \ + | grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"' +``` + +Measured 2026-07-22 on a controller that had reconciled nothing for 15 minutes: + +``` +workqueue_depth 0 +workqueue_longest_running_processor_seconds 0 +workqueue_adds_total 68 <- frozen +``` + +Empty queue, idle processors, and no new adds. Nothing is *blocked* — nothing is being +**enqueued**. The periodic app-resync timer (`appResyncPeriod=2m0s, appResyncJitter=1m0s`, +logged at controller startup) has stopped firing, so no Application is ever queued for +refresh again. + +The trigger is an Application-informer watch break. Every occurrence followed an API server +disruption, and the logs carry the matching line: + +``` +Warning: watch ended with error ... reflector=...reflector.go:290 type=*v1alpha1.Application + err="unable to decode an event from the watch stream: http2: client connection lost" +``` + +The informer re-lists, but the resync does not resume. That is why the controller looks +alive from every angle except the one that matters, and why nothing short of a restart +clears it. Anything that kills kube-apiserver — on a single-control-plane cluster, that +includes memory reclaim stalling `/livez` past its 80s threshold — can silently stop all +GitOps until someone notices. + Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22 wedge before the metrics were checked: