runbook: name the ArgoCD wedge instead of describing its symptoms
ci / lint (push) Successful in 37s
ci / security (push) Successful in 1m10s
ci / types (push) Successful in 2m8s
ci / unit (push) Successful in 1m49s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Failing after 50s
ci / integration (push) Successful in 1m12s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped

Reproduced three times on 2026-07-22, and the controller's own workqueue
metrics say what it is:

    workqueue_depth                             0
    workqueue_longest_running_processor_seconds 0
    workqueue_adds_total                        68   <- frozen

Empty queue, idle processors, no new adds. Nothing is blocked -- nothing is
being enqueued. The periodic app-resync timer stops firing, so no Application
is ever queued for refresh again.

Every occurrence followed an API server disruption, with the matching informer
line in the logs: watch ended with error, http2: client connection lost, on
*v1alpha1.Application. The informer re-lists; the resync does not resume.

That is why the controller reads healthy from every angle except reconcile
count, and why only a restart clears it. On a single-control-plane cluster
anything that kills kube-apiserver -- including memory reclaim stalling /livez
past its 80s threshold -- can silently stop all GitOps.
This commit is contained in:
2026-07-22 03:37:22 +00:00
parent 173acc8612
commit 1b56cda231
+35
View File
@@ -527,6 +527,41 @@ apps, and both read healthy:
Keep the second one anyway — it catches a genuinely stuck queue item, which is a different Keep the second one anyway — it catches a genuinely stuck queue item, which is a different
failure. Just never read a zero from it as "healthy". failure. Just never read a zero from it as "healthy".
**What the wedge actually is.** Read the controller's own metrics during one:
```bash
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- http://argocd-application-controller-metrics.argocd.svc:8082/metrics \
| grep -E '^workqueue_(depth|adds_total|longest_running_processor_seconds)\{controller="app_reconciliation_queue"'
```
Measured 2026-07-22 on a controller that had reconciled nothing for 15 minutes:
```
workqueue_depth 0
workqueue_longest_running_processor_seconds 0
workqueue_adds_total 68 <- frozen
```
Empty queue, idle processors, and no new adds. Nothing is *blocked* — nothing is being
**enqueued**. The periodic app-resync timer (`appResyncPeriod=2m0s, appResyncJitter=1m0s`,
logged at controller startup) has stopped firing, so no Application is ever queued for
refresh again.
The trigger is an Application-informer watch break. Every occurrence followed an API server
disruption, and the logs carry the matching line:
```
Warning: watch ended with error ... reflector=...reflector.go:290 type=*v1alpha1.Application
err="unable to decode an event from the watch stream: http2: client connection lost"
```
The informer re-lists, but the resync does not resume. That is why the controller looks
alive from every angle except the one that matters, and why nothing short of a restart
clears it. Anything that kills kube-apiserver — on a single-control-plane cluster, that
includes memory reclaim stalling `/livez` past its 80s threshold — can silently stop all
GitOps until someone notices.
Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22 Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22
wedge before the metrics were checked: wedge before the metrics were checked: