runbook: correct the wedge-detection metrics
ci / lint (push) Successful in 2m37s
ci / types (push) Successful in 1m10s
ci / unit (push) Successful in 38s
ci / security (push) Successful in 55s
ci / dockerfile (push) Successful in 18s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 1m25s
ci / image (api) (push) Successful in 4m17s
ci / image (reconciler) (push) Successful in 2m8s
ci / image (worker) (push) Successful in 2m3s
ci / bump (push) Successful in 16s
ci / lint (push) Successful in 2m37s
ci / types (push) Successful in 1m10s
ci / unit (push) Successful in 38s
ci / security (push) Successful in 55s
ci / dockerfile (push) Successful in 18s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 1m25s
ci / image (api) (push) Successful in 4m17s
ci / image (reconciler) (push) Successful in 2m8s
ci / image (worker) (push) Successful in 2m3s
ci / bump (push) Successful in 16s
c691f4f told you to check argocd_redis_request_total and
workqueue_unfinished_work_seconds. Both read healthy through an 82-minute wedge
on 2026-07-22 -- 45 cache reads per 15m and a queue depth of zero, while
argocd_app_reconcile_count sat flat at 62 and three Applications were two
commits behind master.
The cache counter measures that the process is running. The queue gauge reads
zero because this wedge blocks goroutines outside the queue, so nothing is in
flight and nothing is being enqueued.
argocd_app_reconcile_count is the one that went flat, and it is the one to
check. The other two are now in the table of things that look like answers and
are not, with what each read during the outage.
This commit is contained in:
+23
-19
@@ -499,32 +499,36 @@ kubectl -n argocd get application svcforge \
|
||||
git -C ~/workspace/svcforge-reference log --oneline origin/master -1
|
||||
```
|
||||
|
||||
**`reconciledAt` is not a heartbeat — do not read it as one.** ArgoCD writes that field only
|
||||
when the computed status *changes*. On a cluster where everything is Synced and nothing is
|
||||
deploying, it stops advancing and grows arbitrarily old while the controller is perfectly
|
||||
healthy. Measured 2026-07-22: six samples 50s apart against a `timeout.reconciliation` of
|
||||
120s, zero movement, controller fine. It is only evidence of a problem when paired with a
|
||||
`sync.revision` that is *behind `master`* — one stale field is normal, both together are the
|
||||
symptom. (A health check in `oci-k8s` used to threshold on its age alone and warned forever;
|
||||
it now measures the controller instead. Same mistake, don't repeat it here.)
|
||||
**`reconciledAt` alone does not tell you.** ArgoCD writes that field only when the computed
|
||||
status *changes*, so on a cluster where everything is Synced and nothing is deploying it can
|
||||
sit still while the controller is fine. It is evidence only when paired with a
|
||||
`sync.revision` that is *behind `master`* — which is exactly the case you are in if you are
|
||||
reading this section.
|
||||
|
||||
So if the revision is stale, ask whether the controller is doing *anything*:
|
||||
**One metric answers it, and only one.** The controller observes
|
||||
`argocd_app_reconcile_count` once per completed app reconciliation, so with a 2-minute
|
||||
comparison expiry it climbs continuously and forever. Flat means stopped:
|
||||
|
||||
```bash
|
||||
# The controller re-reads its cached resource tree out of Redis on every refresh cycle,
|
||||
# whether or not anything changed. Zero over 15m means it has stopped reconciling.
|
||||
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
wget -qO- --post-data='query=sum(increase(argocd_redis_request_total{job="argocd-application-controller-metrics"}[15m]))' \
|
||||
http://localhost:9090/api/v1/query
|
||||
|
||||
# And whether a reconcile started and never finished — this is what a webhook block looks
|
||||
# like. Healthy is single-digit seconds; the 2026-07-21 wedge would have read ~11 hours.
|
||||
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
wget -qO- --post-data='query=max(workqueue_unfinished_work_seconds{name="app_reconciliation_queue"})' \
|
||||
wget -qO- --post-data='query=sum(increase(argocd_app_reconcile_count[10m]))' \
|
||||
http://localhost:9090/api/v1/query
|
||||
```
|
||||
|
||||
Without Prometheus, fall back to the logs:
|
||||
**Two metrics that look like they answer it and do not.** Both were tried on 2026-07-22
|
||||
against a controller that had been wedged for 82 minutes and had reconciled exactly zero
|
||||
apps, and both read healthy:
|
||||
|
||||
| metric | reading during the wedge | why it lies |
|
||||
|---|---|---|
|
||||
| `argocd_redis_request_total` | 45 reads / 15m, climbing | something in the process still touches the cache when the reconcile loop is dead — it measures "the pod is running", which `up` already covers |
|
||||
| `workqueue_unfinished_work_seconds{name="app_reconciliation_queue"}` | 0 | the wedge blocks goroutines *outside* the queue, so nothing is in flight to count and nothing is being enqueued either |
|
||||
|
||||
Keep the second one anyway — it catches a genuinely stuck queue item, which is a different
|
||||
failure. Just never read a zero from it as "healthy".
|
||||
|
||||
Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22
|
||||
wedge before the metrics were checked:
|
||||
|
||||
```bash
|
||||
# Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.
|
||||
|
||||
Reference in New Issue
Block a user