diff --git a/RUNBOOK.md b/RUNBOOK.md index e930d1a..61e15ec 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -499,32 +499,36 @@ kubectl -n argocd get application svcforge \ git -C ~/workspace/svcforge-reference log --oneline origin/master -1 ``` -**`reconciledAt` is not a heartbeat — do not read it as one.** ArgoCD writes that field only -when the computed status *changes*. On a cluster where everything is Synced and nothing is -deploying, it stops advancing and grows arbitrarily old while the controller is perfectly -healthy. Measured 2026-07-22: six samples 50s apart against a `timeout.reconciliation` of -120s, zero movement, controller fine. It is only evidence of a problem when paired with a -`sync.revision` that is *behind `master`* — one stale field is normal, both together are the -symptom. (A health check in `oci-k8s` used to threshold on its age alone and warned forever; -it now measures the controller instead. Same mistake, don't repeat it here.) +**`reconciledAt` alone does not tell you.** ArgoCD writes that field only when the computed +status *changes*, so on a cluster where everything is Synced and nothing is deploying it can +sit still while the controller is fine. It is evidence only when paired with a +`sync.revision` that is *behind `master`* — which is exactly the case you are in if you are +reading this section. -So if the revision is stale, ask whether the controller is doing *anything*: +**One metric answers it, and only one.** The controller observes +`argocd_app_reconcile_count` once per completed app reconciliation, so with a 2-minute +comparison expiry it climbs continuously and forever. Flat means stopped: ```bash -# The controller re-reads its cached resource tree out of Redis on every refresh cycle, -# whether or not anything changed. Zero over 15m means it has stopped reconciling. kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ - wget -qO- --post-data='query=sum(increase(argocd_redis_request_total{job="argocd-application-controller-metrics"}[15m]))' \ - http://localhost:9090/api/v1/query - -# And whether a reconcile started and never finished — this is what a webhook block looks -# like. Healthy is single-digit seconds; the 2026-07-21 wedge would have read ~11 hours. -kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ - wget -qO- --post-data='query=max(workqueue_unfinished_work_seconds{name="app_reconciliation_queue"})' \ + wget -qO- --post-data='query=sum(increase(argocd_app_reconcile_count[10m]))' \ http://localhost:9090/api/v1/query ``` -Without Prometheus, fall back to the logs: +**Two metrics that look like they answer it and do not.** Both were tried on 2026-07-22 +against a controller that had been wedged for 82 minutes and had reconciled exactly zero +apps, and both read healthy: + +| metric | reading during the wedge | why it lies | +|---|---|---| +| `argocd_redis_request_total` | 45 reads / 15m, climbing | something in the process still touches the cache when the reconcile loop is dead — it measures "the pod is running", which `up` already covers | +| `workqueue_unfinished_work_seconds{name="app_reconciliation_queue"}` | 0 | the wedge blocks goroutines *outside* the queue, so nothing is in flight to count and nothing is being enqueued either | + +Keep the second one anyway — it catches a genuinely stuck queue item, which is a different +failure. Just never read a zero from it as "healthy". + +Without Prometheus, fall back to the logs — this works and is what found the 2026-07-22 +wedge before the metrics were checked: ```bash # Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.