diff --git a/RUNBOOK.md b/RUNBOOK.md index 1aad2e2..e930d1a 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -499,7 +499,32 @@ kubectl -n argocd get application svcforge \ git -C ~/workspace/svcforge-reference log --oneline origin/master -1 ``` -If the revision is stale, ask whether the controller is doing *anything*: +**`reconciledAt` is not a heartbeat — do not read it as one.** ArgoCD writes that field only +when the computed status *changes*. On a cluster where everything is Synced and nothing is +deploying, it stops advancing and grows arbitrarily old while the controller is perfectly +healthy. Measured 2026-07-22: six samples 50s apart against a `timeout.reconciliation` of +120s, zero movement, controller fine. It is only evidence of a problem when paired with a +`sync.revision` that is *behind `master`* — one stale field is normal, both together are the +symptom. (A health check in `oci-k8s` used to threshold on its age alone and warned forever; +it now measures the controller instead. Same mistake, don't repeat it here.) + +So if the revision is stale, ask whether the controller is doing *anything*: + +```bash +# The controller re-reads its cached resource tree out of Redis on every refresh cycle, +# whether or not anything changed. Zero over 15m means it has stopped reconciling. +kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + wget -qO- --post-data='query=sum(increase(argocd_redis_request_total{job="argocd-application-controller-metrics"}[15m]))' \ + http://localhost:9090/api/v1/query + +# And whether a reconcile started and never finished — this is what a webhook block looks +# like. Healthy is single-digit seconds; the 2026-07-21 wedge would have read ~11 hours. +kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + wget -qO- --post-data='query=max(workqueue_unfinished_work_seconds{name="app_reconciliation_queue"})' \ + http://localhost:9090/api/v1/query +``` + +Without Prometheus, fall back to the logs: ```bash # Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.