From c691f4f4aad936cd91869ec5527a7e18e40f5a60 Mon Sep 17 00:00:00 2001
From: Nguyen Minh Phuc <minhphuc@protonmail.com>
Date: Wed, 22 Jul 2026 02:25:10 +0000
Subject: [PATCH] runbook: reconciledAt is not a heartbeat

The stuck-deploy section printed `.status.reconciledAt` as a diagnostic without
saying how to read it, which invites exactly the wrong conclusion: ArgoCD only
writes that field when the computed status changes, so on an idle cluster it
stops advancing while the controller is healthy. Measured 2026-07-22 -- six
samples 50s apart against a 120s reconciliation timeout, zero movement.

It only means something alongside a sync.revision that is behind master.

Replaces the "is it doing anything" step with two metrics that answer it
directly: the controller's Redis cache reads, which happen every refresh cycle
regardless of change, and the reconcile queue's unfinished-work seconds, which
is precisely what the 2026-07-21 webhook wedge looked like. Log-based check
kept as the no-Prometheus fallback.
---
 RUNBOOK.md | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/RUNBOOK.md b/RUNBOOK.md
index 1aad2e2..e930d1a 100644
--- a/RUNBOOK.md
+++ b/RUNBOOK.md
@@ -499,7 +499,32 @@ kubectl -n argocd get application svcforge \
 git -C ~/workspace/svcforge-reference log --oneline origin/master -1
 ```
 
-If the revision is stale, ask whether the controller is doing *anything*:
+**`reconciledAt` is not a heartbeat — do not read it as one.** ArgoCD writes that field only
+when the computed status *changes*. On a cluster where everything is Synced and nothing is
+deploying, it stops advancing and grows arbitrarily old while the controller is perfectly
+healthy. Measured 2026-07-22: six samples 50s apart against a `timeout.reconciliation` of
+120s, zero movement, controller fine. It is only evidence of a problem when paired with a
+`sync.revision` that is *behind `master`* — one stale field is normal, both together are the
+symptom. (A health check in `oci-k8s` used to threshold on its age alone and warned forever;
+it now measures the controller instead. Same mistake, don't repeat it here.)
+
+So if the revision is stale, ask whether the controller is doing *anything*:
+
+```bash
+# The controller re-reads its cached resource tree out of Redis on every refresh cycle,
+# whether or not anything changed. Zero over 15m means it has stopped reconciling.
+kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+  wget -qO- --post-data='query=sum(increase(argocd_redis_request_total{job="argocd-application-controller-metrics"}[15m]))' \
+  http://localhost:9090/api/v1/query
+
+# And whether a reconcile started and never finished — this is what a webhook block looks
+# like. Healthy is single-digit seconds; the 2026-07-21 wedge would have read ~11 hours.
+kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+  wget -qO- --post-data='query=max(workqueue_unfinished_work_seconds{name="app_reconciliation_queue"})' \
+  http://localhost:9090/api/v1/query
+```
+
+Without Prometheus, fall back to the logs:
 
 ```bash
 # Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.