runbook: reconciledAt is not a heartbeat
ci / lint (push) Successful in 33s
ci / types (push) Successful in 50s
ci / unit (push) Successful in 31s
ci / security (push) Successful in 1m17s
ci / dockerfile (push) Successful in 10s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 1m6s
ci / image (api) (push) Successful in 4m44s
ci / image (reconciler) (push) Successful in 4m5s
ci / image (worker) (push) Successful in 2m55s
ci / bump (push) Successful in 1m12s

The stuck-deploy section printed `.status.reconciledAt` as a diagnostic without
saying how to read it, which invites exactly the wrong conclusion: ArgoCD only
writes that field when the computed status changes, so on an idle cluster it
stops advancing while the controller is healthy. Measured 2026-07-22 -- six
samples 50s apart against a 120s reconciliation timeout, zero movement.

It only means something alongside a sync.revision that is behind master.

Replaces the "is it doing anything" step with two metrics that answer it
directly: the controller's Redis cache reads, which happen every refresh cycle
regardless of change, and the reconcile queue's unfinished-work seconds, which
is precisely what the 2026-07-21 webhook wedge looked like. Log-based check
kept as the no-Prometheus fallback.
This commit is contained in:
2026-07-22 02:25:10 +00:00
parent a0215ef63f
commit c691f4f4aa
+26 -1
View File
@@ -499,7 +499,32 @@ kubectl -n argocd get application svcforge \
git -C ~/workspace/svcforge-reference log --oneline origin/master -1
```
If the revision is stale, ask whether the controller is doing *anything*:
**`reconciledAt` is not a heartbeat — do not read it as one.** ArgoCD writes that field only
when the computed status *changes*. On a cluster where everything is Synced and nothing is
deploying, it stops advancing and grows arbitrarily old while the controller is perfectly
healthy. Measured 2026-07-22: six samples 50s apart against a `timeout.reconciliation` of
120s, zero movement, controller fine. It is only evidence of a problem when paired with a
`sync.revision` that is *behind `master`* — one stale field is normal, both together are the
symptom. (A health check in `oci-k8s` used to threshold on its age alone and warned forever;
it now measures the controller instead. Same mistake, don't repeat it here.)
So if the revision is stale, ask whether the controller is doing *anything*:
```bash
# The controller re-reads its cached resource tree out of Redis on every refresh cycle,
# whether or not anything changed. Zero over 15m means it has stopped reconciling.
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- --post-data='query=sum(increase(argocd_redis_request_total{job="argocd-application-controller-metrics"}[15m]))' \
http://localhost:9090/api/v1/query
# And whether a reconcile started and never finished — this is what a webhook block looks
# like. Healthy is single-digit seconds; the 2026-07-21 wedge would have read ~11 hours.
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
wget -qO- --post-data='query=max(workqueue_unfinished_work_seconds{name="app_reconciliation_queue"})' \
http://localhost:9090/api/v1/query
```
Without Prometheus, fall back to the logs:
```bash
# Healthy: a few hundred lines an hour. Stalled: exactly 6 — the 10-minute memory heartbeat.