review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

deploy/chart/templates/_helpers.tpl

@@ -53,8 +53,18 @@ comments — the acceptance gate greps for it and does not know what a comment i
 {{- if not $img -}}
 {{- fail (printf "no image config for component %q" .component) -}}
 {{- end -}}
-{{- if not (hasPrefix "sha256:" ($img.digest | default "")) -}}
-{{- fail (printf "image.%s.digest must be a sha256 digest, not a tag — CI bumps it; got %q" .component ($img.digest | default "<empty>")) -}}
+{{- $digest := $img.digest | default "" -}}
+{{/*
+Full-shape match, not `hasPrefix "sha256:"`. A prefix check accepts the all-zeros
+placeholder in values.yaml, so a fresh clone rendered clean and the guard guarded nothing.
+Two conditions, both required: the digest must be sha256: plus exactly 64 lowercase hex
+characters, AND it must not be the placeholder literal.
+*/}}
+{{- if not (regexMatch "^sha256:[0-9a-f]{64}$" $digest) -}}
+{{- fail (printf "image.%s.digest must be a sha256 digest (sha256: + 64 hex chars), not a tag — CI bumps it; got %q" .component ($digest | default "<empty>")) -}}
+{{- end -}}
+{{- if eq $digest "sha256:0000000000000000000000000000000000000000000000000000000000000000" -}}
+{{- fail (printf "image.%s.digest is still the all-zeros placeholder from values.yaml — this chart has never been bumped by CI and must not be deployed" .component) -}}
 {{- end -}}
 {{- printf "%s@%s" $img.repo $img.digest -}}
 {{- end -}}

deploy/chart/templates/migrate-job.yaml

@@ -59,6 +59,12 @@ spec:
             {{- include "svcforge.env" . | nindent 12 }}
             - name: OTEL_SERVICE_NAME
               value: svcforge-migrate
+            # Where services/api/Dockerfile copies migrations/ to. The Dockerfile sets the
+            # same value as an ENV; this states it in the manifest as well so the path is
+            # visible to anyone reading the Job rather than only to whoever opens the
+            # image. The two MUST agree — if one moves, move both.
+            - name: SVCFORGE_MIGRATIONS_DIR
+              value: /app/migrations
           resources:
             {{- toYaml .Values.migrate.resources | nindent 12 }}
           volumeMounts:

deploy/chart/templates/servicemonitor.yaml

@@ -6,6 +6,14 @@ is banned: it drifts from the chart, survives a `helm uninstall`, and nothing ow
 The api is scraped through its Service. The worker and reconciler have no Service — they
 are scraped by pod, which is why their metrics port is named and their pods carry the
 component label.
+
+A ServiceMonitor's selector matches the SERVICE OBJECT's own metadata labels, not the
+Service's pod selector. api-service.yaml labels itself with `svcforge.labels` plus
+`app.kubernetes.io/component: api` — it does NOT carry the `app: api` that
+`svcforge.selectorLabels` adds, because that label exists for the chaos experiments'
+`kubectl delete pod -l app=worker` and belongs on pods. So this must not reuse
+selectorLabels: doing so matched nothing and api metrics were never scraped. These three
+keys are exactly the ones the Service metadata carries, and are enough to be unambiguous.
 */}}
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
@@ -16,7 +24,9 @@ metadata:
 spec:
   selector:
     matchLabels:
-      {{- include "svcforge.selectorLabels" (dict "ctx" $ "component" "api") | nindent 6 }}
+      app.kubernetes.io/name: {{ include "svcforge.name" . }}
+      app.kubernetes.io/instance: {{ .Release.Name }}
+      app.kubernetes.io/component: api
   endpoints:
     - port: http
       path: /metrics

deploy/chart/templates/worker-deployment.yaml

@@ -25,9 +25,19 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       # The load-bearing one. On SIGTERM the loop stops claiming and finishes the task in
-      # hand; 60s is the budget for that. Chaos experiment 2 asserts the pod exits 0 inside
-      # it. Lower this and a rolling deploy starts orphaning tasks to lease expiry.
-      terminationGracePeriodSeconds: 60
+      # hand. Chaos experiment 2 asserts the pod exits 0 inside it. Lower this and a
+      # rolling deploy starts orphaning tasks to lease expiry.
+      #
+      # 660s, and the number is derived, not chosen. The task being drained is a
+      # `helm upgrade --install`, and this MUST exceed the longest that call can run before
+      # it returns:
+      #   - settings.py           helm_timeout_s  = 300  (default)
+      #   - HelmProvisioner       timeout_s       = 600  (default, the ceiling)
+      # 600s is the value to beat; 660 leaves a 60s margin for the pool to close and the
+      # process to exit cleanly. At the old 60s the kubelet SIGKILLed the worker mid-helm —
+      # precisely the orphaning this setting exists to prevent. If either timeout above is
+      # raised, raise this past it first.
+      terminationGracePeriodSeconds: 660
       securityContext:
         {{- include "svcforge.podSecurityContext" . | nindent 8 }}
       containers:

deploy/chart/values.yaml

@@ -148,15 +148,29 @@ prometheusRule:
       annotations:
         summary: svcforge p95 provision time is over 5 minutes
         runbook_url: https://gitea.oci-oci.duckdns.org/gitea_admin/svcforge/src/branch/master/RUNBOOK.md#provision-failing
-    - alert: SvcforgeTaskFailed
-      expr: sum(svcforge_tasks_failed_total) > 0
+    # `increase(...[15m])`, not the raw counter. `sum(counter) > 0` on a monotonic counter
+    # latches: one dead-lettered task at any point keeps this firing until the pod restarts,
+    # and a restart silently clears it — so it can never distinguish "failing now" from
+    # "failed last Tuesday". The dead-letter counter is also the right one: the attempts
+    # counter increments on ordinary transient retries that later succeed.
+    - alert: SvcforgeTaskDeadLettered
+      expr: sum(increase(svcforge_tasks_dead_lettered_total[15m])) > 0
+      for: 5m
       labels:
         severity: warning
       annotations:
         summary: a svcforge task exhausted its retries
         runbook_url: https://gitea.oci-oci.duckdns.org/gitea_admin/svcforge/src/branch/master/RUNBOOK.md#provision-failing
+    # Scoped to the reconciler job, and aggregated with max().
+    #
+    # RECONCILER_LAST_TICK is a module-level Gauge in obs.py, so EVERY service that imports
+    # svcforge_core.obs registers and exports it — api and worker included, permanently at
+    # 0. Unscoped, `time() - 0` is ~1.7e9, so this critical alert fires from the moment the
+    # chart is installed, from pods that have no reconciler in them. Scope by job, then
+    # max() so a rolling restart of the single reconciler does not flap it.
     - alert: SvcforgeReconcilerStale
-      expr: time() - svcforge_reconciler_last_tick_timestamp_seconds > 300
+      expr: time() - max(svcforge_reconciler_last_tick_timestamp_seconds{job=~".*reconciler.*"}) > 300
+      for: 5m
       labels:
         severity: critical
       annotations: