review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

services/reconciler/Dockerfile

@@ -41,7 +41,10 @@ RUN useradd -u 10001 -m -s /usr/sbin/nologin svcforge
 WORKDIR /app
 COPY --from=builder --chown=10001:10001 /app /app
 
-COPY --from=alpine/helm:3.16.2@sha256:a19a2968fd672336d39771f6c899781424d725229148656dbc2a1e305003cdec /usr/bin/helm /usr/local/bin/helm
+# helm 3.21.3, not 3.16.2 — see services/worker/Dockerfile. 3.16.2 is a Go 1.22.9 build
+# carrying CRITICAL CVE-2025-68121 (crypto/tls) and CVE-2026-33186 (grpc) and HIGH
+# CVE-2026-35469 (spdystream). Kept on 3.x on purpose: helm 4 is a breaking change.
+COPY --from=alpine/helm:3.21.3@sha256:35da09ba0716fc7c3cd63b6b31ee380a9c7662e95f29ab0e4ae962420afd315b /usr/bin/helm /usr/local/bin/helm
 
 ENV PATH="/app/.venv/bin:$PATH" \
     PYTHONUNBUFFERED=1 \

services/reconciler/main.py

@@ -81,8 +81,8 @@ class ReconcilerDeps:
     clock: Clock
     catalog: dict[str, CatalogEntry]
     settings: Settings
-    # Whose instances go first in the day-2 work list. Eating your own dog food is an
-    # `order by`, not a policy document — see InstanceRepo.list_upgradable.
+    # Whose instances go first in the day-2 work list. Eating your own dog food is enforced
+    # by an `order by` rather than a policy document — see InstanceRepo.list_upgradable.
     own_team: str
     # A config value, not a scheduler. Leave it at 1 until 1 is too slow.
     max_in_flight: int
@@ -152,15 +152,15 @@ async def check_drift(deps: ReconcilerDeps) -> None:
 async def check_lease_expiry(deps: ReconcilerDeps) -> None:
     """Tasks whose worker died -> back to `queued`.
 
-    A lease, not a lock. No lock survives a power cut: a worker SIGKILLed mid-provision
+    A lease. No lock survives a power cut: a worker SIGKILLed mid-provision
     leaves `state='running'` with `locked_by` set and nobody running it, and no amount of
     cleanup code in the worker helps, because the worker is the part that died. `locked_at`
     plus a timeout is the only thing that recovers the row, which is why `locked_at` exists.
 
-    The 5-minute default is not arbitrary: it must exceed the longest a healthy task can
-    hold a lease, or the reconciler hands a still-running provision to a second worker.
-    Handlers are idempotent, so that is survivable rather than fatal — but survivable is
-    not free, and `lease_seconds` sits above helm's `--timeout` for that reason.
+    The 5-minute default must exceed the longest a healthy task can hold a lease, or the
+    reconciler hands a still-running provision to a second worker. Handlers are idempotent,
+    so that is survivable, though it still costs a duplicated helm run — which is why
+    `lease_seconds` sits above helm's `--timeout`.
     """
     freed = await deps.tasks.reset_expired_leases(deps.settings.lease_seconds)
     if freed: