review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

services/worker/main.py

@@ -13,6 +13,7 @@ import asyncio
 import contextlib
 import signal
 import time
+from collections.abc import Awaitable
 
 from opentelemetry import trace
 
@@ -21,9 +22,10 @@ from services.worker.handlers import HANDLERS
 from svcforge_core import obs
 from svcforge_core.adapters.clock import SystemClock
 from svcforge_core.adapters.helm import HelmProvisioner
+from svcforge_core.adapters.k8s import KubectlClient
 from svcforge_core.adapters.notify import LogNotifier
 from svcforge_core.domain.catalog import load_catalog
-from svcforge_core.domain.models import Task
+from svcforge_core.domain.models import Task, TaskKind
 from svcforge_core.repo.db import make_pool
 from svcforge_core.repo.instances import InstanceRepo
 from svcforge_core.repo.tasks import TaskRepo
@@ -42,8 +44,28 @@ async def _sleep_or_stop(stop: asyncio.Event, seconds: float) -> None:
         await asyncio.wait_for(stop.wait(), timeout=seconds)
 
 
+async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
+    """Run a terminal report, and never let its failure escape.
+
+    Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
+    TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
+    DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
+    not just this one. The task itself is safe either way: it stays `running` and the
+    reconciler's lease sweep returns it to the queue. Losing the report costs one lease
+    interval; losing the siblings costs their work.
+    """
+    try:
+        if not await coro:
+            # The lease was stolen while we were working: another worker owns this task
+            # now and is mid-run. Reporting is theirs to do, not ours.
+            log.warning("lease lost before report; another worker owns this task", task_id=task_id)
+    except Exception:
+        log.exception("could not report task %s (%s); lease will expire", task_id, what)
+
+
 async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None:
     """Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
+    worker_id = deps.settings.worker_id
     try:
         # Every log line from here carries instance_id/task_id/team. Bound once, at claim,
         # rather than passed down: the alternative is threading three arguments through
@@ -55,7 +77,11 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
 
         handler = HANDLERS.get(task.kind)
         if handler is None:
-            await deps.tasks.fail(task.id, f"no handler for {task.kind}", max_attempts=1)
+            await _report(
+                deps.tasks.fail(task.id, f"no handler for {task.kind}", worker_id, max_attempts=1),
+                task.id,
+                "no-handler",
+            )
             return
 
         # Re-parent to the span that enqueued this task. Without the stored traceparent
@@ -77,17 +103,30 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
                 # CancelledError inherits from BaseException, so `except Exception` below
                 # would never see it. Catch it only to release the claim, then ALWAYS
                 # re-raise: swallowing it breaks cancellation for everyone above us.
-                await deps.tasks.fail(task.id, "cancelled", max_attempts=deps.settings.max_attempts)
+                await _report(
+                    deps.tasks.fail(task.id, "cancelled", worker_id, deps.settings.max_attempts),
+                    task.id,
+                    "cancelled",
+                )
                 raise
             except Exception as exc:
                 log.exception("task failed", kind=task.kind.value, error=str(exc))
                 span.record_exception(exc)
                 span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
-                obs.TASKS_FAILED.labels(kind=task.kind.value).inc()
-                await deps.tasks.fail(task.id, str(exc), max_attempts=deps.settings.max_attempts)
+                obs.TASK_ATTEMPTS_FAILED.labels(kind=task.kind.value).inc()
+                await _report(
+                    deps.tasks.fail(task.id, str(exc), worker_id, deps.settings.max_attempts),
+                    task.id,
+                    "fail",
+                )
             else:
-                obs.PROVISION_TIME.observe(time.monotonic() - started)
-                await deps.tasks.complete(task.id)
+                # Only provisions go in the provision histogram. The buckets run 10s..1800s
+                # because they were sized for helm installs; a sub-second `verify` dropped
+                # into the same series drags the p95 down and quietly stops
+                # SvcforgeProvisionSlow from ever firing.
+                if task.kind is TaskKind.PROVISION:
+                    obs.PROVISION_TIME.observe(time.monotonic() - started)
+                await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
     finally:
         sem.release()
 
@@ -135,6 +174,7 @@ async def _amain() -> None:
     # the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
     # registry is up.
     obs.setup("svcforge-worker", settings)
+    settings.check_production()
     obs.start_metrics_server(settings.metrics_port)
 
     pool = make_pool(settings.pg_dsn.unicode_string(), settings.pool_min_size, settings.pool_max_size)
@@ -145,6 +185,7 @@ async def _amain() -> None:
         instances=InstanceRepo(pool),
         tasks=TaskRepo(pool),
         provisioner=HelmProvisioner(helm_bin=settings.helm_bin, timeout_s=int(settings.helm_timeout_s)),
+        namespaces=KubectlClient(kubectl_bin=settings.kubectl_bin),
         notifier=LogNotifier(),
         clock=SystemClock(),
         catalog=load_catalog(settings.catalog_path),