review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

services/worker/handlers.py

@@ -2,7 +2,7 @@
 
 Every handler here obeys one rule: running it twice must equal running it once.
 
-That is not a nicety. A worker can be SIGKILLed after helm has installed the release but
+A worker can be SIGKILLed after helm has installed the release but
 before the DB row says so; the lease expires; another worker claims the same task and runs
 this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
 and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
@@ -63,6 +63,15 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
     # below is idempotent either way, so this is bookkeeping, not a lock.
     await deps.instances.update_state(inst.id, InstanceState.REQUESTED, InstanceState.PROVISIONING)
 
+    # `helm --namespace X` does not create X. Every tenant's first provision lands in a
+    # namespace that does not exist yet, so this has to happen before helm runs or the
+    # install fails with "namespaces not found". Idempotent (kubectl apply of a Namespace
+    # manifest), so it costs one no-op API call on every subsequent provision.
+    await deps.namespaces.ensure_namespace(
+        inst.namespace,
+        labels={"svcforge.io/team": inst.team},
+    )
+
     await deps.provisioner.install(
         release=inst.release_name,
         ns=inst.namespace,
@@ -92,7 +101,15 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
     # `helm uninstall` of an already-gone release is not an error to us: the adapter
     # swallows not-found, because the desired state — no release — is already true.
     await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
-    await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED)
+
+    # Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
+    # the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
+    # and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
+    # asked to delete. Failing loudly turns a silent ping-pong into one visible error.
+    if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
+        raise HandlerError(
+            f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
+        )
 
 
 async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
@@ -138,18 +155,27 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
     if inst.release_name in releases:
         return
 
+    # `returning` + a `where` on the update half tells us whether THIS call was the one
+    # that halted the rollout. The halt itself is idempotent; the page is not. Without the
+    # distinction, a verify that fails its full retry budget sends five identical
+    # notifications for one incident, spread across the backoff curve.
     async with deps.pool.connection() as conn, conn.cursor() as cur:
         await cur.execute(
             """insert into catalog_versions (service_type, rollout_state)
                     values (%s, 'halted')
-               on conflict (service_type) do update set rollout_state = 'halted'""",
+               on conflict (service_type) do update set rollout_state = 'halted'
+                     where catalog_versions.rollout_state <> 'halted'
+               returning service_type""",
             (inst.service_type,),
         )
-    await deps.notifier.send(
-        "rollout.halted",
-        f"rollout halted for {inst.service_type}: {inst.release_name} failed verify",
-        {"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
-    )
+        newly_halted = await cur.fetchone() is not None
+
+    if newly_halted:
+        await deps.notifier.send(
+            "rollout.halted",
+            f"rollout halted for {inst.service_type}: {inst.release_name} failed verify",
+            {"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
+        )
     raise HandlerError(f"verify failed for {inst.release_name}; rollout halted")