review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

scripts/load.py

@@ -1,9 +1,9 @@
 """Throwaway load generator. Enqueue N instances, watch the queue drain, print three numbers.
 
-The point is not a benchmark. It is to find the ceiling on purpose, in a place where finding
-it is free, so that the number in RUNBOOK.md comes from an observation instead of a guess.
+The point is to find the ceiling on purpose, in a place where finding it is free, so that
+the number in RUNBOOK.md comes from an observation instead of a guess.
 
-The ceiling you are looking for is arithmetic, not mysterious:
+The ceiling you are looking for is arithmetic:
 
     total connections = (api_replicas + worker_replicas) x pool_max_size
 
@@ -23,20 +23,28 @@ import argparse
 import asyncio
 import time
 from datetime import UTC, datetime
+from typing import Any
 from uuid import uuid4
 
 import psycopg
 from psycopg.rows import dict_row
 
+from svcforge_core.domain.catalog import load_catalog
 from svcforge_core.settings import load_settings
 
+_SERVICE_TYPE = "elasticsearch"
 
-async def _seed_direct(dsn: str, count: int) -> float:
+
+async def _seed_direct(dsn: str, count: int, chart_version: str) -> float:
     """Insert `count` instances + provision tasks. Returns seconds taken.
 
     --direct exists to separate two questions that a single POST run conflates: "how fast can
     the API accept work" and "how fast can workers drain it". Measure them apart or you will
     tune the wrong one.
+
+    `chart_version` comes from the catalog rather than a literal. Hardcoding it meant the
+    seeded rows carried a version the catalog could not resolve, so every task failed fast
+    and the drain measurement — the whole point of the script — timed the failure path.
     """
     started = time.monotonic()
     async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row) as conn:
@@ -46,9 +54,9 @@ async def _seed_direct(dsn: str, count: int) -> float:
                 await cur.execute(
                     """insert into instances (id, team, service_type, size, state, namespace,
                                               release_name, chart_version)
-                       values (%s, 'loadtest', 'elasticsearch', 'small', 'requested',
-                               'tenant-loadtest', %s, '21.3.19')""",
-                    (iid, f"loadtest-elasticsearch-{str(iid)[:8]}"),
+                       values (%s, 'loadtest', %s, 'small', 'requested',
+                               'tenant-loadtest', %s, %s)""",
+                    (iid, _SERVICE_TYPE, f"loadtest-{_SERVICE_TYPE}-{str(iid)[:8]}", chart_version),
                 )
                 await cur.execute(
                     "insert into tasks (instance_id, kind) values (%s, 'provision')",
@@ -57,38 +65,49 @@ async def _seed_direct(dsn: str, count: int) -> float:
     return time.monotonic() - started
 
 
-async def _depth(dsn: str) -> dict[str, int]:
-    async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row) as conn:
-        cur = await conn.execute("select state, count(*) as n from tasks group by 1")
-        return {str(r["state"]): int(r["n"]) for r in await cur.fetchall()}
+async def _depth(conn: psycopg.AsyncConnection[dict[str, Any]]) -> dict[str, int]:
+    """Task counts by state, on a connection the caller owns."""
+    cur = await conn.execute("select state, count(*) as n from tasks group by 1")
+    return {str(r["state"]): int(r["n"]) for r in await cur.fetchall()}
 
 
 async def _watch(dsn: str, timeout_s: float) -> None:
-    """Print queue depth once a second until it drains. The slope is the number you want."""
+    """Print queue depth once a second until it drains. The slope is the number you want.
+
+    One connection for the whole loop, held open. Reconnecting every second added a
+    connection to the pooler on every tick of a script whose entire purpose is finding the
+    connection ceiling — the measurement was perturbing the thing being measured.
+    """
     started = time.monotonic()
     peak = 0
     print(f"{'t(s)':>6}  {'queued':>7} {'running':>8} {'done':>6} {'failed':>7}  slope/s")
     prev_done, prev_t = 0, started
 
-    while time.monotonic() - started < timeout_s:
-        d = await _depth(dsn)
-        queued, running = d.get("queued", 0), d.get("running", 0)
-        done, failed = d.get("done", 0), d.get("failed", 0)
-        peak = max(peak, queued + running)
+    # autocommit: a held connection without it sits idle-in-transaction between polls, which
+    # pins a snapshot on the pooler and is exactly the pathology this script hunts for.
+    async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row, autocommit=True) as conn:
+        while time.monotonic() - started < timeout_s:
+            d = await _depth(conn)
+            queued, running = d.get("queued", 0), d.get("running", 0)
+            done, failed = d.get("done", 0), d.get("failed", 0)
+            peak = max(peak, queued + running)
 
-        now = time.monotonic()
-        slope = (done - prev_done) / max(now - prev_t, 1e-9)
-        prev_done, prev_t = done, now
+            now = time.monotonic()
+            slope = (done - prev_done) / max(now - prev_t, 1e-9)
+            prev_done, prev_t = done, now
 
-        print(f"{now - started:6.1f}  {queued:7d} {running:8d} {done:6d} {failed:7d}  {slope:7.1f}")
+            print(f"{now - started:6.1f}  {queued:7d} {running:8d} {done:6d} {failed:7d}  {slope:7.1f}")
 
-        if queued == 0 and running == 0:
-            elapsed = now - started
-            print(f"\ndrained in {elapsed:.1f}s   peak depth {peak}   throughput {done / elapsed:.1f} task/s")
-            if failed:
-                print(f"WARNING: {failed} tasks failed — the number above is not a clean drain")
-            return
-        await asyncio.sleep(1.0)
+            if queued == 0 and running == 0:
+                elapsed = now - started
+                print(
+                    f"\ndrained in {elapsed:.1f}s   peak depth {peak}   "
+                    f"throughput {done / elapsed:.1f} task/s"
+                )
+                if failed:
+                    print(f"WARNING: {failed} tasks failed — the number above is not a clean drain")
+                return
+            await asyncio.sleep(1.0)
 
     print(f"\nstill draining after {timeout_s}s — that IS the result. Record it.")
 
@@ -117,8 +136,16 @@ async def _amain() -> None:
             "with k6 (one dependency, not two — do not add locust for this)."
         )
 
+    # The version the workers will actually resolve. Read it rather than restate it: a seeded
+    # row whose chart_version disagrees with the catalog drains through the failure path.
+    catalog = load_catalog(settings.catalog_path)
+    entry = catalog.get(_SERVICE_TYPE)
+    if entry is None:
+        raise SystemExit(f"{settings.catalog_path} has no '{_SERVICE_TYPE}' entry to load-test with")
+
     print(f"seeding {args.count} instances at {datetime.now(UTC).isoformat()} ...")
-    took = await _seed_direct(dsn, args.count)
+    print(f"  service_type={_SERVICE_TYPE} chart_version={entry.chart_version} (from catalog)")
+    took = await _seed_direct(dsn, args.count, entry.chart_version)
     print(f"enqueued {args.count} in {took:.2f}s ({args.count / took:.0f}/s)\n")
 
     if args.watch: