review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

scripts/load.py

@@ -1,9 +1,9 @@
 """Throwaway load generator. Enqueue N instances, watch the queue drain, print three numbers.
 
-The point is not a benchmark. It is to find the ceiling on purpose, in a place where finding
-it is free, so that the number in RUNBOOK.md comes from an observation instead of a guess.
+The point is to find the ceiling on purpose, in a place where finding it is free, so that
+the number in RUNBOOK.md comes from an observation instead of a guess.
 
-The ceiling you are looking for is arithmetic, not mysterious:
+The ceiling you are looking for is arithmetic:
 
     total connections = (api_replicas + worker_replicas) x pool_max_size
 
@@ -23,20 +23,28 @@ import argparse
 import asyncio
 import time
 from datetime import UTC, datetime
+from typing import Any
 from uuid import uuid4
 
 import psycopg
 from psycopg.rows import dict_row
 
+from svcforge_core.domain.catalog import load_catalog
 from svcforge_core.settings import load_settings
 
+_SERVICE_TYPE = "elasticsearch"
 
-async def _seed_direct(dsn: str, count: int) -> float:
+
+async def _seed_direct(dsn: str, count: int, chart_version: str) -> float:
     """Insert `count` instances + provision tasks. Returns seconds taken.
 
     --direct exists to separate two questions that a single POST run conflates: "how fast can
     the API accept work" and "how fast can workers drain it". Measure them apart or you will
     tune the wrong one.
+
+    `chart_version` comes from the catalog rather than a literal. Hardcoding it meant the
+    seeded rows carried a version the catalog could not resolve, so every task failed fast
+    and the drain measurement — the whole point of the script — timed the failure path.
     """
     started = time.monotonic()
     async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row) as conn:
@@ -46,9 +54,9 @@ async def _seed_direct(dsn: str, count: int) -> float:
                 await cur.execute(
                     """insert into instances (id, team, service_type, size, state, namespace,
                                               release_name, chart_version)
-                       values (%s, 'loadtest', 'elasticsearch', 'small', 'requested',
-                               'tenant-loadtest', %s, '21.3.19')""",
-                    (iid, f"loadtest-elasticsearch-{str(iid)[:8]}"),
+                       values (%s, 'loadtest', %s, 'small', 'requested',
+                               'tenant-loadtest', %s, %s)""",
+                    (iid, _SERVICE_TYPE, f"loadtest-{_SERVICE_TYPE}-{str(iid)[:8]}", chart_version),
                 )
                 await cur.execute(
                     "insert into tasks (instance_id, kind) values (%s, 'provision')",
@@ -57,38 +65,49 @@ async def _seed_direct(dsn: str, count: int) -> float:
     return time.monotonic() - started
 
 
-async def _depth(dsn: str) -> dict[str, int]:
-    async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row) as conn:
-        cur = await conn.execute("select state, count(*) as n from tasks group by 1")
-        return {str(r["state"]): int(r["n"]) for r in await cur.fetchall()}
+async def _depth(conn: psycopg.AsyncConnection[dict[str, Any]]) -> dict[str, int]:
+    """Task counts by state, on a connection the caller owns."""
+    cur = await conn.execute("select state, count(*) as n from tasks group by 1")
+    return {str(r["state"]): int(r["n"]) for r in await cur.fetchall()}
 
 
 async def _watch(dsn: str, timeout_s: float) -> None:
-    """Print queue depth once a second until it drains. The slope is the number you want."""
+    """Print queue depth once a second until it drains. The slope is the number you want.
+
+    One connection for the whole loop, held open. Reconnecting every second added a
+    connection to the pooler on every tick of a script whose entire purpose is finding the
+    connection ceiling — the measurement was perturbing the thing being measured.
+    """
     started = time.monotonic()
     peak = 0
     print(f"{'t(s)':>6}  {'queued':>7} {'running':>8} {'done':>6} {'failed':>7}  slope/s")
     prev_done, prev_t = 0, started
 
-    while time.monotonic() - started < timeout_s:
-        d = await _depth(dsn)
-        queued, running = d.get("queued", 0), d.get("running", 0)
-        done, failed = d.get("done", 0), d.get("failed", 0)
-        peak = max(peak, queued + running)
+    # autocommit: a held connection without it sits idle-in-transaction between polls, which
+    # pins a snapshot on the pooler and is exactly the pathology this script hunts for.
+    async with await psycopg.AsyncConnection.connect(dsn, row_factory=dict_row, autocommit=True) as conn:
+        while time.monotonic() - started < timeout_s:
+            d = await _depth(conn)
+            queued, running = d.get("queued", 0), d.get("running", 0)
+            done, failed = d.get("done", 0), d.get("failed", 0)
+            peak = max(peak, queued + running)
 
-        now = time.monotonic()
-        slope = (done - prev_done) / max(now - prev_t, 1e-9)
-        prev_done, prev_t = done, now
+            now = time.monotonic()
+            slope = (done - prev_done) / max(now - prev_t, 1e-9)
+            prev_done, prev_t = done, now
 
-        print(f"{now - started:6.1f}  {queued:7d} {running:8d} {done:6d} {failed:7d}  {slope:7.1f}")
+            print(f"{now - started:6.1f}  {queued:7d} {running:8d} {done:6d} {failed:7d}  {slope:7.1f}")
 
-        if queued == 0 and running == 0:
-            elapsed = now - started
-            print(f"\ndrained in {elapsed:.1f}s   peak depth {peak}   throughput {done / elapsed:.1f} task/s")
-            if failed:
-                print(f"WARNING: {failed} tasks failed — the number above is not a clean drain")
-            return
-        await asyncio.sleep(1.0)
+            if queued == 0 and running == 0:
+                elapsed = now - started
+                print(
+                    f"\ndrained in {elapsed:.1f}s   peak depth {peak}   "
+                    f"throughput {done / elapsed:.1f} task/s"
+                )
+                if failed:
+                    print(f"WARNING: {failed} tasks failed — the number above is not a clean drain")
+                return
+            await asyncio.sleep(1.0)
 
     print(f"\nstill draining after {timeout_s}s — that IS the result. Record it.")
 
@@ -117,8 +136,16 @@ async def _amain() -> None:
             "with k6 (one dependency, not two — do not add locust for this)."
         )
 
+    # The version the workers will actually resolve. Read it rather than restate it: a seeded
+    # row whose chart_version disagrees with the catalog drains through the failure path.
+    catalog = load_catalog(settings.catalog_path)
+    entry = catalog.get(_SERVICE_TYPE)
+    if entry is None:
+        raise SystemExit(f"{settings.catalog_path} has no '{_SERVICE_TYPE}' entry to load-test with")
+
     print(f"seeding {args.count} instances at {datetime.now(UTC).isoformat()} ...")
-    took = await _seed_direct(dsn, args.count)
+    print(f"  service_type={_SERVICE_TYPE} chart_version={entry.chart_version} (from catalog)")
+    took = await _seed_direct(dsn, args.count, entry.chart_version)
     print(f"enqueued {args.count} in {took:.2f}s ({args.count / took:.0f}/s)\n")
 
     if args.watch:

scripts/redis_budget.py

@@ -3,8 +3,17 @@
 
     $ python3 scripts/redis_budget.py
     $ python3 scripts/redis_budget.py --url http://localhost:8000/metrics --budget 500000
+    $ python3 scripts/redis_budget.py --url http://api:8000/metrics \
+                                      --url http://worker:9100/metrics \
+                                      --url http://reconciler:9100/metrics
 
-Upstash's free tier is 500,000 commands/month, which sounds enormous and is not:
+The budget is per Upstash database; the counters are per process. api, worker and
+reconciler each keep their own registry (see obs.py — one process per pod, no multiproc
+directory), so scraping one endpoint measures one third of the burn. Repeat `--url` to sum
+them; a run that covers fewer than three sources says so in its output rather than printing
+a reassuring number derived from one process.
+
+Upstash's free tier is 500,000 commands/month, which is far smaller than it sounds:
 
     500,000 / month = 16,129 / day = 11 / minute = 0.19 / second, sustained
 
@@ -14,7 +23,7 @@ Redis is only ever on the request path here, and why this script exists: the rul
 to state and invisible to violate. A `cache.get()` added inside the reconciler's
 per-instance loop is one line in review and 2,160,000 commands/month in production.
 
-**Why a projection and not an alarm on the counter.** Exhausting the budget is a slow,
+**This projects the burn rather than alarming on the counter.** Exhausting the budget is a slow,
 silent failure with a cliff at the end: nothing degrades, nothing pages, every call
 succeeds, and then the month rolls over and every Redis call starts erroring at once. By
 then the fix is a bill or an outage. A burn rate extrapolated from the counter is visible
@@ -113,7 +122,23 @@ def collect(text: str) -> tuple[dict[str, float], float]:
     return per_op, started_at
 
 
-def report(per_op: dict[str, float], started_at: float, budget: int, now: float) -> int:
+def merge(scrapes: list[tuple[dict[str, float], float]]) -> tuple[dict[str, float], float]:
+    """Fold several processes' scrapes into one budget view.
+
+    The budget is per-Upstash-database, but the counters are per-process: api, worker and
+    reconciler each hold their own prometheus_client registry, so scraping one of them
+    projects a third of the truth. Command totals sum across processes; the window is the
+    EARLIEST start time, because a counter that has been running longest bounds how far back
+    the summed total can be attributed — using the latest would inflate the rate.
+    """
+    per_op: dict[str, float] = {}
+    for scraped, _ in scrapes:
+        for op, value in scraped.items():
+            per_op[op] = per_op.get(op, 0.0) + value
+    return per_op, min(started for _, started in scrapes)
+
+
+def report(per_op: dict[str, float], started_at: float, budget: int, now: float, sources: int = 1) -> int:
     """Print the projection. Returns the process exit code."""
     elapsed_s = max(1.0, now - started_at)
     total = sum(per_op.values())
@@ -128,6 +153,17 @@ def report(per_op: dict[str, float], started_at: float, budget: int, now: float)
     print(f"rate            {rate:.4f} /s   (budget allows {budget / _MONTH_S:.4f} /s sustained)")
     print(f"projected       {projected:,.0f} / month")
     print(f"budget          {budget:,} / month")
+    print(f"sources         {sources} process(es) scraped")
+    if sources < 3:
+        # Be honest about what was measured. api/worker/reconciler each keep their own
+        # in-process registry (obs.py: one process per pod, no multiproc dir), so a
+        # single-endpoint run undercounts the shared Upstash budget by however many
+        # processes were left out. An optimistic verdict here is worse than no verdict.
+        print(
+            "                UNDERCOUNT: svcforge runs api + worker + reconciler, each with "
+            "its own\n                registry. Pass --url once per process for the real "
+            "total; the numbers\n                above cover only what was scraped."
+        )
 
     if total < 100:
         # Extrapolating a month from a handful of commands is astrology. Say so rather than
@@ -147,20 +183,32 @@ def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(
         description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
     )
-    parser.add_argument("--url", default="http://localhost:8000/metrics", help="Prometheus endpoint")
+    parser.add_argument(
+        "--url",
+        action="append",
+        dest="urls",
+        metavar="URL",
+        help="Prometheus endpoint; repeat once per process (api, worker, reconciler)",
+    )
     parser.add_argument("--budget", type=int, default=_FREE_TIER_BUDGET, help="commands per month")
     args = parser.parse_args(argv)
 
-    try:
-        per_op, started_at = collect(scrape(args.url))
-    except BudgetError as exc:
-        print(f"error: {exc}", file=sys.stderr)
-        return 2
-    except OSError as exc:
-        print(f"error: cannot scrape {args.url}: {exc}", file=sys.stderr)
-        return 2
+    # action="append" cannot carry a default (argparse appends to it), so apply it here.
+    urls: list[str] = args.urls or ["http://localhost:8000/metrics"]
 
-    return report(per_op, started_at, args.budget, time.time())
+    scrapes: list[tuple[dict[str, float], float]] = []
+    for url in urls:
+        try:
+            scrapes.append(collect(scrape(url)))
+        except BudgetError as exc:
+            print(f"error: {url}: {exc}", file=sys.stderr)
+            return 2
+        except OSError as exc:
+            print(f"error: cannot scrape {url}: {exc}", file=sys.stderr)
+            return 2
+
+    per_op, started_at = merge(scrapes)
+    return report(per_op, started_at, args.budget, time.time(), sources=len(scrapes))
 
 
 if __name__ == "__main__":