review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

libs/svcforge_core/svcforge_core/adapters/helm.py

@@ -32,6 +32,7 @@ import yaml
 from pydantic import BaseModel, ConfigDict, Field
 
 from svcforge_core.domain.models import CatalogEntry
+from svcforge_core.errors import SvcforgeError
 
 # How long the process group gets to honour SIGTERM before SIGKILL. Helm traps SIGTERM
 # and tries to leave the release in a coherent state; give it a moment to do so.
@@ -45,8 +46,12 @@ _RUN_TIMEOUT_MARGIN_S = 30
 _STDERR_TAIL_BYTES = 2048
 
 
-class HelmError(RuntimeError):
-    """Non-zero exit. str(self) is the stderr tail that lands in instances.error."""
+class HelmError(SvcforgeError, RuntimeError):
+    """Non-zero exit. str(self) is the stderr tail that lands in instances.error.
+
+    RuntimeError stays in the MRO so callers written against it keep catching; SvcforgeError
+    comes first so `except SvcforgeError` can separate a modelled failure from a stray bug.
+    """
 
 
 class ReleaseInfo(BaseModel):
@@ -166,6 +171,19 @@ class HelmProvisioner:
             argv += ["--kubeconfig", str(self._kubeconfig)]
         return argv
 
+    async def _run_helm(self, argv: Sequence[str]) -> str:
+        """`_run`, with the timeout path translated to this adapter's declared error type.
+
+        `_run` raises a bare TimeoutError so that the process-group test can assert on it
+        directly, but every public method here is documented as raising HelmError; a wedged
+        helm arriving as TimeoutError sails straight past a caller's `except HelmError` and
+        fails the task as an unmodelled crash. Translate once, at the public boundary.
+        """
+        try:
+            return await _run(argv, timeout_s=self._run_timeout_s)
+        except TimeoutError as exc:
+            raise HelmError(f"{argv[0]} timed out after {self._run_timeout_s}s and was killed") from exc
+
     async def install(self, release: str, ns: str, entry: CatalogEntry, values: dict[str, Any]) -> None:
         """helm upgrade --install --wait --timeout. Idempotent by construction."""
         # `upgrade --install` is why this is idempotent: a retried task after a crash mid-provision
@@ -190,7 +208,7 @@ class HelmProvisioner:
                 "--timeout",
                 f"{self._timeout_s}s",
             )
-            await _run(argv, timeout_s=self._run_timeout_s)
+            await self._run_helm(argv)
 
     async def uninstall(self, release: str, ns: str) -> None:
         """helm uninstall --wait. `--ignore-not-found` makes the retry of a half-done delete a no-op."""
@@ -204,12 +222,12 @@ class HelmProvisioner:
             "--timeout",
             f"{self._timeout_s}s",
         )
-        await _run(argv, timeout_s=self._run_timeout_s)
+        await self._run_helm(argv)
 
     async def list_releases(self) -> list[ReleaseInfo]:
         """Every release helm knows about, in every namespace. The reconciler's view of reality."""
         argv = self._base_argv("list", "--all-namespaces", "--output", "json")
-        raw = await _run(argv, timeout_s=self._run_timeout_s)
+        raw = await self._run_helm(argv)
         try:
             parsed: Any = json.loads(raw or "[]")
         except json.JSONDecodeError as exc:

libs/svcforge_core/svcforge_core/adapters/k8s.py

@@ -21,12 +21,17 @@ from typing import Any
 import yaml
 
 from svcforge_core.adapters.helm import HelmError, _run
+from svcforge_core.errors import SvcforgeError
 
 _KUBECTL_TIMEOUT_S = 60
 
 
-class K8sError(RuntimeError):
-    """kubectl failed. str(self) is the stderr tail, already truncated by `_run`."""
+class K8sError(SvcforgeError, RuntimeError):
+    """kubectl failed. str(self) is the stderr tail, already truncated by `_run`.
+
+    RuntimeError stays in the MRO so existing `except RuntimeError` callers keep catching;
+    SvcforgeError comes first so a modelled cluster failure is distinguishable from a bug.
+    """
 
 
 class SecretNotFound(K8sError):
@@ -112,6 +117,11 @@ class KubectlClient:
             return await _run(self._base_argv(*args), timeout_s=self._timeout_s)
         except HelmError as exc:
             raise K8sError(str(exc)) from exc
+        except TimeoutError as exc:
+            # `_run` re-raises the bare TimeoutError after killing the process group, so the
+            # timeout path does not come through HelmError. Without this clause a wedged
+            # kubectl surfaces as TimeoutError past a caller written to `except K8sError`.
+            raise K8sError(f"kubectl {args[0] if args else ''} timed out after {self._timeout_s}s") from exc
 
 
 class _ManifestFile:

libs/svcforge_core/svcforge_core/adapters/notify.py

@@ -11,15 +11,29 @@ tests and local dev get) and WebhookNotifier (the one that leaves the process).
 
 from __future__ import annotations
 
-import logging
 from typing import Protocol
 
 import httpx
 
-_log = logging.getLogger(__name__)
+from svcforge_core import obs
+
+# obs imports nothing from adapters (it is settings + structlog + otel only), so this is a
+# plain top-level import and not a lazy one — the cycle it would otherwise create does not
+# exist. Keep it that way: obs must stay importable before any adapter is.
 
 _DEFAULT_TIMEOUT_S = 5.0
 
+# Keys a caller's `fields` must not be allowed to occupy. `event` is structlog's first
+# positional parameter, so passing it as a kwarg is a TypeError, not a shadowed value; the
+# rest are stdlib LogRecord attributes that the ProcessorFormatter bridge refuses to
+# overwrite. A tenant-supplied field named `event` must not be able to crash the notifier.
+_RESERVED_KEYS = frozenset({"event", "msg", "args", "name", "levelname", "exc_info", "stack_info"})
+
+
+def _safe_fields(fields: dict[str, str] | None) -> dict[str, str]:
+    """Fields with reserved names prefixed rather than dropped — the value still gets logged."""
+    return {(f"field_{k}" if k in _RESERVED_KEYS else k): v for k, v in (fields or {}).items()}
+
 
 class Notifier(Protocol):
     """Implemented by LogNotifier, WebhookNotifier, and FakeNotifier (tests/fakes.py)."""
@@ -33,11 +47,20 @@ class LogNotifier:
     """Writes the event to the log. The default: structured logs are already shipped somewhere."""
 
     async def send(self, event: str, message: str, fields: dict[str, str] | None = None) -> None:
-        # NOT extra={"message": ...}. `message` is a reserved LogRecord attribute, and
-        # logging raises KeyError("Attempt to overwrite 'message' in LogRecord") at call
-        # time — so the notifier would crash the caller it was meant to inform. Same trap
-        # for `msg`, `args`, `name`, `levelname`, `exc_info`.
-        _log.info("notify", extra={"event": event, "detail": message, "fields": fields or {}})
+        # structlog kwargs, NOT logging's `extra=`. obs bridges stdlib records through
+        # ProcessorFormatter, which builds the event dict from `record.msg` alone — every
+        # key passed via `extra=` is dropped on the floor, so the default notifier used to
+        # emit a bare {"event": "notify"} with the payload gone.
+        #
+        # Fields are splatted rather than nested under "fields" so each one is its own
+        # queryable key in Loki. `detail`, not `message`: `message` is a reserved LogRecord
+        # attribute and the stdlib bridge raises KeyError on it.
+        #
+        # `notify_event`, not `event`: structlog's first positional parameter IS named
+        # `event` (it becomes the rendered line's "event" key, here the literal "notify"),
+        # so passing event= alongside it is a TypeError at the call, not a rename.
+        log = obs.get_logger(__name__)
+        log.info("notify", notify_event=event, detail=message, **_safe_fields(fields))
 
 
 class WebhookNotifier:
@@ -52,26 +75,33 @@ class WebhookNotifier:
         """
         self._url = url
         self._timeout_s = timeout_s
-        self._client = client
         self._owns_client = client is None
-
-    async def _get_client(self) -> httpx.AsyncClient:
-        if self._client is None:
-            self._client = httpx.AsyncClient(timeout=self._timeout_s)
-        return self._client
+        # Eager, not lazy. `AsyncClient()` does no I/O, so laziness bought nothing and cost a
+        # race: two concurrent `send`s could both see None, both construct a client, and the
+        # loser's connection pool would leak because only one of them survived the assignment.
+        self._client = client if client is not None else httpx.AsyncClient(timeout=self._timeout_s)
 
     async def send(self, event: str, message: str, fields: dict[str, str] | None = None) -> None:
         payload = {"event": event, "message": message, "fields": fields or {}}
         try:
-            client = await self._get_client()
-            resp = await client.post(self._url, json=payload, timeout=self._timeout_s)
+            resp = await self._client.post(self._url, json=payload, timeout=self._timeout_s)
             resp.raise_for_status()
-        except httpx.HTTPError as exc:
-            # Deliberately swallowed. See the module docstring: the work already succeeded.
-            _log.warning("notify webhook failed", extra={"event": event, "error": str(exc)})
+        except Exception as exc:  # the bare `except Exception` IS the specification here
+            # `send` must not raise; that is the contract in the module docstring, and it is
+            # not satisfiable by catching httpx.HTTPError alone. `httpx.InvalidURL` is not an
+            # HTTPError subclass, and posting on an already-aclose()d client raises
+            # RuntimeError — so a typo'd webhook URL would fail a task whose helm work has
+            # already succeeded. exc_info so the traceback survives the swallowing.
+            obs.get_logger(__name__).warning(
+                "notify webhook failed", notify_event=event, error=str(exc), exc_info=exc
+            )
 
     async def aclose(self) -> None:
-        """Close the client, if we made it."""
-        if self._client is not None and self._owns_client:
+        """Close the client, if we made it. Call at process shutdown, next to the pool's close.
+
+        The client reference is kept rather than cleared: a `send` that races shutdown now
+        raises RuntimeError on a closed client, and `send` swallows and logs that like any
+        other delivery failure instead of resurrecting a pool nobody will close.
+        """
+        if self._owns_client:
             await self._client.aclose()
-            self._client = None

libs/svcforge_core/svcforge_core/adapters/redis.py

@@ -5,9 +5,8 @@ holds the instances, the tasks, the leases and the `release_name` UNIQUE constra
 holds a counter, a claim marker and a copy — all of it rebuildable by doing nothing and
 waiting for a TTL.
 
-That framing is not philosophy, it decides the error handling, and the error handling is
-the module. Each class below catches `RedisError` and returns a *safe* answer rather than
-raising:
+That framing decides the error handling, and the error handling is the module. Each class
+below catches `RedisError` and returns a *safe* answer rather than raising:
 
 | Path        | Redis is down            | Why                                              |
 |-------------|--------------------------|--------------------------------------------------|
@@ -22,7 +21,7 @@ Consequently nothing here raises out to a caller, and `/readyz` stays Postgres-o
 Redis outage must not make a single pod unready — that would convert "the cache is down"
 into "the platform is down", which is the exact inversion this module exists to prevent.
 
-**The budget is a design constraint, not a footnote.** Upstash free tier:
+**The budget is a design constraint.** Upstash free tier:
 
     500,000 commands / month = 16,129 / day = 11 / minute = 0.19 / second, sustained
 

libs/svcforge_core/svcforge_core/errors.py

@@ -0,0 +1,21 @@
+"""The one base class every svcforge-raised exception shares.
+
+Without it, a caller that wants "the cluster failed" has to write `except Exception`, which
+also swallows the `AttributeError` from a typo three frames down. The two are not the same
+incident: one is retried, the other is a bug that must reach the dead-letter loudly. A single
+root makes that distinction expressible in one clause.
+
+Subclasses keep their existing stdlib base as well (`HelmError(SvcforgeError, RuntimeError)`),
+so code already written against `except RuntimeError` keeps working. The MRO order matters:
+`SvcforgeError` first, so the svcforge-specific class is the more derived one.
+"""
+
+from __future__ import annotations
+
+
+class SvcforgeError(Exception):
+    """Root of every exception svcforge raises on purpose.
+
+    Catch this to mean "an operation svcforge models failed". Anything not deriving from it
+    that escapes a handler is, by definition, a programming error rather than a modelled one.
+    """

libs/svcforge_core/svcforge_core/migrate.py

@@ -13,6 +13,7 @@ Run:  python -m svcforge_core.migrate
 
 from __future__ import annotations
 
+import os
 import sys
 from pathlib import Path
 
@@ -20,7 +21,30 @@ import psycopg
 
 from svcforge_core.settings import load_settings
 
-MIGRATIONS_DIR = Path(__file__).resolve().parents[3] / "migrations"
+
+def _migrations_dir() -> Path:
+    """Where the .sql files live, in a way that survives being installed as a wheel.
+
+    `Path(__file__).parents[3] / "migrations"` works only for the editable dev install,
+    where the package really does sit three levels under the repo root. The images install
+    a built wheel into site-packages (deliberately — an editable install in an image ships
+    a path, not a package), so the same expression resolves to
+    `/app/.venv/lib/migrations`, which does not exist. The migration Job then finds zero
+    files and exits 1, and the Helm pre-upgrade hook fails on every single sync.
+
+    That failure is invisible in dev and in CI, because both run editable. It only appears
+    the first time you deploy — which is the worst time to discover it.
+
+    So: the image sets SVCFORGE_MIGRATIONS_DIR and copies the directory in; the repo
+    checkout falls back to the path relative to this file.
+    """
+    env = os.getenv("SVCFORGE_MIGRATIONS_DIR")
+    if env:
+        return Path(env)
+    return Path(__file__).resolve().parents[3] / "migrations"
+
+
+MIGRATIONS_DIR = _migrations_dir()
 
 # Advisory lock: two Jobs racing (a retried hook, a hand-run) must not both apply DDL.
 # Session-scoped, so this needs the session pooler (5432), not pgbouncer (6543).

libs/svcforge_core/svcforge_core/obs.py

@@ -62,9 +62,17 @@ TASKS_CLAIMED = Counter(
     ["kind"],
 )
 
-TASKS_FAILED = Counter(
-    "svcforge_tasks_failed_total",
-    "Tasks that exhausted their attempts and went to 'failed'.",
+TASK_ATTEMPTS_FAILED = Counter(
+    "svcforge_task_attempts_failed_total",
+    "Task ATTEMPTS that raised. Not the same as tasks that dead-lettered: one task that "
+    "succeeds on its third try increments this twice. Alert on increase(), not on the raw "
+    "total, or ordinary transient retries page you.",
+    ["kind"],
+)
+
+TASKS_DEAD_LETTERED = Counter(
+    "svcforge_tasks_dead_lettered_total",
+    "Tasks that exhausted max_attempts and went to 'failed'. These need a human.",
     ["kind"],
 )
 
@@ -73,7 +81,7 @@ PROVISION_TIME = Histogram(
     "Wall time of a provision task, claim to terminal report.",
     # NOT the defaults. See the module docstring: the defaults end at 10s and a provision
     # takes minutes. The top finite bucket is 1800 because helm's own --timeout is 600 and
-    # a provision past thirty minutes is not slow, it is broken and belongs in +Inf.
+    # a provision past thirty minutes is broken and belongs in +Inf.
     buckets=(10, 30, 60, 120, 300, 600, 1800, float("inf")),
 )
 
@@ -97,6 +105,9 @@ RECONCILER_LAST_TICK = Gauge(
 
 _TRACER_NAME = "svcforge"
 
+# Set by setup(); re-bound by bind_task_context after it clears the context.
+_service_name: str = "svcforge"
+
 # setup() is idempotent because it is called from three entrypoints and from tests, and
 # because configuring structlog twice silently discards the first configuration while
 # adding a second stdout handler to the root logger — every line then prints twice.
@@ -194,6 +205,11 @@ def _setup_logging(service_name: str, settings: Settings) -> None:
     root.handlers = [handler]
     root.setLevel(level)
 
+    # Remembered so bind_task_context can restore it after clearing. Without this, every
+    # log line emitted inside a task loses `service`, and those are exactly the lines you
+    # filter on when you are trying to tell worker output from reconciler output.
+    global _service_name
+    _service_name = service_name
     structlog.contextvars.bind_contextvars(service=service_name)
 
 
@@ -278,6 +294,9 @@ def bind_task_context(instance_id: UUID, task_id: int, team: str) -> None:
     """
     structlog.contextvars.clear_contextvars()
     structlog.contextvars.bind_contextvars(
+        # `service` is re-bound because the clear above took it with it. It is set once in
+        # setup() and is not per-task, but clear_contextvars() is indiscriminate.
+        service=_service_name,
         instance_id=str(instance_id),
         task_id=task_id,
         team=team,

libs/svcforge_core/svcforge_core/repo/instances.py

@@ -129,14 +129,6 @@ class InstanceRepo:
             )
             return cur.rowcount == 1
 
-    async def set_error(self, id: UUID, error: str) -> None:
-        """Record a terminal failure message. Used when a task exhausts its attempts."""
-        async with self._pool.connection() as conn, conn.cursor() as cur:
-            await cur.execute(
-                "update instances set error = %s, state = %s, updated_at = now() where id = %s",
-                (error[-2000:], InstanceState.FAILED.value, id),
-            )
-
     async def list_upgradable(
         self,
         service_type: str,
@@ -153,8 +145,8 @@ class InstanceRepo:
         rollouts table, no state machine, no pause/resume CLI. You clear it with SQL.
 
         `order by team = %(own_team)s desc` — your own instances upgrade first, so you are
-        the tenant who finds out the chart is broken. Eating your own dog food is a
-        `order by`, not a policy document.
+        the tenant who finds out the chart is broken. Eating your own dog food is enforced
+        by an `order by` rather than a policy document.
 
         `limit %(max_in_flight)s` — a config value, not a scheduler. It stays at 1 until 1
         is too slow, and 1 is what makes the halt meaningful: the fleet stops after the

libs/svcforge_core/svcforge_core/repo/reconcile.py

@@ -93,7 +93,7 @@ class ReconcileRepo:
 
         None if the row moved, or if a provision is already outstanding.
 
-        The two-hop state change is the interesting part, and it is not busywork:
+        The two-hop state change is the interesting part:
 
         * `LEGAL` has no `ready -> provisioning` edge. The tenant-visible lifecycle only
           leaves `ready` through `deleting` or `failed`, and drift is a failure — the

libs/svcforge_core/svcforge_core/repo/tasks.py

@@ -19,7 +19,7 @@ from psycopg import AsyncConnection
 from svcforge_core.domain.backoff import next_attempt_at
 from svcforge_core.domain.models import Task, TaskKind
 from svcforge_core.domain.states import LEGAL, InstanceState
-from svcforge_core.obs import inject_traceparent
+from svcforge_core.obs import TASKS_DEAD_LETTERED, inject_traceparent
 from svcforge_core.repo.db import DictPool
 
 # Which states may legally become `failed`, derived from the domain's own table rather
@@ -49,6 +49,13 @@ _CAN_FAIL: Final[tuple[str, ...]] = tuple(
 # before it has loaded anything. A data-modifying CTE runs exactly once and cannot claim
 # twice. The alternative — a second SELECT for the team — would be a second round trip per
 # task to fetch a column the database already had in hand.
+#
+# LEFT join, not inner. The UPDATE inside the CTE has already taken effect by the time the
+# outer select runs, so an inner join that matches nothing would return no row — and
+# `claim()` would report "queue empty" for a task it had just marked `running`, stranding
+# it until the lease expires and silently burning an attempt. The FK cascade makes that
+# nearly impossible in practice; "nearly" is not a reason to leave a silent failure in the
+# one query the whole system depends on. `Task.team` is already `str | None`.
 _CLAIM_SQL = """
 with claimed as (
   update tasks set state='running', attempts=attempts+1, locked_by=%(worker)s, locked_at=now()
@@ -62,7 +69,7 @@ with claimed as (
   returning *
 )
 select claimed.*, instances.team
-  from claimed join instances on instances.id = claimed.instance_id;
+  from claimed left join instances on instances.id = claimed.instance_id;
 """
 
 
@@ -135,24 +142,33 @@ class TaskRepo:
     async def complete(
         self,
         task_id: int,
+        worker_id: str,
         conn: AsyncConnection[dict[str, Any]] | None = None,
-    ) -> None:
-        """Mark done. Pass `conn` to commit alongside the caller's instance update.
+    ) -> bool:
+        """Mark done, but only if this worker still owns the task. False if it does not.
 
-        Completing the task and recording what it accomplished belong in one transaction:
-        commit them separately and a crash in between leaves a task marked done whose
-        work never landed, or work that landed and will be redone.
+        Pass `conn` to commit alongside the caller's instance update: completing the task
+        and recording what it accomplished belong in one transaction, or a crash between
+        them leaves a task marked done whose work never landed.
+
+        `and state='running' and locked_by=%s` is not defensive padding — without it this
+        is a lost-update bug with a real trigger. A worker that hangs past `lease_seconds`
+        has its task requeued by the reconciler and re-claimed by someone else. When the
+        hung worker finally returns, an unconditional UPDATE here marks the task `done`
+        while the new owner is still running it, and its work goes unaccounted for. The
+        loser gets False and must treat it as "someone else owns this now", not an error.
         """
-        sql = "update tasks set state='done', locked_by=null where id = %s"
+        sql = "update tasks set state='done', locked_by=null where id=%s and state='running' and locked_by=%s"
         if conn is not None:
             async with conn.cursor() as cur:
-                await cur.execute(sql, (task_id,))
-            return
+                await cur.execute(sql, (task_id, worker_id))
+                return cur.rowcount == 1
         async with self._pool.connection() as own, own.cursor() as cur:
-            await cur.execute(sql, (task_id,))
+            await cur.execute(sql, (task_id, worker_id))
+            return cur.rowcount == 1
 
-    async def fail(self, task_id: int, err: str, max_attempts: int = 5) -> None:
-        """Retry with backoff, or give up.
+    async def fail(self, task_id: int, err: str, worker_id: str, max_attempts: int = 5) -> bool:
+        """Retry with backoff, or give up. False if this worker no longer owns the task.
 
         Under max_attempts: back to 'queued' with run_after pushed out by exponential
         backoff with full jitter. Jitter matters — a cluster-wide outage fails every task
@@ -161,17 +177,25 @@ class TaskRepo:
         At max_attempts: 'failed', and the error is copied onto the instance so the tenant
         can see it. A dead-letter state, not an infinite retry: a task that cannot succeed
         must stop and become someone's problem.
+
+        The ownership check in the SELECT is the same lost-lease guard as `complete`, and
+        it matters more here: a stale worker reporting failure would push a task the new
+        owner is actively running back to `queued`, letting a *third* worker claim it.
         """
         now = datetime.now(UTC)
         async with self._pool.connection() as conn:
             async with conn.transaction(), conn.cursor() as cur:
                 await cur.execute(
-                    "select attempts, instance_id from tasks where id = %s for update",
-                    (task_id,),
+                    """select attempts, instance_id, kind from tasks
+                        where id=%s and state='running' and locked_by=%s
+                        for update""",
+                    (task_id, worker_id),
                 )
                 row = await cur.fetchone()
                 if row is None:
-                    return
+                    # Either the task is gone, or the lease was stolen. Both mean: not ours
+                    # to report on. Writing anything here would corrupt the new owner's run.
+                    return False
                 attempts = int(row["attempts"])
                 instance_id = row["instance_id"]
 
@@ -183,7 +207,7 @@ class TaskRepo:
                             where id = %s""",
                         (err[-2000:], next_attempt_at(attempts - 1, now=now), task_id),
                     )
-                    return
+                    return True
 
                 await cur.execute(
                     """update tasks
@@ -199,14 +223,18 @@ class TaskRepo:
                         where id=%s and state = any(%s)""",
                     (err[-2000:], InstanceState.FAILED.value, instance_id, list(_CAN_FAIL)),
                 )
+                # Counted here, not in the worker: this is the only place that knows the
+                # difference between "attempt 2 of 5 failed" and "this task is done trying".
+                TASKS_DEAD_LETTERED.labels(kind=str(row["kind"])).inc()
+                return True
 
     async def reset_expired_leases(self, lease_seconds: int) -> int:
         """Return tasks whose worker died back to the queue. Called by the reconciler.
 
         No distributed lock survives a power cut. A worker that is SIGKILLed leaves
         `state='running'` and `locked_by` set with nobody running it, and that row would
-        sit there forever. The lease is the only thing that recovers it: not a lock, a
-        timeout. This is why `locked_at` exists.
+        sit there forever. The lease is the only thing that recovers it, which is why
+        `locked_at` exists.
         """
         async with self._pool.connection() as conn, conn.cursor() as cur:
             await cur.execute(

libs/svcforge_core/svcforge_core/settings.py

@@ -23,6 +23,11 @@ class Settings(BaseSettings):
         frozen=True,
     )
 
+    # Anything other than "local" makes check_production() enforce. The chart sets it;
+    # a laptop does not. Defaulting to "local" means a forgotten env var costs you a
+    # refused dev shortcut, never an unauthenticated production API.
+    environment: str = "local"
+
     # --- Postgres -----------------------------------------------------------------
     # Transaction pooler (6543 on Supabase). Everything the services do at runtime.
     pg_dsn: PostgresDsn
@@ -35,6 +40,9 @@ class Settings(BaseSettings):
 
     # --- Redis (derived state only; never the source of truth) ---------------------
     redis_dsn: RedisDsn | None = None
+    # Per-team request budget. Generous on purpose: this exists to stop one team's runaway
+    # script from starving the others, not to meter usage.
+    rate_limit_per_minute: int = Field(default=60, ge=1)
 
     # --- API ----------------------------------------------------------------------
     jwks_url: str | None = None
@@ -81,9 +89,22 @@ class Settings(BaseSettings):
         return str(self.pg_dsn_session or self.pg_dsn)
 
     def check_production(self) -> None:
-        """Refuse the dev escape hatches when they would matter."""
+        """Refuse the dev escape hatches outside local development. Call at startup.
+
+        This is a no-op unless `SVCFORGE_ENVIRONMENT` says otherwise, which is what makes
+        it safe to call unconditionally from every entrypoint — and calling it
+        unconditionally is the point. The previous version could only be invoked from a
+        branch that already knew it was production, so no such branch was ever written and
+        the check never ran: `SVCFORGE_AUTH_DISABLED=true` in prod would have started the
+        API with JWT verification off, serving every unauthenticated request as team
+        `platform`, silently.
+        """
+        if self.environment == "local":
+            return
         if self.auth_disabled:
-            raise ValueError("SVCFORGE_AUTH_DISABLED=true is refused outside local development")
+            raise ValueError(
+                f"SVCFORGE_AUTH_DISABLED=true is refused when SVCFORGE_ENVIRONMENT={self.environment!r}"
+            )
 
 
 def load_settings() -> Settings: