review: fix 26 findings from a 4-agent audit

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.

services/api/Dockerfile

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 COPY libs/ libs/
 COPY services/api/ services/api/
 COPY catalog.yaml ./
+# The migrate Job runs from THIS image (migrate-job.yaml pins the api digest), so the SQL
+# has to be in it. svcforge_core.migrate's fallback resolves MIGRATIONS_DIR relative to
+# its own __file__, which lands under site-packages here — a directory that does not and
+# should not contain SQL — so main() returned 1 and the pre-install/pre-upgrade hook
+# failed every sync. SVCFORGE_MIGRATIONS_DIR below points it at this copy instead.
+COPY migrations/ migrations/
 # --no-editable is what turns svcforge-core into a real wheel in site-packages.
 # pyproject.toml declares it `editable = true` for local dev; an editable install in an
 # image points at /app/libs, which is a source tree that need not survive the final stage.
@@ -50,6 +56,7 @@ WORKDIR /app
 COPY --from=builder --chown=10001:10001 /app /app
 ENV PATH="/app/.venv/bin:$PATH" \
     PYTHONUNBUFFERED=1 \
-    PYTHONDONTWRITEBYTECODE=1
+    PYTHONDONTWRITEBYTECODE=1 \
+    SVCFORGE_MIGRATIONS_DIR=/app/migrations
 USER 10001
 ENTRYPOINT ["python", "-m", "services.api"]

services/api/deps.py

@@ -15,6 +15,7 @@ from fastapi import Depends, HTTPException, Request, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from jwt import PyJWKClient
 
+from svcforge_core.adapters.redis import RateLimiterProto
 from svcforge_core.domain.models import CatalogEntry
 from svcforge_core.repo.db import DictPool
 from svcforge_core.repo.instances import InstanceRepo
@@ -146,13 +147,35 @@ async def _signing_key(client: PyJWKClient, token: str) -> jwt.PyJWK:
     return await asyncio.to_thread(client.get_signing_key_from_jwt, token)
 
 
-async def rate_limit(team: Annotated[str, Depends(get_current_team)]) -> None:
-    """Per-team rate limiting. Seam only — Module 10 fills this in (Redis, Lua, token bucket).
+def get_rate_limiter(request: Request) -> RateLimiterProto | None:
+    """The limiter lifespan built, or None when no Redis is configured."""
+    limiter: RateLimiterProto | None = getattr(request.app.state, "rate_limiter", None)
+    return limiter
 
-    It exists now, wired into the routes, so that turning it on is an edit to one function
-    body rather than a change to every handler signature.
+
+async def rate_limit(
+    request: Request,
+    team: Annotated[str, Depends(get_current_team)],
+) -> None:
+    """Per-team rate limiting. One Redis command per check, and it fails OPEN.
+
+    Failing open is the entire policy. Redis holds derived state; losing it must degrade
+    the platform, never stop it. A limiter that fails closed converts a cache outage into
+    a total outage, which is a strictly worse incident than the burst it was protecting
+    against — so `RateLimiter.check` swallows its own errors and returns `allowed=True`.
+    The 429 below therefore only ever comes from a real, counted overage.
     """
-    return None
+    limiter = get_rate_limiter(request)
+    if limiter is None:
+        return
+
+    result = await limiter.check(team)
+    if not result.allowed:
+        raise HTTPException(
+            status_code=status.HTTP_429_TOO_MANY_REQUESTS,
+            detail={"code": "rate_limited", "message": "too many requests"},
+            headers={"Retry-After": str(result.retry_after_s)},
+        )
 
 
 async def idempotency_key(request: Request) -> str | None:

services/api/main.py

@@ -8,7 +8,6 @@ environment at import time — before any fixture can say otherwise.
 from __future__ import annotations
 
 import asyncio
-import logging
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 
@@ -18,11 +17,13 @@ from jwt import PyJWKClient
 
 from services.api.models import ErrorBody
 from services.api.routes import health, instances
+from svcforge_core import obs
+from svcforge_core.adapters.redis import RateLimiter, make_redis
 from svcforge_core.domain.catalog import load_catalog
 from svcforge_core.repo.db import make_pool
 from svcforge_core.settings import Settings, load_settings
 
-log = logging.getLogger(__name__)
+log = obs.get_logger("svcforge.api")
 
 
 @asynccontextmanager
@@ -42,6 +43,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
 
     app.state.catalog = load_catalog(settings.catalog_path)
 
+    # Redis is optional by construction. `make_redis` returns None when no DSN is set, and
+    # every consumer treats None as "skip" — so a deployment without Redis loses rate
+    # limiting and keeps everything else. Built here rather than per request because a
+    # connection pool per request is a connection pool per request.
+    redis = make_redis(settings)
+    app.state.redis = redis
+    app.state.rate_limiter = (
+        RateLimiter(redis, limit=settings.rate_limit_per_minute, window_s=60) if redis is not None else None
+    )
+
     pool = make_pool(str(settings.pg_dsn), settings.pool_min_size, settings.pool_max_size)
     # wait=True fails NOW, loudly, if the DSN is wrong — instead of at the first request,
     # as a PoolTimeout, in front of a user.
@@ -68,6 +79,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
         yield
     finally:
         await pool.close()
+        if redis is not None:
+            await redis.aclose()
 
 
 async def _http_exception_handler(request: Request, exc: Exception) -> JSONResponse:
@@ -93,6 +106,18 @@ def create_app(settings: Settings | None = None) -> FastAPI:
     """App factory: lifespan, routers, exception handler, /metrics."""
     settings = settings or load_settings()
 
+    # FIRST, before any router is built and before any logger is bound. Without this the
+    # API is the one service of three that never configures structlog: its lines go out
+    # through logging.lastResort as bare text on stderr with no service, no trace_id and
+    # no JSON envelope — a parse failure in the collector, and unattributable in Loki.
+    # `settings.log_json` was silently inert here for the same reason.
+    obs.setup("svcforge-api", settings)
+
+    # Refuse the dev escape hatches when SVCFORGE_ENVIRONMENT says this is not a laptop.
+    # Called unconditionally and early: a check that only runs from a branch someone
+    # remembered to write is a check that does not run.
+    settings.check_production()
+
     app = FastAPI(
         title="svcforge",
         version="0.1.0",
@@ -115,7 +140,7 @@ def app() -> FastAPI:
     return create_app()
 
 
-if __name__ == "__main__":  # pragma: no cover
-    import uvicorn
-
-    uvicorn.run("services.api.main:app", factory=True, host="0.0.0.0", port=8000)  # noqa: S104
+# There is deliberately no `if __name__ == "__main__"` here. `services/api/__main__.py` is
+# the single entrypoint, and the image's ENTRYPOINT uses it. A second one in this module
+# drifted from it — different log_level, different access_log — so `python -m services.api`
+# and `python services/api/main.py` started the same app two different ways.

services/api/routes/health.py

@@ -1,7 +1,7 @@
 """Liveness, readiness, metrics.
 
-The distinction between the first two is not pedantry, it is the difference between a
-30-second blip and a fleet-wide outage:
+The distinction between the first two is the difference between a 30-second blip and a
+fleet-wide outage:
 
 * `/healthz` (liveness) answers "is this process wedged?" A failure here gets the
   container KILLED. It must therefore touch NOTHING external. Wire it to the DB and a

services/reconciler/Dockerfile

@@ -41,7 +41,10 @@ RUN useradd -u 10001 -m -s /usr/sbin/nologin svcforge
 WORKDIR /app
 COPY --from=builder --chown=10001:10001 /app /app
 
-COPY --from=alpine/helm:3.16.2@sha256:a19a2968fd672336d39771f6c899781424d725229148656dbc2a1e305003cdec /usr/bin/helm /usr/local/bin/helm
+# helm 3.21.3, not 3.16.2 — see services/worker/Dockerfile. 3.16.2 is a Go 1.22.9 build
+# carrying CRITICAL CVE-2025-68121 (crypto/tls) and CVE-2026-33186 (grpc) and HIGH
+# CVE-2026-35469 (spdystream). Kept on 3.x on purpose: helm 4 is a breaking change.
+COPY --from=alpine/helm:3.21.3@sha256:35da09ba0716fc7c3cd63b6b31ee380a9c7662e95f29ab0e4ae962420afd315b /usr/bin/helm /usr/local/bin/helm
 
 ENV PATH="/app/.venv/bin:$PATH" \
     PYTHONUNBUFFERED=1 \

services/reconciler/main.py

@@ -81,8 +81,8 @@ class ReconcilerDeps:
     clock: Clock
     catalog: dict[str, CatalogEntry]
     settings: Settings
-    # Whose instances go first in the day-2 work list. Eating your own dog food is an
-    # `order by`, not a policy document — see InstanceRepo.list_upgradable.
+    # Whose instances go first in the day-2 work list. Eating your own dog food is enforced
+    # by an `order by` rather than a policy document — see InstanceRepo.list_upgradable.
     own_team: str
     # A config value, not a scheduler. Leave it at 1 until 1 is too slow.
     max_in_flight: int
@@ -152,15 +152,15 @@ async def check_drift(deps: ReconcilerDeps) -> None:
 async def check_lease_expiry(deps: ReconcilerDeps) -> None:
     """Tasks whose worker died -> back to `queued`.
 
-    A lease, not a lock. No lock survives a power cut: a worker SIGKILLed mid-provision
+    A lease. No lock survives a power cut: a worker SIGKILLed mid-provision
     leaves `state='running'` with `locked_by` set and nobody running it, and no amount of
     cleanup code in the worker helps, because the worker is the part that died. `locked_at`
     plus a timeout is the only thing that recovers the row, which is why `locked_at` exists.
 
-    The 5-minute default is not arbitrary: it must exceed the longest a healthy task can
-    hold a lease, or the reconciler hands a still-running provision to a second worker.
-    Handlers are idempotent, so that is survivable rather than fatal — but survivable is
-    not free, and `lease_seconds` sits above helm's `--timeout` for that reason.
+    The 5-minute default must exceed the longest a healthy task can hold a lease, or the
+    reconciler hands a still-running provision to a second worker. Handlers are idempotent,
+    so that is survivable, though it still costs a duplicated helm run — which is why
+    `lease_seconds` sits above helm's `--timeout`.
     """
     freed = await deps.tasks.reset_expired_leases(deps.settings.lease_seconds)
     if freed:

services/worker/Dockerfile

@@ -41,8 +41,21 @@ RUN useradd -u 10001 -m -s /usr/sbin/nologin svcforge
 WORKDIR /app
 COPY --from=builder --chown=10001:10001 /app /app
 
-COPY --from=alpine/helm:3.16.2@sha256:a19a2968fd672336d39771f6c899781424d725229148656dbc2a1e305003cdec /usr/bin/helm /usr/local/bin/helm
-COPY --from=bitnamilegacy/kubectl:1.31.2@sha256:0eab9ec8f5e0f75271277467ebb7513b36dea0122bc68615d18c47fece4fd82c /opt/bitnami/kubectl/bin/kubectl /usr/local/bin/kubectl
+# helm 3.21.3, not 3.16.2. 3.16.2 was built with Go 1.22.9 and carries CRITICAL
+# CVE-2025-68121 (crypto/tls) and CVE-2026-33186 (grpc), plus HIGH CVE-2026-35469
+# (spdystream, fixed in 0.5.1) — trivy fails the build on them and is right to.
+# Deliberately 3.x: helm 4 is a breaking change and is not a CVE fix.
+COPY --from=alpine/helm:3.21.3@sha256:35da09ba0716fc7c3cd63b6b31ee380a9c7662e95f29ab0e4ae962420afd315b /usr/bin/helm /usr/local/bin/helm
+# kubectl 1.35.3 from the OFFICIAL registry.k8s.io image, replacing
+# bitnamilegacy/kubectl:1.31.2. Two reasons, either sufficient:
+#   - CVEs: the bitnami image is Go 1.22.9 and ships containerd < 1.7.29 (HIGH
+#     CVE-2024-25621) along with the same crypto/tls and grpc CRITICALs as helm above.
+#   - Skew: the target cluster runs v1.35.3. 1.31.2 is four minors behind, well outside
+#     kubectl's supported +/-1 window, so it was unsupported against this cluster.
+# bitnamilegacy publishes no 1.35 tag. registry.k8s.io/kubectl is the upstream-maintained
+# image, is a manifest list with linux/arm64 (this cluster is Ampere), and puts the binary
+# at /bin/kubectl rather than bitnami's /opt/bitnami path.
+COPY --from=registry.k8s.io/kubectl:v1.35.3@sha256:8dad99b604a2c0bafe17f53cadf78482d6f667a6da687f385508f5f4e4696d37 /bin/kubectl /usr/local/bin/kubectl
 
 ENV PATH="/app/.venv/bin:$PATH" \
     PYTHONUNBUFFERED=1 \

services/worker/deps.py

@@ -8,6 +8,7 @@ run in milliseconds against a FakeProvisioner instead of needing a cluster.
 from __future__ import annotations
 
 from dataclasses import dataclass
+from typing import Protocol
 
 from svcforge_core.adapters.clock import Clock
 from svcforge_core.adapters.helm import Provisioner
@@ -19,6 +20,20 @@ from svcforge_core.repo.tasks import TaskRepo
 from svcforge_core.settings import Settings
 
 
+class NamespaceEnsurer(Protocol):
+    """The one thing the worker needs from kubectl.
+
+    Narrower than `KubectlClient` on purpose: the handler creates namespaces and does
+    nothing else with the cluster, so that is the whole interface. Satisfied by
+    `KubectlClient` in production and by a fake in tests, which is what keeps the worker
+    suite running in milliseconds with no cluster.
+    """
+
+    async def ensure_namespace(self, ns: str, labels: dict[str, str] | None = None) -> None:
+        """Create the namespace if absent. Idempotent — safe on every retry."""
+        ...
+
+
 @dataclass(frozen=True)
 class WorkerDeps:
     """Everything a handler is allowed to touch."""
@@ -27,6 +42,10 @@ class WorkerDeps:
     instances: InstanceRepo
     tasks: TaskRepo
     provisioner: Provisioner
+    # Creates the tenant namespace before helm is pointed at it. `helm --namespace X`
+    # does NOT create X, so without this the very first provision for a new team fails
+    # with "namespace not found" — the one path that is guaranteed untested by a fake.
+    namespaces: NamespaceEnsurer
     notifier: Notifier
     clock: Clock
     catalog: dict[str, CatalogEntry]

services/worker/handlers.py

@@ -2,7 +2,7 @@
 
 Every handler here obeys one rule: running it twice must equal running it once.
 
-That is not a nicety. A worker can be SIGKILLed after helm has installed the release but
+A worker can be SIGKILLed after helm has installed the release but
 before the DB row says so; the lease expires; another worker claims the same task and runs
 this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
 and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
@@ -63,6 +63,15 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
     # below is idempotent either way, so this is bookkeeping, not a lock.
     await deps.instances.update_state(inst.id, InstanceState.REQUESTED, InstanceState.PROVISIONING)
 
+    # `helm --namespace X` does not create X. Every tenant's first provision lands in a
+    # namespace that does not exist yet, so this has to happen before helm runs or the
+    # install fails with "namespaces not found". Idempotent (kubectl apply of a Namespace
+    # manifest), so it costs one no-op API call on every subsequent provision.
+    await deps.namespaces.ensure_namespace(
+        inst.namespace,
+        labels={"svcforge.io/team": inst.team},
+    )
+
     await deps.provisioner.install(
         release=inst.release_name,
         ns=inst.namespace,
@@ -92,7 +101,15 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
     # `helm uninstall` of an already-gone release is not an error to us: the adapter
     # swallows not-found, because the desired state — no release — is already true.
     await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
-    await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED)
+
+    # Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
+    # the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
+    # and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
+    # asked to delete. Failing loudly turns a silent ping-pong into one visible error.
+    if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
+        raise HandlerError(
+            f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
+        )
 
 
 async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
@@ -138,18 +155,27 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
     if inst.release_name in releases:
         return
 
+    # `returning` + a `where` on the update half tells us whether THIS call was the one
+    # that halted the rollout. The halt itself is idempotent; the page is not. Without the
+    # distinction, a verify that fails its full retry budget sends five identical
+    # notifications for one incident, spread across the backoff curve.
     async with deps.pool.connection() as conn, conn.cursor() as cur:
         await cur.execute(
             """insert into catalog_versions (service_type, rollout_state)
                     values (%s, 'halted')
-               on conflict (service_type) do update set rollout_state = 'halted'""",
+               on conflict (service_type) do update set rollout_state = 'halted'
+                     where catalog_versions.rollout_state <> 'halted'
+               returning service_type""",
             (inst.service_type,),
         )
-    await deps.notifier.send(
-        "rollout.halted",
-        f"rollout halted for {inst.service_type}: {inst.release_name} failed verify",
-        {"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
-    )
+        newly_halted = await cur.fetchone() is not None
+
+    if newly_halted:
+        await deps.notifier.send(
+            "rollout.halted",
+            f"rollout halted for {inst.service_type}: {inst.release_name} failed verify",
+            {"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
+        )
     raise HandlerError(f"verify failed for {inst.release_name}; rollout halted")
 
 

services/worker/main.py

@@ -13,6 +13,7 @@ import asyncio
 import contextlib
 import signal
 import time
+from collections.abc import Awaitable
 
 from opentelemetry import trace
 
@@ -21,9 +22,10 @@ from services.worker.handlers import HANDLERS
 from svcforge_core import obs
 from svcforge_core.adapters.clock import SystemClock
 from svcforge_core.adapters.helm import HelmProvisioner
+from svcforge_core.adapters.k8s import KubectlClient
 from svcforge_core.adapters.notify import LogNotifier
 from svcforge_core.domain.catalog import load_catalog
-from svcforge_core.domain.models import Task
+from svcforge_core.domain.models import Task, TaskKind
 from svcforge_core.repo.db import make_pool
 from svcforge_core.repo.instances import InstanceRepo
 from svcforge_core.repo.tasks import TaskRepo
@@ -42,8 +44,28 @@ async def _sleep_or_stop(stop: asyncio.Event, seconds: float) -> None:
         await asyncio.wait_for(stop.wait(), timeout=seconds)
 
 
+async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
+    """Run a terminal report, and never let its failure escape.
+
+    Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
+    TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
+    DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
+    not just this one. The task itself is safe either way: it stays `running` and the
+    reconciler's lease sweep returns it to the queue. Losing the report costs one lease
+    interval; losing the siblings costs their work.
+    """
+    try:
+        if not await coro:
+            # The lease was stolen while we were working: another worker owns this task
+            # now and is mid-run. Reporting is theirs to do, not ours.
+            log.warning("lease lost before report; another worker owns this task", task_id=task_id)
+    except Exception:
+        log.exception("could not report task %s (%s); lease will expire", task_id, what)
+
+
 async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None:
     """Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
+    worker_id = deps.settings.worker_id
     try:
         # Every log line from here carries instance_id/task_id/team. Bound once, at claim,
         # rather than passed down: the alternative is threading three arguments through
@@ -55,7 +77,11 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
 
         handler = HANDLERS.get(task.kind)
         if handler is None:
-            await deps.tasks.fail(task.id, f"no handler for {task.kind}", max_attempts=1)
+            await _report(
+                deps.tasks.fail(task.id, f"no handler for {task.kind}", worker_id, max_attempts=1),
+                task.id,
+                "no-handler",
+            )
             return
 
         # Re-parent to the span that enqueued this task. Without the stored traceparent
@@ -77,17 +103,30 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
                 # CancelledError inherits from BaseException, so `except Exception` below
                 # would never see it. Catch it only to release the claim, then ALWAYS
                 # re-raise: swallowing it breaks cancellation for everyone above us.
-                await deps.tasks.fail(task.id, "cancelled", max_attempts=deps.settings.max_attempts)
+                await _report(
+                    deps.tasks.fail(task.id, "cancelled", worker_id, deps.settings.max_attempts),
+                    task.id,
+                    "cancelled",
+                )
                 raise
             except Exception as exc:
                 log.exception("task failed", kind=task.kind.value, error=str(exc))
                 span.record_exception(exc)
                 span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
-                obs.TASKS_FAILED.labels(kind=task.kind.value).inc()
-                await deps.tasks.fail(task.id, str(exc), max_attempts=deps.settings.max_attempts)
+                obs.TASK_ATTEMPTS_FAILED.labels(kind=task.kind.value).inc()
+                await _report(
+                    deps.tasks.fail(task.id, str(exc), worker_id, deps.settings.max_attempts),
+                    task.id,
+                    "fail",
+                )
             else:
-                obs.PROVISION_TIME.observe(time.monotonic() - started)
-                await deps.tasks.complete(task.id)
+                # Only provisions go in the provision histogram. The buckets run 10s..1800s
+                # because they were sized for helm installs; a sub-second `verify` dropped
+                # into the same series drags the p95 down and quietly stops
+                # SvcforgeProvisionSlow from ever firing.
+                if task.kind is TaskKind.PROVISION:
+                    obs.PROVISION_TIME.observe(time.monotonic() - started)
+                await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
     finally:
         sem.release()
 
@@ -135,6 +174,7 @@ async def _amain() -> None:
     # the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
     # registry is up.
     obs.setup("svcforge-worker", settings)
+    settings.check_production()
     obs.start_metrics_server(settings.metrics_port)
 
     pool = make_pool(settings.pg_dsn.unicode_string(), settings.pool_min_size, settings.pool_max_size)
@@ -145,6 +185,7 @@ async def _amain() -> None:
         instances=InstanceRepo(pool),
         tasks=TaskRepo(pool),
         provisioner=HelmProvisioner(helm_bin=settings.helm_bin, timeout_s=int(settings.helm_timeout_s)),
+        namespaces=KubectlClient(kubectl_bin=settings.kubectl_bin),
         notifier=LogNotifier(),
         clock=SystemClock(),
         catalog=load_catalog(settings.catalog_path),