review: fix 26 findings from a 4-agent audit
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.
This commit is contained in:
Nguyen Minh Phuc
2026-07-18 12:13:49 +00:00
parent 77d560ddae
commit c76154aeaa
45 changed files with 1520 additions and 216 deletions
+28 -5
View File
@@ -15,6 +15,7 @@ from fastapi import Depends, HTTPException, Request, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jwt import PyJWKClient
from svcforge_core.adapters.redis import RateLimiterProto
from svcforge_core.domain.models import CatalogEntry
from svcforge_core.repo.db import DictPool
from svcforge_core.repo.instances import InstanceRepo
@@ -146,13 +147,35 @@ async def _signing_key(client: PyJWKClient, token: str) -> jwt.PyJWK:
return await asyncio.to_thread(client.get_signing_key_from_jwt, token)
async def rate_limit(team: Annotated[str, Depends(get_current_team)]) -> None:
"""Per-team rate limiting. Seam only — Module 10 fills this in (Redis, Lua, token bucket).
def get_rate_limiter(request: Request) -> RateLimiterProto | None:
"""The limiter lifespan built, or None when no Redis is configured."""
limiter: RateLimiterProto | None = getattr(request.app.state, "rate_limiter", None)
return limiter
It exists now, wired into the routes, so that turning it on is an edit to one function
body rather than a change to every handler signature.
async def rate_limit(
request: Request,
team: Annotated[str, Depends(get_current_team)],
) -> None:
"""Per-team rate limiting. One Redis command per check, and it fails OPEN.
Failing open is the entire policy. Redis holds derived state; losing it must degrade
the platform, never stop it. A limiter that fails closed converts a cache outage into
a total outage, which is a strictly worse incident than the burst it was protecting
against — so `RateLimiter.check` swallows its own errors and returns `allowed=True`.
The 429 below therefore only ever comes from a real, counted overage.
"""
return None
limiter = get_rate_limiter(request)
if limiter is None:
return
result = await limiter.check(team)
if not result.allowed:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail={"code": "rate_limited", "message": "too many requests"},
headers={"Retry-After": str(result.retry_after_s)},
)
async def idempotency_key(request: Request) -> str | None: