review: fix 26 findings from a 4-agent audit
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.
This commit is contained in:
Nguyen Minh Phuc
2026-07-18 12:13:49 +00:00
parent 77d560ddae
commit c76154aeaa
45 changed files with 1520 additions and 216 deletions
+20
View File
@@ -41,10 +41,21 @@ def _settings(**over: object) -> Settings:
return Settings(**base) # type: ignore[arg-type]
class FakeNamespaceEnsurer:
"""Records the namespaces it was asked to create. Satisfies NamespaceEnsurer."""
def __init__(self) -> None:
self.ensured: list[str] = []
async def ensure_namespace(self, ns: str, labels: dict[str, str] | None = None) -> None:
self.ensured.append(ns)
def _deps(
pool: DictPool,
prov: FakeProvisioner,
notifier: FakeNotifier | None = None,
namespaces: FakeNamespaceEnsurer | None = None,
**over: object,
) -> WorkerDeps:
return WorkerDeps(
@@ -52,6 +63,7 @@ def _deps(
instances=InstanceRepo(pool),
tasks=TaskRepo(pool),
provisioner=prov,
namespaces=namespaces or FakeNamespaceEnsurer(),
notifier=notifier or FakeNotifier(),
clock=SystemClock(),
catalog=CATALOG,
@@ -200,4 +212,12 @@ async def test_concurrency_cap_is_respected(pool: DictPool, concurrency: int) ->
stop.set()
await asyncio.wait_for(worker, timeout=10)
# BOTH bounds. `<= concurrency` alone passes on a worker whose semaphore is broken to 1,
# or that lost concurrency entirely — it only proves the cap is not exceeded, never that
# concurrency exists. With 6 tasks seeded and a 0.2s handler, a working worker reaches
# its cap exactly.
assert prov.max_concurrent <= concurrency, f"ran {prov.max_concurrent} at once, cap was {concurrency}"
assert prov.max_concurrent == concurrency, (
f"only reached {prov.max_concurrent} concurrent with a cap of {concurrency}: "
"the worker is not actually running tasks in parallel"
)