Nguyen Minh Phuc
c76154aeaa
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.
53 lines
1.9 KiB
Python
53 lines
1.9 KiB
Python
"""What a worker needs to do its job.
|
|
|
|
One frozen bag of collaborators, constructed once in main() and passed down. Handlers
|
|
take `deps` rather than reaching for globals, which is the entire reason the worker tests
|
|
run in milliseconds against a FakeProvisioner instead of needing a cluster.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
from typing import Protocol
|
|
|
|
from svcforge_core.adapters.clock import Clock
|
|
from svcforge_core.adapters.helm import Provisioner
|
|
from svcforge_core.adapters.notify import Notifier
|
|
from svcforge_core.domain.models import CatalogEntry
|
|
from svcforge_core.repo.db import DictPool
|
|
from svcforge_core.repo.instances import InstanceRepo
|
|
from svcforge_core.repo.tasks import TaskRepo
|
|
from svcforge_core.settings import Settings
|
|
|
|
|
|
class NamespaceEnsurer(Protocol):
|
|
"""The one thing the worker needs from kubectl.
|
|
|
|
Narrower than `KubectlClient` on purpose: the handler creates namespaces and does
|
|
nothing else with the cluster, so that is the whole interface. Satisfied by
|
|
`KubectlClient` in production and by a fake in tests, which is what keeps the worker
|
|
suite running in milliseconds with no cluster.
|
|
"""
|
|
|
|
async def ensure_namespace(self, ns: str, labels: dict[str, str] | None = None) -> None:
|
|
"""Create the namespace if absent. Idempotent — safe on every retry."""
|
|
...
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class WorkerDeps:
|
|
"""Everything a handler is allowed to touch."""
|
|
|
|
pool: DictPool
|
|
instances: InstanceRepo
|
|
tasks: TaskRepo
|
|
provisioner: Provisioner
|
|
# Creates the tenant namespace before helm is pointed at it. `helm --namespace X`
|
|
# does NOT create X, so without this the very first provision for a new team fails
|
|
# with "namespace not found" — the one path that is guaranteed untested by a fake.
|
|
namespaces: NamespaceEnsurer
|
|
notifier: Notifier
|
|
clock: Clock
|
|
catalog: dict[str, CatalogEntry]
|
|
settings: Settings
|