6 Commits

Author	SHA1	Message	Date
Nguyen Minh Phuc	4193a18cae	reconciler: read release secrets directly instead of shelling helm ... helm's list gunzips every release payload to build its table. The only fields either caller reads are name and namespace: reconciler/main.py live = {(r.name, r.namespace) for r in releases} worker/handlers.py releases = {r.name for r in await ...list_releases()} Both live in the release secret's labels and metadata, so nothing needs decompressing. Measured from a pod on this cluster: 21ms and 31KB, against helm's 4392ms. That is the whole fix for a drift check that was timing out at 330s every tick and OOMKilling the container at 256Mi. The cause of both was decompressing 96 releases to extract two strings each. It also means the container no longer cares that a cluster policy mutates its CPU request to 0 — at 21ms there is nothing left to starve. Three details carry the correctness, each with a test: - PartialObjectMetadataList in the Accept header asks for metadata only. Without it every release's gzipped manifest crosses the wire to be thrown away, which is the cost this exists to avoid. - The status selector drops superseded revisions server-side: 96 release secrets here, 25 live. Failed and pending states are kept, matching what `helm list` shows, because a failed release does exist and calling it missing would have the reconciler re-provision on top of it. - helm writes one secret per revision, up to 10 per release here, so the newest version label wins. Control-tested with a string compare, which picks "9" over "10". Out of cluster there is no ServiceAccount, so it falls back to helm and the e2e suite and laptop runs are unchanged. An API error raises rather than falling back: the fallback is for a known-absent ServiceAccount, and quietly retrying through helm would swap a visible error for the timeout this removes. chart and app_version are empty on this path rather than wrong — they live only in the compressed payload, and nothing reads them.	2026-07-20 13:05:25 +00:00
Nguyen Minh Phuc	2356ac4ef3	reconciler: scope the drift check to svcforge's own releases ... check_drift diffs helm against the database in both directions, and the second one is `live - known` -> drift.orphan_release at ERROR. `live` was every release in the cluster, so argocd, longhorn, gitea and cert-manager were all reported as orphans svcforge is failing to account for, on every sweep. They are not orphans; they were never svcforge's to know about. install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases() selects on it. Releases provisioned before this see one sweep as missing and get re-provisioned, which is safe by construction — `helm upgrade --install` against a deterministic release name — and that re-provision applies the label. This does NOT fix the timeout, and the docstring says so. Measured, not assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is not pushed down as a server-side selector, so helm still fetches and decompresses every release secret and filters what it already parsed. Around 10%, not the order of magnitude the flag's shape suggests. The 330s timeouts need CPU for the container or a different read path. The two sides of the label are asserted against each other rather than a literal, so a rename that updates only one fails in tests instead of in production as a silently empty drift check. Control-tested: renaming the read side alone fails two of the three.	2026-07-20 12:53:16 +00:00
Nguyen Minh Phuc	c537073c21	deps: everything to latest stable ... Python 3.12 -> 3.14, postgres 16 -> 18, uv 0.5.11 -> 0.11.29, trivy 0.58.1 -> 0.72.0, gitleaks 8.21.2 -> 8.30.1, yq 4.44.6 -> 4.53.3, and every action re-pinned to the SHA of its latest tag (checkout v7, setup-uv v8, buildx v4, login v4, hadolint v3.3.0). helm stays 3.21.3: already current for 3.x, and helm 4 is a breaking change, not a CVE fix. trivy mattered most. A vulnerability scanner fourteen minor versions behind is the one stale pin that hides all the others. ruff target-version is deliberately py313 while the runtime is 3.14. It controls the syntax the formatter may emit, and at py314 it rewrites 'except (A, B):' into PEP 758's unparenthesized form — which reads exactly like Python 2's 'except E, name:' and is a hard SyntaxError below 3.14. No semantic gain, real readability cost, in a repo meant to be read. Verified on 3.14: ruff, ruff format, mypy --strict, 166 tests, helm lint, bandit, pip-audit. The digest guard still rejects placeholder digests. Risk carried knowingly: the bumped actions run on node24. If act_runner only provides node20, every job fails at action startup and this commit is the revert.	2026-07-19 09:39:38 +00:00
Nguyen Minh Phuc	4a426dbe50	worker: create namespaces with helm, drop the kubectl binary ... trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the newest kubectl that exists. No release clears them; upstream has not rebuilt against the patched Go yet. Chasing the version further has no end. kubectl was in that image for exactly one call: ensure_namespace before helm. `helm upgrade --install --create-namespace` does the same thing, idempotently, as part of the install it already runs. So the binary goes, and its vendored CVEs go with it. read_secret had no callers. Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since --create-namespace makes a bare one. Nothing reads that label today.	2026-07-18 13:05:22 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00