8 Commits

Author	SHA1	Message	Date
Nguyen Minh Phuc	08a529fa63	Get off Docker Hub, add a catalog values passthrough, fix the dind prune ... Docker Hub rate-limits anonymous pulls per source IP and every node here shares one NAT address, so a busy afternoon fails an unrelated build with `toomanyrequests`. Nothing in this repo needs to be there. Every base image now comes from mirror.gcr.io (python, alpine/helm, postgres) or ghcr.io (uv, trivy). Verified digest-for-digest against Docker Hub before switching, including the superseded postgres digest this repo still pins, so every existing pin stays valid — same bytes, different transport. catalog.yaml: the three bitnami entries named `bitnamilegacy/<chart>`, a repo alias nothing in the worker image configures, so they could never resolve at provision time. All five entries are now `oci://` refs, which need no `helm repo add`, and all are on latest stable: elasticsearch 21.3.15 -> 22.1.6 redis 20.6.2 -> 27.0.15 postgresql 16.4.5 -> 18.8.0 podinfo 6.7.1 -> 6.14.0 Moving the chart pull is only half of it, though: a bitnami chart defaults its own images to registry-1.docker.io. CatalogEntry gains a `values:` dict, merged under the size's replicas and resources, so an entry can set `global.imageRegistry` and move the image pull too. Size wins on conflict — otherwise an entry setting replicaCount would make every size deploy the same shape. Deep merge, because a shallow one drops sibling keys of a shared nested map. Bitnami charts reject a substituted registry unless `global.security.allowInsecureImages` is set. That check is about provenance, and the mirror serves byte-identical manifests, so it is set deliberately and only for entries whose digests were verified. The dind prune had `--filter until=168h` on both prunes, and it got both cases exactly backwards. `until` reads an image's CREATED time, so it deleted trivy every leg (a released tool image is always older than any window) while protecting the dangling build layers it existed to remove. Measured on node0: 21 dangling images / 5.96GB, and exactly 1 of them older than 168h. Trivy is protected by a tag now, so the image prune drops the filter; buildx keeps it, where age genuinely matters. Tests: +10 unit (deep merge, precedence, no-mutation, and a guard that fails if any catalog entry points at Docker Hub). Both new guards were control-tested by breaking the code and watching them fail. The API test that hardcoded `21.3.15` now reads the catalog — its subject is where the value comes from, not what it is.	2026-07-21 15:32:12 +00:00
Nguyen Minh Phuc	6974b3620f	catalog: two tiny services, so the loop can be exercised on a full cluster ... The three existing entries are sized like real products: `elasticsearch` small asks for 1Gi and medium for 4Gi across three replicas. On this cluster that is a request that never schedules, so provisioning them proves something about the node and nothing about svcforge. podinfo 16Mi/10m — one small Go binary, no dependencies, no PVC nginx 32Mi/10m — recognisable, still small Both are addressed as `oci://`, which is load-bearing rather than cosmetic. An OCI chart is pulled by reference with no `helm repo add` first. The three existing entries name `bitnamilegacy/<chart>`, a classic repo alias that nothing in the worker image configures — so as written they cannot resolve at provision time. OCI is the form that works from a bare container, and it is why the e2e test already provisions podinfo. Chart versions were resolved against the real registries before committing (podinfo 6.7.1, nginx 25.0.14 / app 1.31.3) rather than guessed, since a wrong version fails only at provision time. Also updates the CLI ServiceType enum, the catalog test's expected set, and the service list in the OpenAPI description.	2026-07-21 05:47:41 +00:00
Nguyen Minh Phuc	d64c3c9f39	fix: three correctness bugs found in review + a dropped-log ... fail() blanket-marked the instance `failed` for every dead-lettered task kind. Only provision is correct. The others each left the instance in a state the UPDATE then corrupted: - deprovision: `deleting` -> `failed` stranded the instance, because due_for_deprovision only re-selects ready/deleting, leaking the release reconcile.py promises to reclaim. - upgrade: helm --atomic rolled back, so the instance was still `ready` and serving the old version; `failed` mislabelled a healthy service and dropped it off the upgrade work-list. - verify: handle_verify already halted the rollout; the instance was `ready`. Now gated on kind == provision, with a regression test per kind (control-tested against the blanket UPDATE, which fails all three). The API exception handler was registered on fastapi.HTTPException, a subclass of starlette's. Starlette matches handlers by walking type(exc).__mro__, so framework-raised 404/405 never hit it and returned {"detail": ...} instead of ErrorBody. Registered on the starlette parent, and added a RequestValidationError handler so body-validation 422s share the shape too. Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field. helm._list_releases_via_api built the httpx client with verify=<ca path>, which loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an error the except clause did not catch, crashing the reconciler tick as a bare bug. Now the CA is checked for readability alongside the token, and a missing one means "not in-cluster" and falls back to helm. Also documents the no-limit pagination invariant and pins it with a test. redis.py logged through stdlib logging with extra={"team": team}, which the structlog bridge drops on the floor — the trap notify.py documents. Switched to a bound logger with team as a kwarg.	2026-07-21 01:46:54 +00:00
Nguyen Minh Phuc	76cadca8e3	fix: Kubernetes serialises an empty item list as null, not [] ... The drift check failed on every tick with: TypeError: 'NoneType' object is not iterable `resp.json().get("items", [])` cannot defend against this. The key IS present, with the value null, so the default never fires. An empty PartialObjectMetadataList comes back as `"items": null`, which is exactly what happens once the release selector matches nothing — the normal state of a cluster with no tenant releases provisioned yet. `or []` handles both shapes. The tests could not have caught this: all of them passed a real empty list, which is the one shape that works. The new test sends `"items": null` as the API server actually sends it, and is control-tested — restoring the old expression fails it with the same TypeError seen in production. Worth recording why this survived review. The API read was verified against a live cluster before shipping, but that cluster had 25 matching release secrets, so the empty case never ran. A measurement on real data proved the fast path worked and said nothing about the path taken when there is no data.	2026-07-20 15:17:51 +00:00
Nguyen Minh Phuc	4193a18cae	reconciler: read release secrets directly instead of shelling helm ... helm's list gunzips every release payload to build its table. The only fields either caller reads are name and namespace: reconciler/main.py live = {(r.name, r.namespace) for r in releases} worker/handlers.py releases = {r.name for r in await ...list_releases()} Both live in the release secret's labels and metadata, so nothing needs decompressing. Measured from a pod on this cluster: 21ms and 31KB, against helm's 4392ms. That is the whole fix for a drift check that was timing out at 330s every tick and OOMKilling the container at 256Mi. The cause of both was decompressing 96 releases to extract two strings each. It also means the container no longer cares that a cluster policy mutates its CPU request to 0 — at 21ms there is nothing left to starve. Three details carry the correctness, each with a test: - PartialObjectMetadataList in the Accept header asks for metadata only. Without it every release's gzipped manifest crosses the wire to be thrown away, which is the cost this exists to avoid. - The status selector drops superseded revisions server-side: 96 release secrets here, 25 live. Failed and pending states are kept, matching what `helm list` shows, because a failed release does exist and calling it missing would have the reconciler re-provision on top of it. - helm writes one secret per revision, up to 10 per release here, so the newest version label wins. Control-tested with a string compare, which picks "9" over "10". Out of cluster there is no ServiceAccount, so it falls back to helm and the e2e suite and laptop runs are unchanged. An API error raises rather than falling back: the fallback is for a known-absent ServiceAccount, and quietly retrying through helm would swap a visible error for the timeout this removes. chart and app_version are empty on this path rather than wrong — they live only in the compressed payload, and nothing reads them.	2026-07-20 13:05:25 +00:00
Nguyen Minh Phuc	2356ac4ef3	reconciler: scope the drift check to svcforge's own releases ... check_drift diffs helm against the database in both directions, and the second one is `live - known` -> drift.orphan_release at ERROR. `live` was every release in the cluster, so argocd, longhorn, gitea and cert-manager were all reported as orphans svcforge is failing to account for, on every sweep. They are not orphans; they were never svcforge's to know about. install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases() selects on it. Releases provisioned before this see one sweep as missing and get re-provisioned, which is safe by construction — `helm upgrade --install` against a deterministic release name — and that re-provision applies the label. This does NOT fix the timeout, and the docstring says so. Measured, not assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is not pushed down as a server-side selector, so helm still fetches and decompresses every release secret and filters what it already parsed. Around 10%, not the order of magnitude the flag's shape suggests. The 330s timeouts need CPU for the container or a different read path. The two sides of the label are asserted against each other rather than a literal, so a rename that updates only one fails in tests instead of in production as a silently empty drift check. Control-tested: renaming the read side alone fails two of the three.	2026-07-20 12:53:16 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00