19 Commits

Author	SHA1	Message	Date
Nguyen Minh Phuc	08a529fa63	Get off Docker Hub, add a catalog values passthrough, fix the dind prune ... Docker Hub rate-limits anonymous pulls per source IP and every node here shares one NAT address, so a busy afternoon fails an unrelated build with `toomanyrequests`. Nothing in this repo needs to be there. Every base image now comes from mirror.gcr.io (python, alpine/helm, postgres) or ghcr.io (uv, trivy). Verified digest-for-digest against Docker Hub before switching, including the superseded postgres digest this repo still pins, so every existing pin stays valid — same bytes, different transport. catalog.yaml: the three bitnami entries named `bitnamilegacy/<chart>`, a repo alias nothing in the worker image configures, so they could never resolve at provision time. All five entries are now `oci://` refs, which need no `helm repo add`, and all are on latest stable: elasticsearch 21.3.15 -> 22.1.6 redis 20.6.2 -> 27.0.15 postgresql 16.4.5 -> 18.8.0 podinfo 6.7.1 -> 6.14.0 Moving the chart pull is only half of it, though: a bitnami chart defaults its own images to registry-1.docker.io. CatalogEntry gains a `values:` dict, merged under the size's replicas and resources, so an entry can set `global.imageRegistry` and move the image pull too. Size wins on conflict — otherwise an entry setting replicaCount would make every size deploy the same shape. Deep merge, because a shallow one drops sibling keys of a shared nested map. Bitnami charts reject a substituted registry unless `global.security.allowInsecureImages` is set. That check is about provenance, and the mirror serves byte-identical manifests, so it is set deliberately and only for entries whose digests were verified. The dind prune had `--filter until=168h` on both prunes, and it got both cases exactly backwards. `until` reads an image's CREATED time, so it deleted trivy every leg (a released tool image is always older than any window) while protecting the dangling build layers it existed to remove. Measured on node0: 21 dangling images / 5.96GB, and exactly 1 of them older than 168h. Trivy is protected by a tag now, so the image prune drops the filter; buildx keeps it, where age genuinely matters. Tests: +10 unit (deep merge, precedence, no-mutation, and a guard that fails if any catalog entry points at Docker Hub). Both new guards were control-tested by breaking the code and watching them fail. The API test that hardcoded `21.3.15` now reads the catalog — its subject is where the value comes from, not what it is.	2026-07-21 15:32:12 +00:00
Nguyen Minh Phuc	58ffb9c2e0	ci: tag the trivy image so the reclaim step stops deleting it ... Run #68 re-pulled trivy in all three matrix legs — 178MB each, ~7s each — because the reclaim step deletes it at the end of every leg: Unable to find image 'aquasec/trivy:0.72.0@sha256:cffe...' locally untagged: aquasec/trivy@sha256:cffe... Total reclaimed space: 178.6MB The `--filter until=168h` I added for exactly this reason does not work. That filter reads the image's CREATED timestamp, not the pull time, and aquasec/trivy:0.72.0 was built months ago, so it matches immediately. The comment claiming the age filter fixed this was wrong; corrected in place. Pull by digest, tag it, run the tag. A tagged image is not dangling, which is what actually takes it out of `docker image prune`'s scope. The digest is still the pin — enforced at pull time. Everything else in run #68 is clean: 76 unit (98.59% coverage), 112 integration, 18 CACHED layers per image job, zero uv hardlink warnings, and the trivy DB volume hit on 2 of 3 legs (one download, two silent).	2026-07-21 15:11:27 +00:00
Nguyen Minh Phuc	64b18d2823	ci: set UV_LINK_MODE=copy to stop the hardlink warning ... Every uv job logged this five times a run: warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. The uv cache lives on the runner's cache volume and the venv is on the job container's filesystem, so hardlinking cannot work and uv copies all 86 packages regardless. Declaring `copy` does not make anything slower — it is already copying — it removes a warning that appears in every green run, and a warning nobody can act on is one that teaches people to skim CI logs. Found by reading the logs of a passing run rather than a failing one. The Dockerfiles already set this; CI did not.	2026-07-21 06:02:32 +00:00
Nguyen Minh Phuc	e971e04d75	fix: shared runtime helper must live where every image packages it ... services/_runtime.py crashed the worker and reconciler on boot with `ModuleNotFoundError: No module named 'services._runtime'`, while every unit and integration gate was green and the API rolled out fine. The cause is packaging, not code. Each service Dockerfile copies only its own `services/<svc>/` subdir — `services/` itself is a namespace package with no __init__.py, so a file added at the `services/` root is never copied into any image. Tests import from the source tree, where the file exists, so nothing below the image boundary could catch it. The API survived only because it does not import the helper. Moved to svcforge_core.runtime, which `COPY libs/ libs/` packages into every image, next to adapters/tempyaml.py for the same reason. Added an import smoke test to the image job: `docker run --entrypoint python <image> -c "import services.<svc>.main"` loads the whole transitive graph inside the built image and fails the build before the digest is pushed. This is the one check the test suite structurally cannot perform — it runs against source, the image is a different filesystem — and it is exactly the gap this bug fell through.	2026-07-21 02:40:38 +00:00
Nguyen Minh Phuc	e8b6116a58	ci: age-filter the image prune so digest-pulled images survive ... Run #17 showed `docker image prune -f` deleting the trivy image, which buys back a 157MB pull on every subsequent run. An image pulled by digest carries no tag, so it is dangling the moment its container exits, and bare dangling-only pruning is not as narrow as it reads. The act runner image survived the same prune only by accident: this step runs inside an act container, so that image happens to be in use exactly while the prune executes. That is luck, not design, and it would not hold if the step ever moved. An until=168h filter on both prunes keeps a week of recently used layers, matching the buildx cache window already in place. Verified in run #17 before this change: trivy logged "Need to update DB" zero times, so the named-volume DB cache works, and registry digests equal the chart's for all three images.	2026-07-20 05:17:13 +00:00
Nguyen Minh Phuc	37b297bf5c	ci: reclaim dind disk after each image build ... dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed.	2026-07-20 04:48:48 +00:00
Nguyen Minh Phuc	b02d4e85c6	ci: record that act_runner ignores max-parallel ... The previous commit claimed max-parallel: 1 would stop two image builds from starving each other. It does not — act_runner ignores strategy.max-parallel. Run #14 had it set and still ran the worker leg from 04:25:47 while reconciler was still building, after api had run alone from 04:23:03 to 04:24:04. That run failed a different way: actions/checkout could not reach Gitea at all (`Failed to connect to gitea-http:3000 after 3105 ms`) while two runs were in flight, with node0 at 140% memory. Same root cause, new mask. The setting stays, since it is correct on runners that honour it, but the comment no longer claims it does anything here. The binding lever is the runner's `capacity`, dropped 2 -> 1 in oci-k8s.	2026-07-20 04:27:48 +00:00
Nguyen Minh Phuc	d70a7b622c	ci: cache the trivy vuln DB in a named volume, not a $PWD bind ... `-v "$PWD/.trivycache:/root/.cache/trivy"` was the third instance of the bug that made gitleaks scan nothing. $PWD is a path in the job container, but the -v is resolved by the daemon in the dind sidecar, which has no such directory and silently creates an empty one. Every run logged `[vulndb] Need to update DB` and re-downloaded the DB, and the cache it wrote went to a throwaway directory inside dind. This file warns about the same trap in two other places. A named volume lives in the dind daemon's own storage, which is the one thing both sides agree on. Verified in dind rather than assumed: cold pass logs `Need to update DB` / `Downloading vulnerability DB` / `Artifact successfully downloaded` and leaves 1.1G in the volume; warm pass logs none of them. The old comment's "~50MB" was wrong by 20x, so the comment now records the real figure and what reclaims it. The two sibling pipelines on this runner (cicd-smoke, otel-demo) were checked for the same bug and do not have it — they use `docker build <context>`, where the client streams the context to the daemon.	2026-07-20 04:13:30 +00:00
Nguyen Minh Phuc	2098443f31	ci: build images one at a time ... `image (api)` and `image (reconciler)` both died in run #13 with: ERROR: failed to solve: DeadlineExceeded: no active session for ylux5s1p4xyb5e5tmnz4gljhq: context deadline exceeded That is the buildkit session heartbeat between buildx in the job container and buildkitd in the dind sidecar, not anything in the Dockerfiles. The runner has capacity 2, so two legs of the matrix built concurrently on a 2-core burstable node already sitting at 81% CPU and 87% memory, and starved each other. The evidence is in the same run: `image (worker)` was still building when the other two failed, had the node to itself afterwards, and went green on an unchanged Dockerfile. max-parallel: 1 trades wall-clock for builds that finish. Every other gate in run #13 — lint, types, unit, integration, security, dockerfile, chart — was already green, so this is the last thing standing between here and a full run.	2026-07-20 04:11:04 +00:00
Nguyen Minh Phuc	b35c551160	ci: the digest-guard test depended on state CI itself mutates ... The chart job asserted that a bare `helm template` FAILS, on the assumption that values.yaml always holds all-zeros placeholder digests. That assumption dies the first time the bump job runs: bump commits real digests into values.yaml, so the bare render then succeeds and the assertion reports 'the guard is not guarding' about a guard that is fine. A test whose expected result flips depending on whether CI has run before is not a test. Now it feeds the guard four distinct bad digests explicitly with --set — all-zeros, a bare tag, right-prefix-wrong-length, and empty — and requires each to be rejected. Verified locally: all four rejected, and a well-formed digest still renders.	2026-07-20 03:48:34 +00:00
Nguyen Minh Phuc	c537073c21	deps: everything to latest stable ... Python 3.12 -> 3.14, postgres 16 -> 18, uv 0.5.11 -> 0.11.29, trivy 0.58.1 -> 0.72.0, gitleaks 8.21.2 -> 8.30.1, yq 4.44.6 -> 4.53.3, and every action re-pinned to the SHA of its latest tag (checkout v7, setup-uv v8, buildx v4, login v4, hadolint v3.3.0). helm stays 3.21.3: already current for 3.x, and helm 4 is a breaking change, not a CVE fix. trivy mattered most. A vulnerability scanner fourteen minor versions behind is the one stale pin that hides all the others. ruff target-version is deliberately py313 while the runtime is 3.14. It controls the syntax the formatter may emit, and at py314 it rewrites 'except (A, B):' into PEP 758's unparenthesized form — which reads exactly like Python 2's 'except E, name:' and is a hard SyntaxError below 3.14. No semantic gain, real readability cost, in a repo meant to be read. Verified on 3.14: ruff, ruff format, mypy --strict, 166 tests, helm lint, bandit, pip-audit. The digest guard still rejects placeholder digests. Risk carried knowingly: the bumped actions run on node24. If act_runner only provides node20, every job fails at action startup and this commit is the revert.	2026-07-19 09:39:38 +00:00
Nguyen Minh Phuc	ca21b6e70d	ci: gitleaks was passing without scanning anything ... The secret-scanning gate has been green and meaningless. `docker run -v "$PWD:/repo"` resolves the bind source on the dind sidecar's daemon, not in the job container, so gitleaks received an empty mount: ERR [git] fatal: not a git repository (or any parent up to mount point /) ERR failed to scan Git repository error="stderr is not empty" INF scan completed in 35.2ms INF no leaks found <- exit 0 It logged the failure and exited 0. A scanner that reports success without looking is worse than no scanner. The 35ms runtime was the tell. Now a checksum-pinned binary, plus a `git rev-list --count HEAD` assertion so an unscannable checkout fails the job instead of passing it. Same root cause and same fix for yq in bump-digests.sh, which failed the bump job with "stat deploy/chart/values.yaml: no such file or directory". helm was fixed this way earlier. Nothing on this runner should mount $PWD into a container.	2026-07-19 09:11:22 +00:00
Nguyen Minh Phuc	dc51bd1d9d	ci: move the bump job's master guard from job level to step level ... bump skipped at 0s on a genuine push to master (event: push, head_branch: master) — before the image job it depends on had even started. The identical expression on the 'push by digest' STEP inside the matrix job evaluates true and runs, so the expression is fine; Gitea does not resolve a job-level if correctly when needs points at a matrix job. needs: [image] still orders it and still gates on all three legs. The steps carry the guard in the form this runner is known to evaluate. A PR now starts the job and no-ops every step, which is a few seconds for a guard that actually fires.	2026-07-19 09:00:43 +00:00
Nguyen Minh Phuc	a51227d952	ci: run helm from a checksum-pinned binary, not a container ... The chart gate failed with 'stat deploy/chart/Chart.yaml: no such file or directory' while the file plainly existed in the checkout. `docker run -v "$PWD:/repo"` is interpreted by the dind sidecar's daemon rather than by the job container, so the bind source has to exist on the daemon's side of that boundary. Downloading the binary removes the boundary entirely. Pinned by sha256 for the same reason every image here is pinned by digest.	2026-07-18 12:20:16 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	77d560ddae	ci: move buildx layer cache to the registry ... act_runner's cache PVC is 1Gi and also holds .runner, the runner's own registration file. --cache-to type=gha,mode=max for three images is several GB; filling that volume breaks the runner, not just the cache. act_runner also evicts by age with no size cap, so it fills whatever it is given. type=registry has no such limit and lives beside the images it caches. The uv cache still uses the runner's cache service, which is a few hundred MB. Registry login is no longer gated to master: the build now reads and writes the cache on every run. Pushing the release image keeps its own master-only gate.	2026-07-18 11:24:44 +00:00
Nguyen Minh Phuc	c9d0176bb3	ci: run trivy directly; document CI/CD setup in RUNBOOK ... trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache.	2026-07-17 11:05:45 +00:00
Nguyen Minh Phuc	851f8919a8	ci: fix coverage target, pip-audit scope, bandit config ... - --cov pointed at libs/svcforge_core/domain, a path that does not exist (the package nests one level deeper). Coverage measured 0.00% of the code. Use the module form, which is layout-independent. - pip-audit --strict cannot audit our own editable, not-on-PyPI packages. Audit the locked dependency set instead and keep --strict. - bandit re-reports B608/B104, which ruff's S ruleset already enforces with justified per-line noqa it cannot see. Skipped in config, with reasons. - registry host was git.oci-oci; it is gitea.oci-oci. - integration job set SVCFORGE_PG_DSN; conftest reads SVCFORGE_TEST_DSN.	2026-07-17 10:51:59 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00