63 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nguyen Minh Phuc	4544765ec5	test: scope the process-group assertion to this run's own children ... test_timeout_kills_the_whole_process_group asserted that `pgrep -f 'sleep 300'` returns nothing. That is machine-global: it matches a leftover from an earlier run of the same test, any unrelated 'sleep 300' on the box, and the shell running pgrep, whose own command line contains the pattern being searched for. Observed three spurious matches on a dev box, which produced a red run that looked like a Python 3.14 regression in the helm timeout kill. It was not — the same test failed identically on 3.12. A test that fails for reasons unrelated to the code is as useless as one that cannot fail. The sleep duration is now derived from the pid, so it cannot collide with another run, and the test asserts up front that its own pattern matches nothing before it starts. Verified: 3 consecutive passes on each of Python 3.12 and 3.14.	2026-07-19 09:39:38 +00:00
svcforge-ci	f2b159ef7e	ci: bump image digests to ca21b6e70d ... Built and scanned by ca21b6e70d. ArgoCD syncs from this commit. [skip ci]	2026-07-19 09:18:15 +00:00
Nguyen Minh Phuc	ca21b6e70d	ci: gitleaks was passing without scanning anything ... The secret-scanning gate has been green and meaningless. `docker run -v "$PWD:/repo"` resolves the bind source on the dind sidecar's daemon, not in the job container, so gitleaks received an empty mount: ERR [git] fatal: not a git repository (or any parent up to mount point /) ERR failed to scan Git repository error="stderr is not empty" INF scan completed in 35.2ms INF no leaks found <- exit 0 It logged the failure and exited 0. A scanner that reports success without looking is worse than no scanner. The 35ms runtime was the tell. Now a checksum-pinned binary, plus a `git rev-list --count HEAD` assertion so an unscannable checkout fails the job instead of passing it. Same root cause and same fix for yq in bump-digests.sh, which failed the bump job with "stat deploy/chart/values.yaml: no such file or directory". helm was fixed this way earlier. Nothing on this runner should mount $PWD into a container.	2026-07-19 09:11:22 +00:00
Nguyen Minh Phuc	dc51bd1d9d	ci: move the bump job's master guard from job level to step level ... bump skipped at 0s on a genuine push to master (event: push, head_branch: master) — before the image job it depends on had even started. The identical expression on the 'push by digest' STEP inside the matrix job evaluates true and runs, so the expression is fine; Gitea does not resolve a job-level if correctly when needs points at a matrix job. needs: [image] still orders it and still gates on all three legs. The steps carry the guard in the form this runner is known to evaluate. A PR now starts the job and no-ops every step, which is a few seconds for a guard that actually fires.	2026-07-19 09:00:43 +00:00
Nguyen Minh Phuc	78a0a6d500	runbook: record how the image gate was taken to zero findings ... trivy on the worker image: 39 findings (2 CRITICAL) -> 18 -> 5 -> 0. Verified in CI run #7 on 4a426db: 'Total: 0 (HIGH: 0, CRITICAL: 0)' for all three images. The last five lived in kubectl's vendored golang.org/x/net and Go stdlib, inside the newest kubectl published. No version cleared them; removing the binary did.	2026-07-19 08:48:51 +00:00
Nguyen Minh Phuc	4a426dbe50	worker: create namespaces with helm, drop the kubectl binary ... trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the newest kubectl that exists. No release clears them; upstream has not rebuilt against the patched Go yet. Chasing the version further has no end. kubectl was in that image for exactly one call: ensure_namespace before helm. `helm upgrade --install --create-namespace` does the same thing, idempotently, as part of the install it already runs. So the binary goes, and its vendored CVEs go with it. read_secret had no callers. Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since --create-namespace makes a bare one. Nothing reads that label today.	2026-07-18 13:05:22 +00:00
Nguyen Minh Phuc	a59d3848ae	ci: kubectl v1.35.3 -> v1.36.2 to clear CVE-2026-35469 ... trivy took the worker image from 39 findings (2 CRITICAL) to 18 (0 CRITICAL) after the previous bump. The residue is kubectl's vendored deps: k8s v1.35.x pins github.com/moby/spdystream v0.5.0, and the fix is in 0.5.1. v1.36.2 vendors 0.5.1. One minor ahead of the v1.35.3 API server, which is inside kubectl's supported +/-1 window. reconciler was already green because it ships helm only.	2026-07-18 12:43:36 +00:00
Nguyen Minh Phuc	a51227d952	ci: run helm from a checksum-pinned binary, not a container ... The chart gate failed with 'stat deploy/chart/Chart.yaml: no such file or directory' while the file plainly existed in the checkout. `docker run -v "$PWD:/repo"` is interpreted by the dind sidecar's daemon rather than by the job container, so the bind source has to exist on the daemon's side of that boundary. Downloading the binary removes the boundary entirely. Pinned by sha256 for the same reason every image here is pinned by digest.	2026-07-18 12:20:16 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	77d560ddae	ci: move buildx layer cache to the registry ... act_runner's cache PVC is 1Gi and also holds .runner, the runner's own registration file. --cache-to type=gha,mode=max for three images is several GB; filling that volume breaks the runner, not just the cache. act_runner also evicts by age with no size cap, so it fills whatever it is given. type=registry has no such limit and lives beside the images it caches. The uv cache still uses the runner's cache service, which is a few hundred MB. Registry login is no longer gated to master: the build now reads and writes the cache on every run. Pushing the release image keeps its own master-only gate.	2026-07-18 11:24:44 +00:00
Nguyen Minh Phuc	c9d0176bb3	ci: run trivy directly; document CI/CD setup in RUNBOOK ... trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache.	2026-07-17 11:05:45 +00:00
Nguyen Minh Phuc	851f8919a8	ci: fix coverage target, pip-audit scope, bandit config ... - --cov pointed at libs/svcforge_core/domain, a path that does not exist (the package nests one level deeper). Coverage measured 0.00% of the code. Use the module form, which is layout-independent. - pip-audit --strict cannot audit our own editable, not-on-PyPI packages. Audit the locked dependency set instead and keep --strict. - bandit re-reports B608/B104, which ruff's S ruleset already enforces with justified per-line noqa it cannot see. Skipped in config, with reasons. - registry host was git.oci-oci; it is gitea.oci-oci. - integration job set SVCFORGE_PG_DSN; conftest reads SVCFORGE_TEST_DSN.	2026-07-17 10:51:59 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00