25 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nguyen Minh Phuc	89ad9625f3	chart: disable externalSecret on this cluster ... The block describes a ClusterSecretStore named `vault` with HashiCorp-style key/property refs. This cluster has `oci-vault` — OCI Vault via InstancePrincipal — whose provider addresses a secret by name and takes a JSON property, so those remoteRefs do not translate as written. There are no ExternalSecrets anywhere on the cluster, so the path had never been exercised. svcforge.secretName still resolves through targetName, so the deployments and the migrate hook read a Secret named svcforge-secrets, created out of band from ~/.config/svcforge/secrets.env. This is a deviation and the comment says so. ArgoCD does not manage that Secret, so prune and selfHeal cannot touch it, and it is the one part of the deployment that cannot be read from git. Restoring the intended design means adding oci_vault_secret resources to oci-k8s/infra/vault.tf and repointing secretStoreRef at oci-vault.	2026-07-20 06:26:13 +00:00
Nguyen Minh Phuc	e8b6116a58	ci: age-filter the image prune so digest-pulled images survive ... Run #17 showed `docker image prune -f` deleting the trivy image, which buys back a 157MB pull on every subsequent run. An image pulled by digest carries no tag, so it is dangling the moment its container exits, and bare dangling-only pruning is not as narrow as it reads. The act runner image survived the same prune only by accident: this step runs inside an act container, so that image happens to be in use exactly while the prune executes. That is luck, not design, and it would not hold if the step ever moved. An until=168h filter on both prunes keeps a week of recently used layers, matching the buildx cache window already in place. Verified in run #17 before this change: trivy logged "Need to update DB" zero times, so the named-volume DB cache works, and registry digests equal the chart's for all three images.	2026-07-20 05:17:13 +00:00
Nguyen Minh Phuc	5f18f9eeeb	runbook: correct the claim that runner restarts orphan jobs ... The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel.	2026-07-20 05:17:13 +00:00
svcforge-ci	1a36f43c69	ci: bump image digests to 37b297bf5c ... Built and scanned by 37b297bf5c. ArgoCD syncs from this commit. [skip ci]	2026-07-20 05:12:26 +00:00
Nguyen Minh Phuc	37b297bf5c	ci: reclaim dind disk after each image build ... dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed.	2026-07-20 04:48:48 +00:00
Nguyen Minh Phuc	5d7f46483e	runbook: recovering the queue after a runner restart ... Restarting act_runner leaves its in-flight jobs in_progress with nothing behind them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run index, not the database id.	2026-07-20 04:41:15 +00:00
Nguyen Minh Phuc	b02d4e85c6	ci: record that act_runner ignores max-parallel ... The previous commit claimed max-parallel: 1 would stop two image builds from starving each other. It does not — act_runner ignores strategy.max-parallel. Run #14 had it set and still ran the worker leg from 04:25:47 while reconciler was still building, after api had run alone from 04:23:03 to 04:24:04. That run failed a different way: actions/checkout could not reach Gitea at all (`Failed to connect to gitea-http:3000 after 3105 ms`) while two runs were in flight, with node0 at 140% memory. Same root cause, new mask. The setting stays, since it is correct on runners that honour it, but the comment no longer claims it does anything here. The binding lever is the runner's `capacity`, dropped 2 -> 1 in oci-k8s.	2026-07-20 04:27:48 +00:00
Nguyen Minh Phuc	d70a7b622c	ci: cache the trivy vuln DB in a named volume, not a $PWD bind ... `-v "$PWD/.trivycache:/root/.cache/trivy"` was the third instance of the bug that made gitleaks scan nothing. $PWD is a path in the job container, but the -v is resolved by the daemon in the dind sidecar, which has no such directory and silently creates an empty one. Every run logged `[vulndb] Need to update DB` and re-downloaded the DB, and the cache it wrote went to a throwaway directory inside dind. This file warns about the same trap in two other places. A named volume lives in the dind daemon's own storage, which is the one thing both sides agree on. Verified in dind rather than assumed: cold pass logs `Need to update DB` / `Downloading vulnerability DB` / `Artifact successfully downloaded` and leaves 1.1G in the volume; warm pass logs none of them. The old comment's "~50MB" was wrong by 20x, so the comment now records the real figure and what reclaims it. The two sibling pipelines on this runner (cicd-smoke, otel-demo) were checked for the same bug and do not have it — they use `docker build <context>`, where the client streams the context to the daemon.	2026-07-20 04:13:30 +00:00
Nguyen Minh Phuc	2098443f31	ci: build images one at a time ... `image (api)` and `image (reconciler)` both died in run #13 with: ERROR: failed to solve: DeadlineExceeded: no active session for ylux5s1p4xyb5e5tmnz4gljhq: context deadline exceeded That is the buildkit session heartbeat between buildx in the job container and buildkitd in the dind sidecar, not anything in the Dockerfiles. The runner has capacity 2, so two legs of the matrix built concurrently on a 2-core burstable node already sitting at 81% CPU and 87% memory, and starved each other. The evidence is in the same run: `image (worker)` was still building when the other two failed, had the node to itself afterwards, and went green on an unchanged Dockerfile. max-parallel: 1 trades wall-clock for builds that finish. Every other gate in run #13 — lint, types, unit, integration, security, dockerfile, chart — was already green, so this is the last thing standing between here and a full run.	2026-07-20 04:11:04 +00:00
Nguyen Minh Phuc	b35c551160	ci: the digest-guard test depended on state CI itself mutates ... The chart job asserted that a bare `helm template` FAILS, on the assumption that values.yaml always holds all-zeros placeholder digests. That assumption dies the first time the bump job runs: bump commits real digests into values.yaml, so the bare render then succeeds and the assertion reports 'the guard is not guarding' about a guard that is fine. A test whose expected result flips depending on whether CI has run before is not a test. Now it feeds the guard four distinct bad digests explicitly with --set — all-zeros, a bare tag, right-prefix-wrong-length, and empty — and requires each to be rejected. Verified locally: all four rejected, and a well-formed digest still renders.	2026-07-20 03:48:34 +00:00
Nguyen Minh Phuc	f87d8d4d78	runbook: the 14-minute 'Set up job' failure and its cause ... Every job was failing at Set up job after ~14 minutes, then the first real step died at 0s. It read like a broken action; it was an empty dind image cache re-pulling the 1.6GB act job image on every run. dind has no volume for /var/lib/docker, so the cache lives in its writable layer and dies with every pod restart. A restart-looping runner therefore never keeps one. Warming it by hand took lint from failure to success with no code change.	2026-07-20 03:35:12 +00:00
Nguyen Minh Phuc	c537073c21	deps: everything to latest stable ... Python 3.12 -> 3.14, postgres 16 -> 18, uv 0.5.11 -> 0.11.29, trivy 0.58.1 -> 0.72.0, gitleaks 8.21.2 -> 8.30.1, yq 4.44.6 -> 4.53.3, and every action re-pinned to the SHA of its latest tag (checkout v7, setup-uv v8, buildx v4, login v4, hadolint v3.3.0). helm stays 3.21.3: already current for 3.x, and helm 4 is a breaking change, not a CVE fix. trivy mattered most. A vulnerability scanner fourteen minor versions behind is the one stale pin that hides all the others. ruff target-version is deliberately py313 while the runtime is 3.14. It controls the syntax the formatter may emit, and at py314 it rewrites 'except (A, B):' into PEP 758's unparenthesized form — which reads exactly like Python 2's 'except E, name:' and is a hard SyntaxError below 3.14. No semantic gain, real readability cost, in a repo meant to be read. Verified on 3.14: ruff, ruff format, mypy --strict, 166 tests, helm lint, bandit, pip-audit. The digest guard still rejects placeholder digests. Risk carried knowingly: the bumped actions run on node24. If act_runner only provides node20, every job fails at action startup and this commit is the revert.	2026-07-19 09:39:38 +00:00
Nguyen Minh Phuc	4544765ec5	test: scope the process-group assertion to this run's own children ... test_timeout_kills_the_whole_process_group asserted that `pgrep -f 'sleep 300'` returns nothing. That is machine-global: it matches a leftover from an earlier run of the same test, any unrelated 'sleep 300' on the box, and the shell running pgrep, whose own command line contains the pattern being searched for. Observed three spurious matches on a dev box, which produced a red run that looked like a Python 3.14 regression in the helm timeout kill. It was not — the same test failed identically on 3.12. A test that fails for reasons unrelated to the code is as useless as one that cannot fail. The sleep duration is now derived from the pid, so it cannot collide with another run, and the test asserts up front that its own pattern matches nothing before it starts. Verified: 3 consecutive passes on each of Python 3.12 and 3.14.	2026-07-19 09:39:38 +00:00
svcforge-ci	f2b159ef7e	ci: bump image digests to ca21b6e70d ... Built and scanned by ca21b6e70d. ArgoCD syncs from this commit. [skip ci]	2026-07-19 09:18:15 +00:00
Nguyen Minh Phuc	ca21b6e70d	ci: gitleaks was passing without scanning anything ... The secret-scanning gate has been green and meaningless. `docker run -v "$PWD:/repo"` resolves the bind source on the dind sidecar's daemon, not in the job container, so gitleaks received an empty mount: ERR [git] fatal: not a git repository (or any parent up to mount point /) ERR failed to scan Git repository error="stderr is not empty" INF scan completed in 35.2ms INF no leaks found <- exit 0 It logged the failure and exited 0. A scanner that reports success without looking is worse than no scanner. The 35ms runtime was the tell. Now a checksum-pinned binary, plus a `git rev-list --count HEAD` assertion so an unscannable checkout fails the job instead of passing it. Same root cause and same fix for yq in bump-digests.sh, which failed the bump job with "stat deploy/chart/values.yaml: no such file or directory". helm was fixed this way earlier. Nothing on this runner should mount $PWD into a container.	2026-07-19 09:11:22 +00:00
Nguyen Minh Phuc	dc51bd1d9d	ci: move the bump job's master guard from job level to step level ... bump skipped at 0s on a genuine push to master (event: push, head_branch: master) — before the image job it depends on had even started. The identical expression on the 'push by digest' STEP inside the matrix job evaluates true and runs, so the expression is fine; Gitea does not resolve a job-level if correctly when needs points at a matrix job. needs: [image] still orders it and still gates on all three legs. The steps carry the guard in the form this runner is known to evaluate. A PR now starts the job and no-ops every step, which is a few seconds for a guard that actually fires.	2026-07-19 09:00:43 +00:00
Nguyen Minh Phuc	78a0a6d500	runbook: record how the image gate was taken to zero findings ... trivy on the worker image: 39 findings (2 CRITICAL) -> 18 -> 5 -> 0. Verified in CI run #7 on 4a426db: 'Total: 0 (HIGH: 0, CRITICAL: 0)' for all three images. The last five lived in kubectl's vendored golang.org/x/net and Go stdlib, inside the newest kubectl published. No version cleared them; removing the binary did.	2026-07-19 08:48:51 +00:00
Nguyen Minh Phuc	4a426dbe50	worker: create namespaces with helm, drop the kubectl binary ... trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the newest kubectl that exists. No release clears them; upstream has not rebuilt against the patched Go yet. Chasing the version further has no end. kubectl was in that image for exactly one call: ensure_namespace before helm. `helm upgrade --install --create-namespace` does the same thing, idempotently, as part of the install it already runs. So the binary goes, and its vendored CVEs go with it. read_secret had no callers. Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since --create-namespace makes a bare one. Nothing reads that label today.	2026-07-18 13:05:22 +00:00
Nguyen Minh Phuc	a59d3848ae	ci: kubectl v1.35.3 -> v1.36.2 to clear CVE-2026-35469 ... trivy took the worker image from 39 findings (2 CRITICAL) to 18 (0 CRITICAL) after the previous bump. The residue is kubectl's vendored deps: k8s v1.35.x pins github.com/moby/spdystream v0.5.0, and the fix is in 0.5.1. v1.36.2 vendors 0.5.1. One minor ahead of the v1.35.3 API server, which is inside kubectl's supported +/-1 window. reconciler was already green because it ships helm only.	2026-07-18 12:43:36 +00:00
Nguyen Minh Phuc	a51227d952	ci: run helm from a checksum-pinned binary, not a container ... The chart gate failed with 'stat deploy/chart/Chart.yaml: no such file or directory' while the file plainly existed in the checkout. `docker run -v "$PWD:/repo"` is interpreted by the dind sidecar's daemon rather than by the job container, so the bind source has to exist on the daemon's side of that boundary. Downloading the binary removes the boundary entirely. Pinned by sha256 for the same reason every image here is pinned by digest.	2026-07-18 12:20:16 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	77d560ddae	ci: move buildx layer cache to the registry ... act_runner's cache PVC is 1Gi and also holds .runner, the runner's own registration file. --cache-to type=gha,mode=max for three images is several GB; filling that volume breaks the runner, not just the cache. act_runner also evicts by age with no size cap, so it fills whatever it is given. type=registry has no such limit and lives beside the images it caches. The uv cache still uses the runner's cache service, which is a few hundred MB. Registry login is no longer gated to master: the build now reads and writes the cache on every run. Pushing the release image keeps its own master-only gate.	2026-07-18 11:24:44 +00:00
Nguyen Minh Phuc	c9d0176bb3	ci: run trivy directly; document CI/CD setup in RUNBOOK ... trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache.	2026-07-17 11:05:45 +00:00
Nguyen Minh Phuc	851f8919a8	ci: fix coverage target, pip-audit scope, bandit config ... - --cov pointed at libs/svcforge_core/domain, a path that does not exist (the package nests one level deeper). Coverage measured 0.00% of the code. Use the module form, which is layout-independent. - pip-audit --strict cannot audit our own editable, not-on-PyPI packages. Audit the locked dependency set instead and keep --strict. - bandit re-reports B608/B104, which ruff's S ruleset already enforces with justified per-line noqa it cannot see. Skipped in config, with reasons. - registry host was git.oci-oci; it is gitea.oci-oci. - integration job set SVCFORGE_PG_DSN; conftest reads SVCFORGE_TEST_DSN.	2026-07-17 10:51:59 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00