45 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
svcforge-ci	51a1bb62c0	ci: bump image digests to 6974b3620f ... Built and scanned by 6974b3620f. ArgoCD syncs from this commit. [skip ci]	2026-07-21 05:59:55 +00:00
Nguyen Minh Phuc	6974b3620f	catalog: two tiny services, so the loop can be exercised on a full cluster ... The three existing entries are sized like real products: `elasticsearch` small asks for 1Gi and medium for 4Gi across three replicas. On this cluster that is a request that never schedules, so provisioning them proves something about the node and nothing about svcforge. podinfo 16Mi/10m — one small Go binary, no dependencies, no PVC nginx 32Mi/10m — recognisable, still small Both are addressed as `oci://`, which is load-bearing rather than cosmetic. An OCI chart is pulled by reference with no `helm repo add` first. The three existing entries name `bitnamilegacy/<chart>`, a classic repo alias that nothing in the worker image configures — so as written they cannot resolve at provision time. OCI is the form that works from a bare container, and it is why the e2e test already provisions podinfo. Chart versions were resolved against the real registries before committing (podinfo 6.7.1, nginx 25.0.14 / app 1.31.3) rather than guessed, since a wrong version fails only at provision time. Also updates the CLI ServiceType enum, the catalog test's expected set, and the service list in the OpenAPI description.	2026-07-21 05:47:41 +00:00
Nguyen Minh Phuc	bb8b14ef03	api: make the OpenAPI spec usable as documentation ... FastAPI already served /docs, /redoc and /openapi.json, and the ingress already passed them through — the mechanism was there, the content was not. The schema alone cannot tell a caller the three things they most need to know, and the route docstrings explain implementation reasoning to a maintainer rather than usage to a consumer. Added to the spec itself, so it travels with the API rather than living in a README the caller does not have: - An app description covering bearer auth, that every write is 202 + poll, the instance lifecycle, and the uniform {"code", "message"} error body. - Tag descriptions for `instances` and `ops`. - Field descriptions and worked examples on CreateInstanceRequest, InstanceResponse and ErrorBody, so /docs shows a valid payload instead of leaving callers to infer one. The bearer scheme was already exposed via HTTPBearer, which is what makes the Authorize button in /docs work; there is now a test asserting it stays, along with the description, the tags and the request example. Docs that are not tested rot silently, and this is the artifact other teams integrate against. README documents the three URLs and how to generate a client from the spec.	2026-07-21 05:33:44 +00:00
svcforge-ci	72296ace84	ci: bump image digests to e971e04d75 ... Built and scanned by e971e04d75. ArgoCD syncs from this commit. [skip ci]	2026-07-21 02:52:27 +00:00
Nguyen Minh Phuc	e971e04d75	fix: shared runtime helper must live where every image packages it ... services/_runtime.py crashed the worker and reconciler on boot with `ModuleNotFoundError: No module named 'services._runtime'`, while every unit and integration gate was green and the API rolled out fine. The cause is packaging, not code. Each service Dockerfile copies only its own `services/<svc>/` subdir — `services/` itself is a namespace package with no __init__.py, so a file added at the `services/` root is never copied into any image. Tests import from the source tree, where the file exists, so nothing below the image boundary could catch it. The API survived only because it does not import the helper. Moved to svcforge_core.runtime, which `COPY libs/ libs/` packages into every image, next to adapters/tempyaml.py for the same reason. Added an import smoke test to the image job: `docker run --entrypoint python <image> -c "import services.<svc>.main"` loads the whole transitive graph inside the built image and fails the build before the digest is pushed. This is the one check the test suite structurally cannot perform — it runs against source, the image is a different filesystem — and it is exactly the gap this bug fell through.	2026-07-21 02:40:38 +00:00
svcforge-ci	9c8d10ce1f	ci: bump image digests to 7079d6340f ... Built and scanned by 7079d6340f. ArgoCD syncs from this commit. [skip ci]	2026-07-21 02:25:04 +00:00
Nguyen Minh Phuc	7079d6340f	docs: bring RUNBOOK and ARCHITECTURE up to date ... The runner moved to node0 and the drift check reads release secrets via the Kubernetes API in-cluster; the docs still described node2 and `helm list`. - RUNBOOK: the durable image-cache fix is now the node0 hostPath store, not a node2 pin; /data is NFS RWX, so the Multi-Attach wait is gone. Cross-references entries 9 and 10. - ARCHITECTURE: the reconciler's edge to the cluster is "list releases", not "helm list" (helm is the out-of-cluster fallback). - ARCHITECTURE: the state diagram and its prose described fail() moving every dead-lettered instance to `failed`. Corrected to the per-kind behaviour — only provision fails the instance; deprovision stays `deleting` for retry, upgrade and verify stay `ready` — matching the fix in tasks.py.	2026-07-21 01:46:54 +00:00
Nguyen Minh Phuc	66eb6cb0ee	refactor: converge the patterns multiple authors left divergent ... The codebase was written by several agents and had the same concept done more than one way. This makes it read as one voice, with no behaviour change. Dedup, each to a single canonical form: - INSTANCE_COLUMNS: the 13-column instances SELECT list existed as _COLUMNS in instances.py and reconcile.py (byte-identical) and inlined a third time in the worker. One exported constant now. - Settings.runtime_dsn: the three entrypoints each chose between str(pg_dsn) and pg_dsn.unicode_string(). One property. - yaml_tempfile: helm._ValuesFile and k8s._ManifestFile were the same write-yaml-to-a-temp-dir context manager. One helper in adapters/tempyaml.py. - services/_runtime.py: sleep_or_stop and install_stop_signals were copied between the worker and reconciler loops. One module, so shutdown behaviour cannot drift between them. - k8s.ensure_namespace used MANAGED_BY_LABEL/VALUE from helm.py instead of a hardcoded literal, so the managed-by label has one definition. - SvcforgeError is now the root of every svcforge exception (CatalogError, IllegalTransition, BadWindow, HandlerError), keeping each stdlib base in the MRO, so `except SvcforgeError` means what errors.py says it does. - ERROR_MAX_CHARS replaces the repeated `[-2000:]` truncation feeding the same error columns. - the reconciler reads settings.metrics_port like the worker, dropping its duplicate DEFAULT_METRICS_PORT and redundant --metrics-port option; the SVCFORGE_METRICS_PORT env override still applies through pydantic. Two smaller correctness/consistency fixes: - RateLimitResult.retry_after_s computed its delta against datetime.now(UTC) while the limiter runs on an injectable clock, so it was meaningless under a FakeClock and drifted by request latency in production. It now carries a checked_at from the same clock as reset_at. - handle_provision's notifier.send is wrapped like the reconciler's: a flaky webhook after the READY CAS would fail the task, and the retry would hit the READY early-return and drop the notification, turning a good provision into a failed one.	2026-07-21 01:46:54 +00:00
Nguyen Minh Phuc	d64c3c9f39	fix: three correctness bugs found in review + a dropped-log ... fail() blanket-marked the instance `failed` for every dead-lettered task kind. Only provision is correct. The others each left the instance in a state the UPDATE then corrupted: - deprovision: `deleting` -> `failed` stranded the instance, because due_for_deprovision only re-selects ready/deleting, leaking the release reconcile.py promises to reclaim. - upgrade: helm --atomic rolled back, so the instance was still `ready` and serving the old version; `failed` mislabelled a healthy service and dropped it off the upgrade work-list. - verify: handle_verify already halted the rollout; the instance was `ready`. Now gated on kind == provision, with a regression test per kind (control-tested against the blanket UPDATE, which fails all three). The API exception handler was registered on fastapi.HTTPException, a subclass of starlette's. Starlette matches handlers by walking type(exc).__mro__, so framework-raised 404/405 never hit it and returned {"detail": ...} instead of ErrorBody. Registered on the starlette parent, and added a RequestValidationError handler so body-validation 422s share the shape too. Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field. helm._list_releases_via_api built the httpx client with verify=<ca path>, which loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an error the except clause did not catch, crashing the reconciler tick as a bare bug. Now the CA is checked for readability alongside the token, and a missing one means "not in-cluster" and falls back to helm. Also documents the no-limit pagination invariant and pins it with a test. redis.py logged through stdlib logging with extra={"team": team}, which the structlog bridge drops on the floor — the trap notify.py documents. Switched to a bound logger with team as a kwarg.	2026-07-21 01:46:54 +00:00
svcforge-ci	d6c1b64512	ci: bump image digests to 76cadca8e3 ... Built and scanned by 76cadca8e3. ArgoCD syncs from this commit. [skip ci]	2026-07-20 15:27:19 +00:00
Nguyen Minh Phuc	76cadca8e3	fix: Kubernetes serialises an empty item list as null, not [] ... The drift check failed on every tick with: TypeError: 'NoneType' object is not iterable `resp.json().get("items", [])` cannot defend against this. The key IS present, with the value null, so the default never fires. An empty PartialObjectMetadataList comes back as `"items": null`, which is exactly what happens once the release selector matches nothing — the normal state of a cluster with no tenant releases provisioned yet. `or []` handles both shapes. The tests could not have caught this: all of them passed a real empty list, which is the one shape that works. The new test sends `"items": null` as the API server actually sends it, and is control-tested — restoring the old expression fails it with the same TypeError seen in production. Worth recording why this survived review. The API read was verified against a live cluster before shipping, but that cluster had 25 matching release secrets, so the empty case never ran. A measurement on real data proved the fast path worked and said nothing about the path taken when there is no data.	2026-07-20 15:17:51 +00:00
svcforge-ci	66336d3648	ci: bump image digests to a843494627 ... Built and scanned by a843494627. ArgoCD syncs from this commit. [skip ci]	2026-07-20 15:01:21 +00:00
Nguyen Minh Phuc	a843494627	runbook: the runner's three ephemeral caches and the boot cascade ... Three caches on this runner were container-layer only, each found because something was slow: dind's image store, the trivy vuln DB, and act's action clones. All three now have real storage. act clones actions with full history, not shallow — 66.7MB/538 commits for setup-uv, 24.4MB/222 for actions/checkout, and this workflow uses five. After a restart that made `Set up job` an 11-minute step with the job container sat idle running `sleep` while the runner cloned GitHub. Includes the command to tell those two apart. Also records the cascade the dind fix creates. A persistent image store makes dockerd scan on boot — 38s, 2m13s, or over 5 minutes depending on node load — and two timeouts then fire: dind's startup probe, and the runner image's own hardcoded `Docker wait timeout of 5m0s`, which the chart cannot configure. The runner exits 1, restarts, and kills the running job, which looks like every step failing at once with no error after a green `Set up job`. It self-heals in about ten minutes at the cost of one CI run, so runner restarts are now something to do deliberately rather than casually.	2026-07-20 14:10:28 +00:00
Nguyen Minh Phuc	4193a18cae	reconciler: read release secrets directly instead of shelling helm ... helm's list gunzips every release payload to build its table. The only fields either caller reads are name and namespace: reconciler/main.py live = {(r.name, r.namespace) for r in releases} worker/handlers.py releases = {r.name for r in await ...list_releases()} Both live in the release secret's labels and metadata, so nothing needs decompressing. Measured from a pod on this cluster: 21ms and 31KB, against helm's 4392ms. That is the whole fix for a drift check that was timing out at 330s every tick and OOMKilling the container at 256Mi. The cause of both was decompressing 96 releases to extract two strings each. It also means the container no longer cares that a cluster policy mutates its CPU request to 0 — at 21ms there is nothing left to starve. Three details carry the correctness, each with a test: - PartialObjectMetadataList in the Accept header asks for metadata only. Without it every release's gzipped manifest crosses the wire to be thrown away, which is the cost this exists to avoid. - The status selector drops superseded revisions server-side: 96 release secrets here, 25 live. Failed and pending states are kept, matching what `helm list` shows, because a failed release does exist and calling it missing would have the reconciler re-provision on top of it. - helm writes one secret per revision, up to 10 per release here, so the newest version label wins. Control-tested with a string compare, which picks "9" over "10". Out of cluster there is no ServiceAccount, so it falls back to helm and the e2e suite and laptop runs are unchanged. An API error raises rather than falling back: the fallback is for a known-absent ServiceAccount, and quietly retrying through helm would swap a visible error for the timeout this removes. chart and app_version are empty on this path rather than wrong — they live only in the compressed payload, and nothing reads them.	2026-07-20 13:05:25 +00:00
Nguyen Minh Phuc	2356ac4ef3	reconciler: scope the drift check to svcforge's own releases ... check_drift diffs helm against the database in both directions, and the second one is `live - known` -> drift.orphan_release at ERROR. `live` was every release in the cluster, so argocd, longhorn, gitea and cert-manager were all reported as orphans svcforge is failing to account for, on every sweep. They are not orphans; they were never svcforge's to know about. install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases() selects on it. Releases provisioned before this see one sweep as missing and get re-provisioned, which is safe by construction — `helm upgrade --install` against a deterministic release name — and that re-provision applies the label. This does NOT fix the timeout, and the docstring says so. Measured, not assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is not pushed down as a server-side selector, so helm still fetches and decompresses every release secret and filters what it already parsed. Around 10%, not the order of magnitude the flag's shape suggests. The 330s timeouts need CPU for the container or a different read path. The two sides of the label are asserted against each other rather than a literal, so a rename that updates only one fails in tests instead of in production as a silently empty drift check. Control-tested: renaming the read side alone fails two of the three.	2026-07-20 12:53:16 +00:00
Nguyen Minh Phuc	c2a27952d1	runbook: fix section ordering and a duplicate number ... The dind entry landed ahead of the postgres one, and the postgres entry I added earlier was numbered 7 while 'Verify the whole loop' already was. Now 7 postgres, 8 dind, 9 verify.	2026-07-20 07:39:23 +00:00
Nguyen Minh Phuc	0dbb5af1d3	runbook: record the dind liveness probe as an open issue ... dind is killed by a probe whose only job is to check a socket exists, with a 1s timeout the nodes cannot always meet: 27 failures over 156 minutes. Worth recording because it probably explains build failures already attributed to something else. `DeadlineExceeded: no active session` was blamed on CPU starvation and addressed by dropping runner capacity to 1; the likelier mechanism is kubelet killing dind mid-build and taking the buildkit session with it. Capacity reduced the load that trips the probe, which fits #17 passing and #18-#21 failing anyway. Written as a hypothesis with the command to confirm it, not as a conclusion. The chart exposes no probe knobs, so the candidate fix is a Kyverno mutation in Ansible, following the existing force-best-effort-cpu precedent.	2026-07-20 07:38:51 +00:00
Nguyen Minh Phuc	d4ac3801a3	runbook: salvaging a faulted Longhorn volume ... The documented scale 0/1 recovery did not work this time: the volume returned detached/faulted and refused to attach, so the pod sat in ContainerCreating. auto-salvage: true cannot rescue it, because salvage happens during attach and a faulted volume never gets that far. The replica's failedAt timestamp is the only thing holding it faulted. Clearing it brought the volume to attached/healthy and postgres to 1/1 in under a minute, with repo data and CI history intact. Adds the backup check first, which matters more now that Longhorn runs at one replica and there is no second copy to fall back on.	2026-07-20 07:10:04 +00:00
Nguyen Minh Phuc	6093c52160	argocd: ignore the cpu request this cluster rewrites ... A Kyverno ClusterPolicy, force-best-effort-cpu, rewrites every container's CPU request to "0" at admission. It is deliberate and predates this app by well over a year: the nodes are oversubscribed and BestEffort-on-CPU is how everything gets scheduled. The chart asks for 50m and the cluster writes 0, so all three Deployments sat permanently OutOfSync while perfectly Healthy — the failure mode where a dashboard is always yellow, everyone learns to ignore it, and it stops meaning anything the day it goes yellow for a real reason. Ignored on the Application rather than capitulating in the chart. What the chart asks for is the honest intent; what the cluster does with it is the cluster's business, and a reader of the repo should see the former. jqPathExpressions rather than jsonPointers, which would have to name a container index and this has to hold for every container in all three Deployments. Takes effect only after a hard refresh.	2026-07-20 06:58:28 +00:00
Nguyen Minh Phuc	31fa9165ff	chart: give the migrate hook its own ServiceAccount ... The Job ran as the api ServiceAccount, which is an ordinary chart resource. Hooks are created before the release's ordinary manifests, so on a first install that account does not exist and the Job never starts: Error creating: pods "svcforge-migrate-" is forbidden: error looking up service account svcforge/svcforge-api: serviceaccount "svcforge-api" not found Not an ArgoCD quirk — helm orders hooks the same way, so both deploy paths failed identically. It survived review because the chart was only ever checked with `helm template` and `helm install --dry-run=server`, and neither creates a Job. The pod is what fails, so only a real install can catch it. The new account is a hook at weight -10, ahead of the Job at -5, and is bound to no Role: the migration talks to Postgres and wants nothing from the Kubernetes API. automountServiceAccountToken is off for the same reason. Verified with a real `helm install` into a scratch namespace: STATUS deployed, job Complete 1/1, four migrations applied, and the hook ServiceAccount was 27s old against 10s for the ordinary ones — the ordering the bug turned on.	2026-07-20 06:33:31 +00:00
Nguyen Minh Phuc	89ad9625f3	chart: disable externalSecret on this cluster ... The block describes a ClusterSecretStore named `vault` with HashiCorp-style key/property refs. This cluster has `oci-vault` — OCI Vault via InstancePrincipal — whose provider addresses a secret by name and takes a JSON property, so those remoteRefs do not translate as written. There are no ExternalSecrets anywhere on the cluster, so the path had never been exercised. svcforge.secretName still resolves through targetName, so the deployments and the migrate hook read a Secret named svcforge-secrets, created out of band from ~/.config/svcforge/secrets.env. This is a deviation and the comment says so. ArgoCD does not manage that Secret, so prune and selfHeal cannot touch it, and it is the one part of the deployment that cannot be read from git. Restoring the intended design means adding oci_vault_secret resources to oci-k8s/infra/vault.tf and repointing secretStoreRef at oci-vault.	2026-07-20 06:26:13 +00:00
Nguyen Minh Phuc	e8b6116a58	ci: age-filter the image prune so digest-pulled images survive ... Run #17 showed `docker image prune -f` deleting the trivy image, which buys back a 157MB pull on every subsequent run. An image pulled by digest carries no tag, so it is dangling the moment its container exits, and bare dangling-only pruning is not as narrow as it reads. The act runner image survived the same prune only by accident: this step runs inside an act container, so that image happens to be in use exactly while the prune executes. That is luck, not design, and it would not hold if the step ever moved. An until=168h filter on both prunes keeps a week of recently used layers, matching the buildx cache window already in place. Verified in run #17 before this change: trivy logged "Need to update DB" zero times, so the named-volume DB cache works, and registry digests equal the chart's for all three images.	2026-07-20 05:17:13 +00:00
Nguyen Minh Phuc	5f18f9eeeb	runbook: correct the claim that runner restarts orphan jobs ... The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel.	2026-07-20 05:17:13 +00:00
svcforge-ci	1a36f43c69	ci: bump image digests to 37b297bf5c ... Built and scanned by 37b297bf5c. ArgoCD syncs from this commit. [skip ci]	2026-07-20 05:12:26 +00:00
Nguyen Minh Phuc	37b297bf5c	ci: reclaim dind disk after each image build ... dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed.	2026-07-20 04:48:48 +00:00
Nguyen Minh Phuc	5d7f46483e	runbook: recovering the queue after a runner restart ... Restarting act_runner leaves its in-flight jobs in_progress with nothing behind them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run index, not the database id.	2026-07-20 04:41:15 +00:00
Nguyen Minh Phuc	b02d4e85c6	ci: record that act_runner ignores max-parallel ... The previous commit claimed max-parallel: 1 would stop two image builds from starving each other. It does not — act_runner ignores strategy.max-parallel. Run #14 had it set and still ran the worker leg from 04:25:47 while reconciler was still building, after api had run alone from 04:23:03 to 04:24:04. That run failed a different way: actions/checkout could not reach Gitea at all (`Failed to connect to gitea-http:3000 after 3105 ms`) while two runs were in flight, with node0 at 140% memory. Same root cause, new mask. The setting stays, since it is correct on runners that honour it, but the comment no longer claims it does anything here. The binding lever is the runner's `capacity`, dropped 2 -> 1 in oci-k8s.	2026-07-20 04:27:48 +00:00
Nguyen Minh Phuc	d70a7b622c	ci: cache the trivy vuln DB in a named volume, not a $PWD bind ... `-v "$PWD/.trivycache:/root/.cache/trivy"` was the third instance of the bug that made gitleaks scan nothing. $PWD is a path in the job container, but the -v is resolved by the daemon in the dind sidecar, which has no such directory and silently creates an empty one. Every run logged `[vulndb] Need to update DB` and re-downloaded the DB, and the cache it wrote went to a throwaway directory inside dind. This file warns about the same trap in two other places. A named volume lives in the dind daemon's own storage, which is the one thing both sides agree on. Verified in dind rather than assumed: cold pass logs `Need to update DB` / `Downloading vulnerability DB` / `Artifact successfully downloaded` and leaves 1.1G in the volume; warm pass logs none of them. The old comment's "~50MB" was wrong by 20x, so the comment now records the real figure and what reclaims it. The two sibling pipelines on this runner (cicd-smoke, otel-demo) were checked for the same bug and do not have it — they use `docker build <context>`, where the client streams the context to the daemon.	2026-07-20 04:13:30 +00:00
Nguyen Minh Phuc	2098443f31	ci: build images one at a time ... `image (api)` and `image (reconciler)` both died in run #13 with: ERROR: failed to solve: DeadlineExceeded: no active session for ylux5s1p4xyb5e5tmnz4gljhq: context deadline exceeded That is the buildkit session heartbeat between buildx in the job container and buildkitd in the dind sidecar, not anything in the Dockerfiles. The runner has capacity 2, so two legs of the matrix built concurrently on a 2-core burstable node already sitting at 81% CPU and 87% memory, and starved each other. The evidence is in the same run: `image (worker)` was still building when the other two failed, had the node to itself afterwards, and went green on an unchanged Dockerfile. max-parallel: 1 trades wall-clock for builds that finish. Every other gate in run #13 — lint, types, unit, integration, security, dockerfile, chart — was already green, so this is the last thing standing between here and a full run.	2026-07-20 04:11:04 +00:00
Nguyen Minh Phuc	b35c551160	ci: the digest-guard test depended on state CI itself mutates ... The chart job asserted that a bare `helm template` FAILS, on the assumption that values.yaml always holds all-zeros placeholder digests. That assumption dies the first time the bump job runs: bump commits real digests into values.yaml, so the bare render then succeeds and the assertion reports 'the guard is not guarding' about a guard that is fine. A test whose expected result flips depending on whether CI has run before is not a test. Now it feeds the guard four distinct bad digests explicitly with --set — all-zeros, a bare tag, right-prefix-wrong-length, and empty — and requires each to be rejected. Verified locally: all four rejected, and a well-formed digest still renders.	2026-07-20 03:48:34 +00:00
Nguyen Minh Phuc	f87d8d4d78	runbook: the 14-minute 'Set up job' failure and its cause ... Every job was failing at Set up job after ~14 minutes, then the first real step died at 0s. It read like a broken action; it was an empty dind image cache re-pulling the 1.6GB act job image on every run. dind has no volume for /var/lib/docker, so the cache lives in its writable layer and dies with every pod restart. A restart-looping runner therefore never keeps one. Warming it by hand took lint from failure to success with no code change.	2026-07-20 03:35:12 +00:00
Nguyen Minh Phuc	c537073c21	deps: everything to latest stable ... Python 3.12 -> 3.14, postgres 16 -> 18, uv 0.5.11 -> 0.11.29, trivy 0.58.1 -> 0.72.0, gitleaks 8.21.2 -> 8.30.1, yq 4.44.6 -> 4.53.3, and every action re-pinned to the SHA of its latest tag (checkout v7, setup-uv v8, buildx v4, login v4, hadolint v3.3.0). helm stays 3.21.3: already current for 3.x, and helm 4 is a breaking change, not a CVE fix. trivy mattered most. A vulnerability scanner fourteen minor versions behind is the one stale pin that hides all the others. ruff target-version is deliberately py313 while the runtime is 3.14. It controls the syntax the formatter may emit, and at py314 it rewrites 'except (A, B):' into PEP 758's unparenthesized form — which reads exactly like Python 2's 'except E, name:' and is a hard SyntaxError below 3.14. No semantic gain, real readability cost, in a repo meant to be read. Verified on 3.14: ruff, ruff format, mypy --strict, 166 tests, helm lint, bandit, pip-audit. The digest guard still rejects placeholder digests. Risk carried knowingly: the bumped actions run on node24. If act_runner only provides node20, every job fails at action startup and this commit is the revert.	2026-07-19 09:39:38 +00:00
Nguyen Minh Phuc	4544765ec5	test: scope the process-group assertion to this run's own children ... test_timeout_kills_the_whole_process_group asserted that `pgrep -f 'sleep 300'` returns nothing. That is machine-global: it matches a leftover from an earlier run of the same test, any unrelated 'sleep 300' on the box, and the shell running pgrep, whose own command line contains the pattern being searched for. Observed three spurious matches on a dev box, which produced a red run that looked like a Python 3.14 regression in the helm timeout kill. It was not — the same test failed identically on 3.12. A test that fails for reasons unrelated to the code is as useless as one that cannot fail. The sleep duration is now derived from the pid, so it cannot collide with another run, and the test asserts up front that its own pattern matches nothing before it starts. Verified: 3 consecutive passes on each of Python 3.12 and 3.14.	2026-07-19 09:39:38 +00:00
svcforge-ci	f2b159ef7e	ci: bump image digests to ca21b6e70d ... Built and scanned by ca21b6e70d. ArgoCD syncs from this commit. [skip ci]	2026-07-19 09:18:15 +00:00
Nguyen Minh Phuc	ca21b6e70d	ci: gitleaks was passing without scanning anything ... The secret-scanning gate has been green and meaningless. `docker run -v "$PWD:/repo"` resolves the bind source on the dind sidecar's daemon, not in the job container, so gitleaks received an empty mount: ERR [git] fatal: not a git repository (or any parent up to mount point /) ERR failed to scan Git repository error="stderr is not empty" INF scan completed in 35.2ms INF no leaks found <- exit 0 It logged the failure and exited 0. A scanner that reports success without looking is worse than no scanner. The 35ms runtime was the tell. Now a checksum-pinned binary, plus a `git rev-list --count HEAD` assertion so an unscannable checkout fails the job instead of passing it. Same root cause and same fix for yq in bump-digests.sh, which failed the bump job with "stat deploy/chart/values.yaml: no such file or directory". helm was fixed this way earlier. Nothing on this runner should mount $PWD into a container.	2026-07-19 09:11:22 +00:00
Nguyen Minh Phuc	dc51bd1d9d	ci: move the bump job's master guard from job level to step level ... bump skipped at 0s on a genuine push to master (event: push, head_branch: master) — before the image job it depends on had even started. The identical expression on the 'push by digest' STEP inside the matrix job evaluates true and runs, so the expression is fine; Gitea does not resolve a job-level if correctly when needs points at a matrix job. needs: [image] still orders it and still gates on all three legs. The steps carry the guard in the form this runner is known to evaluate. A PR now starts the job and no-ops every step, which is a few seconds for a guard that actually fires.	2026-07-19 09:00:43 +00:00
Nguyen Minh Phuc	78a0a6d500	runbook: record how the image gate was taken to zero findings ... trivy on the worker image: 39 findings (2 CRITICAL) -> 18 -> 5 -> 0. Verified in CI run #7 on 4a426db: 'Total: 0 (HIGH: 0, CRITICAL: 0)' for all three images. The last five lived in kubectl's vendored golang.org/x/net and Go stdlib, inside the newest kubectl published. No version cleared them; removing the binary did.	2026-07-19 08:48:51 +00:00
Nguyen Minh Phuc	4a426dbe50	worker: create namespaces with helm, drop the kubectl binary ... trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the newest kubectl that exists. No release clears them; upstream has not rebuilt against the patched Go yet. Chasing the version further has no end. kubectl was in that image for exactly one call: ensure_namespace before helm. `helm upgrade --install --create-namespace` does the same thing, idempotently, as part of the install it already runs. So the binary goes, and its vendored CVEs go with it. read_secret had no callers. Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since --create-namespace makes a bare one. Nothing reads that label today.	2026-07-18 13:05:22 +00:00
Nguyen Minh Phuc	a59d3848ae	ci: kubectl v1.35.3 -> v1.36.2 to clear CVE-2026-35469 ... trivy took the worker image from 39 findings (2 CRITICAL) to 18 (0 CRITICAL) after the previous bump. The residue is kubectl's vendored deps: k8s v1.35.x pins github.com/moby/spdystream v0.5.0, and the fix is in 0.5.1. v1.36.2 vendors 0.5.1. One minor ahead of the v1.35.3 API server, which is inside kubectl's supported +/-1 window. reconciler was already green because it ships helm only.	2026-07-18 12:43:36 +00:00
Nguyen Minh Phuc	a51227d952	ci: run helm from a checksum-pinned binary, not a container ... The chart gate failed with 'stat deploy/chart/Chart.yaml: no such file or directory' while the file plainly existed in the checkout. `docker run -v "$PWD:/repo"` is interpreted by the dind sidecar's daemon rather than by the job container, so the bind source has to exist on the daemon's side of that boundary. Downloading the binary removes the boundary entirely. Pinned by sha256 for the same reason every image here is pinned by digest.	2026-07-18 12:20:16 +00:00
Nguyen Minh Phuc	c76154aeaa	review: fix 26 findings from a 4-agent audit ... CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.	2026-07-18 12:13:49 +00:00
Nguyen Minh Phuc	77d560ddae	ci: move buildx layer cache to the registry ... act_runner's cache PVC is 1Gi and also holds .runner, the runner's own registration file. --cache-to type=gha,mode=max for three images is several GB; filling that volume breaks the runner, not just the cache. act_runner also evicts by age with no size cap, so it fills whatever it is given. type=registry has no such limit and lives beside the images it caches. The uv cache still uses the runner's cache service, which is a few hundred MB. Registry login is no longer gated to master: the build now reads and writes the cache on every run. Pushing the release image keeps its own master-only gate.	2026-07-18 11:24:44 +00:00
Nguyen Minh Phuc	c9d0176bb3	ci: run trivy directly; document CI/CD setup in RUNBOOK ... trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache.	2026-07-17 11:05:45 +00:00
Nguyen Minh Phuc	851f8919a8	ci: fix coverage target, pip-audit scope, bandit config ... - --cov pointed at libs/svcforge_core/domain, a path that does not exist (the package nests one level deeper). Coverage measured 0.00% of the code. Use the module form, which is layout-independent. - pip-audit --strict cannot audit our own editable, not-on-PyPI packages. Audit the locked dependency set instead and keep --strict. - bandit re-reports B608/B104, which ruff's S ruleset already enforces with justified per-line noqa it cannot see. Skipped in config, with reasons. - registry host was git.oci-oci; it is gitea.oci-oci. - integration job set SVCFORGE_PG_DSN; conftest reads SVCFORGE_TEST_DSN.	2026-07-17 10:51:59 +00:00
Nguyen Minh Phuc	50c2fe2a1e	svcforge: reference implementation ... Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%.	2026-07-17 10:44:54 +00:00