65 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
svcforge-ci	fca9a8b187	ci: bump image digests to 7a3dfb06db ... Built and scanned by 7a3dfb06db. ArgoCD syncs from this commit. [skip ci]	2026-07-22 06:28:37 +00:00
gitea_admin	7a3dfb06db	ci: exercise the runner CPU limit and webhook end to end ... A full run under the new 600m cap, to confirm the checkout no longer loses its race with the build for node1's single core, and that the resulting digest bump deploys through the webhook without a nudge.	2026-07-22 06:18:22 +00:00
gitea_admin	d40bc682ce	runbook: the Gitea webhook now covers the broken poll [skip ci] ... Registered as gogs type on all three ArgoCD-tracked repos, secret shared with argocd-secret's webhook.gogs.secret. Records how to read a failed delivery from both ends, and that a drifted secret fails validation silently -- which looks exactly like having no webhook.	2026-07-22 06:17:32 +00:00
svcforge-ci	cbd0709281	ci: bump image digests to cd61eca12e ... Built and scanned by cd61eca12e. ArgoCD syncs from this commit. [skip ci]	2026-07-22 03:58:22 +00:00
gitea_admin	cd61eca12e	runbook: correct 1b56cda -- the poll never runs, watch breaks are not the cause ... 1b56cda blamed the Application-informer watch break after an API server kill. That is not it. Watched a freshly restarted controller for 11 minutes on a fully healthy API server: 03:37:03 startup refresh, 3 apps adds_total = 3 03:38:31 adds_total = 3 03:41:33 adds_total = 3 03:44:34 adds_total = 3 03:47:35 adds_total = 3 expiry is 2m0s, jitter 60s The periodic poll does not fire, ever, with or without a prior disruption. The 68 adds the earlier pod accumulated were cluster events during a rollout, and it went flat the moment the cluster quieted -- same shape, no watch break involved. Refreshes come from startup and from cluster events only. A commit that changes just the repo is never noticed. Every "auto-sync" seen on 2026-07-22 landed within seconds of a controller restart, which is the startup refresh -- I read one of those as proof the pipeline worked, and it was not. Also records that no Gitea webhook exists, so nothing covers for the broken poll. Wiring one is worth doing regardless of whether the poll is ever repaired.	2026-07-22 03:49:04 +00:00
gitea_admin	1b56cda231	runbook: name the ArgoCD wedge instead of describing its symptoms ... Reproduced three times on 2026-07-22, and the controller's own workqueue metrics say what it is: workqueue_depth 0 workqueue_longest_running_processor_seconds 0 workqueue_adds_total 68 <- frozen Empty queue, idle processors, no new adds. Nothing is blocked -- nothing is being enqueued. The periodic app-resync timer stops firing, so no Application is ever queued for refresh again. Every occurrence followed an API server disruption, with the matching informer line in the logs: watch ended with error, http2: client connection lost, on *v1alpha1.Application. The informer re-lists; the resync does not resume. That is why the controller reads healthy from every angle except reconcile count, and why only a restart clears it. On a single-control-plane cluster anything that kills kube-apiserver -- including memory reclaim stalling /livez past its 80s threshold -- can silently stop all GitOps.	2026-07-22 03:37:22 +00:00
svcforge-ci	173acc8612	ci: bump image digests to 60ee0f1cbf ... Built and scanned by 60ee0f1cbf. ArgoCD syncs from this commit. [skip ci]	2026-07-22 03:15:59 +00:00
gitea_admin	60ee0f1cbf	runbook: correct the wedge-detection metrics ... c691f4f told you to check argocd_redis_request_total and workqueue_unfinished_work_seconds. Both read healthy through an 82-minute wedge on 2026-07-22 -- 45 cache reads per 15m and a queue depth of zero, while argocd_app_reconcile_count sat flat at 62 and three Applications were two commits behind master. The cache counter measures that the process is running. The queue gauge reads zero because this wedge blocks goroutines outside the queue, so nothing is in flight and nothing is being enqueued. argocd_app_reconcile_count is the one that went flat, and it is the one to check. The other two are now in the table of things that look like answers and are not, with what each read during the outage.	2026-07-22 02:59:54 +00:00
svcforge-ci	9249051ee2	ci: bump image digests to c691f4f4aa ... Built and scanned by c691f4f4aa. ArgoCD syncs from this commit. [skip ci]	2026-07-22 02:42:54 +00:00
gitea_admin	c691f4f4aa	runbook: reconciledAt is not a heartbeat ... The stuck-deploy section printed `.status.reconciledAt` as a diagnostic without saying how to read it, which invites exactly the wrong conclusion: ArgoCD only writes that field when the computed status changes, so on an idle cluster it stops advancing while the controller is healthy. Measured 2026-07-22 -- six samples 50s apart against a 120s reconciliation timeout, zero movement. It only means something alongside a sync.revision that is behind master. Replaces the "is it doing anything" step with two metrics that answer it directly: the controller's Redis cache reads, which happen every refresh cycle regardless of change, and the reconcile queue's unfinished-work seconds, which is precisely what the 2026-07-21 webhook wedge looked like. Log-based check kept as the no-Prometheus fallback.	2026-07-22 02:25:10 +00:00
gitea_admin	a0215ef63f	runbook: ArgoCD synced-but-stale, and how to tell a wedged controller from a slow poll ... Cost 11 hours yesterday and looked like nothing was wrong, because the Application reported Synced the whole time — at a revision five commits behind. Records the diagnosis that actually works (log-lines-per-hour, and a flat goroutine count meaning blocked rather than idle), the Kyverno failurePolicy root cause, and the two escalating fixes. Folds in the hook-finalizer deadlock, which has now happened four times and was only written down in chat.	2026-07-22 01:23:26 +00:00
svcforge-ci	b2cdcecdc1	ci: bump image digests to e7496562b3 ... Built and scanned by e7496562b3. ArgoCD syncs from this commit. [skip ci]	2026-07-22 01:08:19 +00:00
Nguyen Minh Phuc	e7496562b3	ci: pull the BuildKit frontend from the mirror too ... Run #70's logs still showed one Docker Hub reference, from the one place that does not look like an image reference: #2 resolve image config for docker-image://docker.io/docker/dockerfile:1.10 `# syntax=` is an image pull. Same mirror, same digest, and pinned now — it was the only unpinned image left in the build.	2026-07-21 15:50:20 +00:00
svcforge-ci	3b957a8566	ci: bump image digests to 08a529fa63 ... Built and scanned by 08a529fa63. ArgoCD syncs from this commit. [skip ci]	2026-07-21 15:42:05 +00:00
Nguyen Minh Phuc	08a529fa63	Get off Docker Hub, add a catalog values passthrough, fix the dind prune ... Docker Hub rate-limits anonymous pulls per source IP and every node here shares one NAT address, so a busy afternoon fails an unrelated build with `toomanyrequests`. Nothing in this repo needs to be there. Every base image now comes from mirror.gcr.io (python, alpine/helm, postgres) or ghcr.io (uv, trivy). Verified digest-for-digest against Docker Hub before switching, including the superseded postgres digest this repo still pins, so every existing pin stays valid — same bytes, different transport. catalog.yaml: the three bitnami entries named `bitnamilegacy/<chart>`, a repo alias nothing in the worker image configures, so they could never resolve at provision time. All five entries are now `oci://` refs, which need no `helm repo add`, and all are on latest stable: elasticsearch 21.3.15 -> 22.1.6 redis 20.6.2 -> 27.0.15 postgresql 16.4.5 -> 18.8.0 podinfo 6.7.1 -> 6.14.0 Moving the chart pull is only half of it, though: a bitnami chart defaults its own images to registry-1.docker.io. CatalogEntry gains a `values:` dict, merged under the size's replicas and resources, so an entry can set `global.imageRegistry` and move the image pull too. Size wins on conflict — otherwise an entry setting replicaCount would make every size deploy the same shape. Deep merge, because a shallow one drops sibling keys of a shared nested map. Bitnami charts reject a substituted registry unless `global.security.allowInsecureImages` is set. That check is about provenance, and the mirror serves byte-identical manifests, so it is set deliberately and only for entries whose digests were verified. The dind prune had `--filter until=168h` on both prunes, and it got both cases exactly backwards. `until` reads an image's CREATED time, so it deleted trivy every leg (a released tool image is always older than any window) while protecting the dangling build layers it existed to remove. Measured on node0: 21 dangling images / 5.96GB, and exactly 1 of them older than 168h. Trivy is protected by a tag now, so the image prune drops the filter; buildx keeps it, where age genuinely matters. Tests: +10 unit (deep merge, precedence, no-mutation, and a guard that fails if any catalog entry points at Docker Hub). Both new guards were control-tested by breaking the code and watching them fail. The API test that hardcoded `21.3.15` now reads the catalog — its subject is where the value comes from, not what it is.	2026-07-21 15:32:12 +00:00
Nguyen Minh Phuc	58ffb9c2e0	ci: tag the trivy image so the reclaim step stops deleting it ... Run #68 re-pulled trivy in all three matrix legs — 178MB each, ~7s each — because the reclaim step deletes it at the end of every leg: Unable to find image 'aquasec/trivy:0.72.0@sha256:cffe...' locally untagged: aquasec/trivy@sha256:cffe... Total reclaimed space: 178.6MB The `--filter until=168h` I added for exactly this reason does not work. That filter reads the image's CREATED timestamp, not the pull time, and aquasec/trivy:0.72.0 was built months ago, so it matches immediately. The comment claiming the age filter fixed this was wrong; corrected in place. Pull by digest, tag it, run the tag. A tagged image is not dangling, which is what actually takes it out of `docker image prune`'s scope. The digest is still the pin — enforced at pull time. Everything else in run #68 is clean: 76 unit (98.59% coverage), 112 integration, 18 CACHED layers per image job, zero uv hardlink warnings, and the trivy DB volume hit on 2 of 3 legs (one download, two silent).	2026-07-21 15:11:27 +00:00
svcforge-ci	95a894d816	ci: bump image digests to c53734d2bc ... Built and scanned by c53734d2bc. ArgoCD syncs from this commit. [skip ci]	2026-07-21 15:05:04 +00:00
Nguyen Minh Phuc	c53734d2bc	docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN ... The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.	2026-07-21 14:51:26 +00:00
svcforge-ci	7918dc2b37	ci: bump image digests to 64b18d2823 ... Built and scanned by 64b18d2823. ArgoCD syncs from this commit. [skip ci]	2026-07-21 06:33:09 +00:00
Nguyen Minh Phuc	64b18d2823	ci: set UV_LINK_MODE=copy to stop the hardlink warning ... Every uv job logged this five times a run: warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. The uv cache lives on the runner's cache volume and the venv is on the job container's filesystem, so hardlinking cannot work and uv copies all 86 packages regardless. Declaring `copy` does not make anything slower — it is already copying — it removes a warning that appears in every green run, and a warning nobody can act on is one that teaches people to skim CI logs. Found by reading the logs of a passing run rather than a failing one. The Dockerfiles already set this; CI did not.	2026-07-21 06:02:32 +00:00
svcforge-ci	51a1bb62c0	ci: bump image digests to 6974b3620f ... Built and scanned by 6974b3620f. ArgoCD syncs from this commit. [skip ci]	2026-07-21 05:59:55 +00:00
Nguyen Minh Phuc	6974b3620f	catalog: two tiny services, so the loop can be exercised on a full cluster ... The three existing entries are sized like real products: `elasticsearch` small asks for 1Gi and medium for 4Gi across three replicas. On this cluster that is a request that never schedules, so provisioning them proves something about the node and nothing about svcforge. podinfo 16Mi/10m — one small Go binary, no dependencies, no PVC nginx 32Mi/10m — recognisable, still small Both are addressed as `oci://`, which is load-bearing rather than cosmetic. An OCI chart is pulled by reference with no `helm repo add` first. The three existing entries name `bitnamilegacy/<chart>`, a classic repo alias that nothing in the worker image configures — so as written they cannot resolve at provision time. OCI is the form that works from a bare container, and it is why the e2e test already provisions podinfo. Chart versions were resolved against the real registries before committing (podinfo 6.7.1, nginx 25.0.14 / app 1.31.3) rather than guessed, since a wrong version fails only at provision time. Also updates the CLI ServiceType enum, the catalog test's expected set, and the service list in the OpenAPI description.	2026-07-21 05:47:41 +00:00
Nguyen Minh Phuc	bb8b14ef03	api: make the OpenAPI spec usable as documentation ... FastAPI already served /docs, /redoc and /openapi.json, and the ingress already passed them through — the mechanism was there, the content was not. The schema alone cannot tell a caller the three things they most need to know, and the route docstrings explain implementation reasoning to a maintainer rather than usage to a consumer. Added to the spec itself, so it travels with the API rather than living in a README the caller does not have: - An app description covering bearer auth, that every write is 202 + poll, the instance lifecycle, and the uniform {"code", "message"} error body. - Tag descriptions for `instances` and `ops`. - Field descriptions and worked examples on CreateInstanceRequest, InstanceResponse and ErrorBody, so /docs shows a valid payload instead of leaving callers to infer one. The bearer scheme was already exposed via HTTPBearer, which is what makes the Authorize button in /docs work; there is now a test asserting it stays, along with the description, the tags and the request example. Docs that are not tested rot silently, and this is the artifact other teams integrate against. README documents the three URLs and how to generate a client from the spec.	2026-07-21 05:33:44 +00:00
svcforge-ci	72296ace84	ci: bump image digests to e971e04d75 ... Built and scanned by e971e04d75. ArgoCD syncs from this commit. [skip ci]	2026-07-21 02:52:27 +00:00
Nguyen Minh Phuc	e971e04d75	fix: shared runtime helper must live where every image packages it ... services/_runtime.py crashed the worker and reconciler on boot with `ModuleNotFoundError: No module named 'services._runtime'`, while every unit and integration gate was green and the API rolled out fine. The cause is packaging, not code. Each service Dockerfile copies only its own `services/<svc>/` subdir — `services/` itself is a namespace package with no __init__.py, so a file added at the `services/` root is never copied into any image. Tests import from the source tree, where the file exists, so nothing below the image boundary could catch it. The API survived only because it does not import the helper. Moved to svcforge_core.runtime, which `COPY libs/ libs/` packages into every image, next to adapters/tempyaml.py for the same reason. Added an import smoke test to the image job: `docker run --entrypoint python <image> -c "import services.<svc>.main"` loads the whole transitive graph inside the built image and fails the build before the digest is pushed. This is the one check the test suite structurally cannot perform — it runs against source, the image is a different filesystem — and it is exactly the gap this bug fell through.	2026-07-21 02:40:38 +00:00
svcforge-ci	9c8d10ce1f	ci: bump image digests to 7079d6340f ... Built and scanned by 7079d6340f. ArgoCD syncs from this commit. [skip ci]	2026-07-21 02:25:04 +00:00
Nguyen Minh Phuc	7079d6340f	docs: bring RUNBOOK and ARCHITECTURE up to date ... The runner moved to node0 and the drift check reads release secrets via the Kubernetes API in-cluster; the docs still described node2 and `helm list`. - RUNBOOK: the durable image-cache fix is now the node0 hostPath store, not a node2 pin; /data is NFS RWX, so the Multi-Attach wait is gone. Cross-references entries 9 and 10. - ARCHITECTURE: the reconciler's edge to the cluster is "list releases", not "helm list" (helm is the out-of-cluster fallback). - ARCHITECTURE: the state diagram and its prose described fail() moving every dead-lettered instance to `failed`. Corrected to the per-kind behaviour — only provision fails the instance; deprovision stays `deleting` for retry, upgrade and verify stay `ready` — matching the fix in tasks.py.	2026-07-21 01:46:54 +00:00
Nguyen Minh Phuc	66eb6cb0ee	refactor: converge the patterns multiple authors left divergent ... The codebase was written by several agents and had the same concept done more than one way. This makes it read as one voice, with no behaviour change. Dedup, each to a single canonical form: - INSTANCE_COLUMNS: the 13-column instances SELECT list existed as _COLUMNS in instances.py and reconcile.py (byte-identical) and inlined a third time in the worker. One exported constant now. - Settings.runtime_dsn: the three entrypoints each chose between str(pg_dsn) and pg_dsn.unicode_string(). One property. - yaml_tempfile: helm._ValuesFile and k8s._ManifestFile were the same write-yaml-to-a-temp-dir context manager. One helper in adapters/tempyaml.py. - services/_runtime.py: sleep_or_stop and install_stop_signals were copied between the worker and reconciler loops. One module, so shutdown behaviour cannot drift between them. - k8s.ensure_namespace used MANAGED_BY_LABEL/VALUE from helm.py instead of a hardcoded literal, so the managed-by label has one definition. - SvcforgeError is now the root of every svcforge exception (CatalogError, IllegalTransition, BadWindow, HandlerError), keeping each stdlib base in the MRO, so `except SvcforgeError` means what errors.py says it does. - ERROR_MAX_CHARS replaces the repeated `[-2000:]` truncation feeding the same error columns. - the reconciler reads settings.metrics_port like the worker, dropping its duplicate DEFAULT_METRICS_PORT and redundant --metrics-port option; the SVCFORGE_METRICS_PORT env override still applies through pydantic. Two smaller correctness/consistency fixes: - RateLimitResult.retry_after_s computed its delta against datetime.now(UTC) while the limiter runs on an injectable clock, so it was meaningless under a FakeClock and drifted by request latency in production. It now carries a checked_at from the same clock as reset_at. - handle_provision's notifier.send is wrapped like the reconciler's: a flaky webhook after the READY CAS would fail the task, and the retry would hit the READY early-return and drop the notification, turning a good provision into a failed one.	2026-07-21 01:46:54 +00:00
Nguyen Minh Phuc	d64c3c9f39	fix: three correctness bugs found in review + a dropped-log ... fail() blanket-marked the instance `failed` for every dead-lettered task kind. Only provision is correct. The others each left the instance in a state the UPDATE then corrupted: - deprovision: `deleting` -> `failed` stranded the instance, because due_for_deprovision only re-selects ready/deleting, leaking the release reconcile.py promises to reclaim. - upgrade: helm --atomic rolled back, so the instance was still `ready` and serving the old version; `failed` mislabelled a healthy service and dropped it off the upgrade work-list. - verify: handle_verify already halted the rollout; the instance was `ready`. Now gated on kind == provision, with a regression test per kind (control-tested against the blanket UPDATE, which fails all three). The API exception handler was registered on fastapi.HTTPException, a subclass of starlette's. Starlette matches handlers by walking type(exc).__mro__, so framework-raised 404/405 never hit it and returned {"detail": ...} instead of ErrorBody. Registered on the starlette parent, and added a RequestValidationError handler so body-validation 422s share the shape too. Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field. helm._list_releases_via_api built the httpx client with verify=<ca path>, which loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an error the except clause did not catch, crashing the reconciler tick as a bare bug. Now the CA is checked for readability alongside the token, and a missing one means "not in-cluster" and falls back to helm. Also documents the no-limit pagination invariant and pins it with a test. redis.py logged through stdlib logging with extra={"team": team}, which the structlog bridge drops on the floor — the trap notify.py documents. Switched to a bound logger with team as a kwarg.	2026-07-21 01:46:54 +00:00
svcforge-ci	d6c1b64512	ci: bump image digests to 76cadca8e3 ... Built and scanned by 76cadca8e3. ArgoCD syncs from this commit. [skip ci]	2026-07-20 15:27:19 +00:00
Nguyen Minh Phuc	76cadca8e3	fix: Kubernetes serialises an empty item list as null, not [] ... The drift check failed on every tick with: TypeError: 'NoneType' object is not iterable `resp.json().get("items", [])` cannot defend against this. The key IS present, with the value null, so the default never fires. An empty PartialObjectMetadataList comes back as `"items": null`, which is exactly what happens once the release selector matches nothing — the normal state of a cluster with no tenant releases provisioned yet. `or []` handles both shapes. The tests could not have caught this: all of them passed a real empty list, which is the one shape that works. The new test sends `"items": null` as the API server actually sends it, and is control-tested — restoring the old expression fails it with the same TypeError seen in production. Worth recording why this survived review. The API read was verified against a live cluster before shipping, but that cluster had 25 matching release secrets, so the empty case never ran. A measurement on real data proved the fast path worked and said nothing about the path taken when there is no data.	2026-07-20 15:17:51 +00:00
svcforge-ci	66336d3648	ci: bump image digests to a843494627 ... Built and scanned by a843494627. ArgoCD syncs from this commit. [skip ci]	2026-07-20 15:01:21 +00:00
Nguyen Minh Phuc	a843494627	runbook: the runner's three ephemeral caches and the boot cascade ... Three caches on this runner were container-layer only, each found because something was slow: dind's image store, the trivy vuln DB, and act's action clones. All three now have real storage. act clones actions with full history, not shallow — 66.7MB/538 commits for setup-uv, 24.4MB/222 for actions/checkout, and this workflow uses five. After a restart that made `Set up job` an 11-minute step with the job container sat idle running `sleep` while the runner cloned GitHub. Includes the command to tell those two apart. Also records the cascade the dind fix creates. A persistent image store makes dockerd scan on boot — 38s, 2m13s, or over 5 minutes depending on node load — and two timeouts then fire: dind's startup probe, and the runner image's own hardcoded `Docker wait timeout of 5m0s`, which the chart cannot configure. The runner exits 1, restarts, and kills the running job, which looks like every step failing at once with no error after a green `Set up job`. It self-heals in about ten minutes at the cost of one CI run, so runner restarts are now something to do deliberately rather than casually.	2026-07-20 14:10:28 +00:00
Nguyen Minh Phuc	4193a18cae	reconciler: read release secrets directly instead of shelling helm ... helm's list gunzips every release payload to build its table. The only fields either caller reads are name and namespace: reconciler/main.py live = {(r.name, r.namespace) for r in releases} worker/handlers.py releases = {r.name for r in await ...list_releases()} Both live in the release secret's labels and metadata, so nothing needs decompressing. Measured from a pod on this cluster: 21ms and 31KB, against helm's 4392ms. That is the whole fix for a drift check that was timing out at 330s every tick and OOMKilling the container at 256Mi. The cause of both was decompressing 96 releases to extract two strings each. It also means the container no longer cares that a cluster policy mutates its CPU request to 0 — at 21ms there is nothing left to starve. Three details carry the correctness, each with a test: - PartialObjectMetadataList in the Accept header asks for metadata only. Without it every release's gzipped manifest crosses the wire to be thrown away, which is the cost this exists to avoid. - The status selector drops superseded revisions server-side: 96 release secrets here, 25 live. Failed and pending states are kept, matching what `helm list` shows, because a failed release does exist and calling it missing would have the reconciler re-provision on top of it. - helm writes one secret per revision, up to 10 per release here, so the newest version label wins. Control-tested with a string compare, which picks "9" over "10". Out of cluster there is no ServiceAccount, so it falls back to helm and the e2e suite and laptop runs are unchanged. An API error raises rather than falling back: the fallback is for a known-absent ServiceAccount, and quietly retrying through helm would swap a visible error for the timeout this removes. chart and app_version are empty on this path rather than wrong — they live only in the compressed payload, and nothing reads them.	2026-07-20 13:05:25 +00:00
Nguyen Minh Phuc	2356ac4ef3	reconciler: scope the drift check to svcforge's own releases ... check_drift diffs helm against the database in both directions, and the second one is `live - known` -> drift.orphan_release at ERROR. `live` was every release in the cluster, so argocd, longhorn, gitea and cert-manager were all reported as orphans svcforge is failing to account for, on every sweep. They are not orphans; they were never svcforge's to know about. install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases() selects on it. Releases provisioned before this see one sweep as missing and get re-provisioned, which is safe by construction — `helm upgrade --install` against a deterministic release name — and that re-provision applies the label. This does NOT fix the timeout, and the docstring says so. Measured, not assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is not pushed down as a server-side selector, so helm still fetches and decompresses every release secret and filters what it already parsed. Around 10%, not the order of magnitude the flag's shape suggests. The 330s timeouts need CPU for the container or a different read path. The two sides of the label are asserted against each other rather than a literal, so a rename that updates only one fails in tests instead of in production as a silently empty drift check. Control-tested: renaming the read side alone fails two of the three.	2026-07-20 12:53:16 +00:00
Nguyen Minh Phuc	c2a27952d1	runbook: fix section ordering and a duplicate number ... The dind entry landed ahead of the postgres one, and the postgres entry I added earlier was numbered 7 while 'Verify the whole loop' already was. Now 7 postgres, 8 dind, 9 verify.	2026-07-20 07:39:23 +00:00
Nguyen Minh Phuc	0dbb5af1d3	runbook: record the dind liveness probe as an open issue ... dind is killed by a probe whose only job is to check a socket exists, with a 1s timeout the nodes cannot always meet: 27 failures over 156 minutes. Worth recording because it probably explains build failures already attributed to something else. `DeadlineExceeded: no active session` was blamed on CPU starvation and addressed by dropping runner capacity to 1; the likelier mechanism is kubelet killing dind mid-build and taking the buildkit session with it. Capacity reduced the load that trips the probe, which fits #17 passing and #18-#21 failing anyway. Written as a hypothesis with the command to confirm it, not as a conclusion. The chart exposes no probe knobs, so the candidate fix is a Kyverno mutation in Ansible, following the existing force-best-effort-cpu precedent.	2026-07-20 07:38:51 +00:00
Nguyen Minh Phuc	d4ac3801a3	runbook: salvaging a faulted Longhorn volume ... The documented scale 0/1 recovery did not work this time: the volume returned detached/faulted and refused to attach, so the pod sat in ContainerCreating. auto-salvage: true cannot rescue it, because salvage happens during attach and a faulted volume never gets that far. The replica's failedAt timestamp is the only thing holding it faulted. Clearing it brought the volume to attached/healthy and postgres to 1/1 in under a minute, with repo data and CI history intact. Adds the backup check first, which matters more now that Longhorn runs at one replica and there is no second copy to fall back on.	2026-07-20 07:10:04 +00:00
Nguyen Minh Phuc	6093c52160	argocd: ignore the cpu request this cluster rewrites ... A Kyverno ClusterPolicy, force-best-effort-cpu, rewrites every container's CPU request to "0" at admission. It is deliberate and predates this app by well over a year: the nodes are oversubscribed and BestEffort-on-CPU is how everything gets scheduled. The chart asks for 50m and the cluster writes 0, so all three Deployments sat permanently OutOfSync while perfectly Healthy — the failure mode where a dashboard is always yellow, everyone learns to ignore it, and it stops meaning anything the day it goes yellow for a real reason. Ignored on the Application rather than capitulating in the chart. What the chart asks for is the honest intent; what the cluster does with it is the cluster's business, and a reader of the repo should see the former. jqPathExpressions rather than jsonPointers, which would have to name a container index and this has to hold for every container in all three Deployments. Takes effect only after a hard refresh.	2026-07-20 06:58:28 +00:00
Nguyen Minh Phuc	31fa9165ff	chart: give the migrate hook its own ServiceAccount ... The Job ran as the api ServiceAccount, which is an ordinary chart resource. Hooks are created before the release's ordinary manifests, so on a first install that account does not exist and the Job never starts: Error creating: pods "svcforge-migrate-" is forbidden: error looking up service account svcforge/svcforge-api: serviceaccount "svcforge-api" not found Not an ArgoCD quirk — helm orders hooks the same way, so both deploy paths failed identically. It survived review because the chart was only ever checked with `helm template` and `helm install --dry-run=server`, and neither creates a Job. The pod is what fails, so only a real install can catch it. The new account is a hook at weight -10, ahead of the Job at -5, and is bound to no Role: the migration talks to Postgres and wants nothing from the Kubernetes API. automountServiceAccountToken is off for the same reason. Verified with a real `helm install` into a scratch namespace: STATUS deployed, job Complete 1/1, four migrations applied, and the hook ServiceAccount was 27s old against 10s for the ordinary ones — the ordering the bug turned on.	2026-07-20 06:33:31 +00:00
Nguyen Minh Phuc	89ad9625f3	chart: disable externalSecret on this cluster ... The block describes a ClusterSecretStore named `vault` with HashiCorp-style key/property refs. This cluster has `oci-vault` — OCI Vault via InstancePrincipal — whose provider addresses a secret by name and takes a JSON property, so those remoteRefs do not translate as written. There are no ExternalSecrets anywhere on the cluster, so the path had never been exercised. svcforge.secretName still resolves through targetName, so the deployments and the migrate hook read a Secret named svcforge-secrets, created out of band from ~/.config/svcforge/secrets.env. This is a deviation and the comment says so. ArgoCD does not manage that Secret, so prune and selfHeal cannot touch it, and it is the one part of the deployment that cannot be read from git. Restoring the intended design means adding oci_vault_secret resources to oci-k8s/infra/vault.tf and repointing secretStoreRef at oci-vault.	2026-07-20 06:26:13 +00:00
Nguyen Minh Phuc	e8b6116a58	ci: age-filter the image prune so digest-pulled images survive ... Run #17 showed `docker image prune -f` deleting the trivy image, which buys back a 157MB pull on every subsequent run. An image pulled by digest carries no tag, so it is dangling the moment its container exits, and bare dangling-only pruning is not as narrow as it reads. The act runner image survived the same prune only by accident: this step runs inside an act container, so that image happens to be in use exactly while the prune executes. That is luck, not design, and it would not hold if the step ever moved. An until=168h filter on both prunes keeps a week of recently used layers, matching the buildx cache window already in place. Verified in run #17 before this change: trivy logged "Need to update DB" zero times, so the named-volume DB cache works, and registry digests equal the chart's for all three images.	2026-07-20 05:17:13 +00:00
Nguyen Minh Phuc	5f18f9eeeb	runbook: correct the claim that runner restarts orphan jobs ... The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel.	2026-07-20 05:17:13 +00:00
svcforge-ci	1a36f43c69	ci: bump image digests to 37b297bf5c ... Built and scanned by 37b297bf5c. ArgoCD syncs from this commit. [skip ci]	2026-07-20 05:12:26 +00:00
Nguyen Minh Phuc	37b297bf5c	ci: reclaim dind disk after each image build ... dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed.	2026-07-20 04:48:48 +00:00
Nguyen Minh Phuc	5d7f46483e	runbook: recovering the queue after a runner restart ... Restarting act_runner leaves its in-flight jobs in_progress with nothing behind them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run index, not the database id.	2026-07-20 04:41:15 +00:00
Nguyen Minh Phuc	b02d4e85c6	ci: record that act_runner ignores max-parallel ... The previous commit claimed max-parallel: 1 would stop two image builds from starving each other. It does not — act_runner ignores strategy.max-parallel. Run #14 had it set and still ran the worker leg from 04:25:47 while reconciler was still building, after api had run alone from 04:23:03 to 04:24:04. That run failed a different way: actions/checkout could not reach Gitea at all (`Failed to connect to gitea-http:3000 after 3105 ms`) while two runs were in flight, with node0 at 140% memory. Same root cause, new mask. The setting stays, since it is correct on runners that honour it, but the comment no longer claims it does anything here. The binding lever is the runner's `capacity`, dropped 2 -> 1 in oci-k8s.	2026-07-20 04:27:48 +00:00
Nguyen Minh Phuc	d70a7b622c	ci: cache the trivy vuln DB in a named volume, not a $PWD bind ... `-v "$PWD/.trivycache:/root/.cache/trivy"` was the third instance of the bug that made gitleaks scan nothing. $PWD is a path in the job container, but the -v is resolved by the daemon in the dind sidecar, which has no such directory and silently creates an empty one. Every run logged `[vulndb] Need to update DB` and re-downloaded the DB, and the cache it wrote went to a throwaway directory inside dind. This file warns about the same trap in two other places. A named volume lives in the dind daemon's own storage, which is the one thing both sides agree on. Verified in dind rather than assumed: cold pass logs `Need to update DB` / `Downloading vulnerability DB` / `Artifact successfully downloaded` and leaves 1.1G in the volume; warm pass logs none of them. The old comment's "~50MB" was wrong by 20x, so the comment now records the real figure and what reclaims it. The two sibling pipelines on this runner (cicd-smoke, otel-demo) were checked for the same bug and do not have it — they use `docker build <context>`, where the client streams the context to the daemon.	2026-07-20 04:13:30 +00:00
Nguyen Minh Phuc	2098443f31	ci: build images one at a time ... `image (api)` and `image (reconciler)` both died in run #13 with: ERROR: failed to solve: DeadlineExceeded: no active session for ylux5s1p4xyb5e5tmnz4gljhq: context deadline exceeded That is the buildkit session heartbeat between buildx in the job container and buildkitd in the dind sidecar, not anything in the Dockerfiles. The runner has capacity 2, so two legs of the matrix built concurrently on a 2-core burstable node already sitting at 81% CPU and 87% memory, and starved each other. The evidence is in the same run: `image (worker)` was still building when the other two failed, had the node to itself afterwards, and went green on an unchanged Dockerfile. max-parallel: 1 trades wall-clock for builds that finish. Every other gate in run #13 — lint, types, unit, integration, security, dockerfile, chart — was already green, so this is the last thing standing between here and a full run.	2026-07-20 04:11:04 +00:00
Nguyen Minh Phuc	b35c551160	ci: the digest-guard test depended on state CI itself mutates ... The chart job asserted that a bare `helm template` FAILS, on the assumption that values.yaml always holds all-zeros placeholder digests. That assumption dies the first time the bump job runs: bump commits real digests into values.yaml, so the bare render then succeeds and the assertion reports 'the guard is not guarding' about a guard that is fine. A test whose expected result flips depending on whether CI has run before is not a test. Now it feeds the guard four distinct bad digests explicitly with --set — all-zeros, a bare tag, right-prefix-wrong-length, and empty — and requires each to be rejected. Verified locally: all four rejected, and a well-formed digest still renders.	2026-07-20 03:48:34 +00:00