4193a18cae86f11dfb6cd683df8bff4a73c0cc91
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
4193a18cae |
reconciler: read release secrets directly instead of shelling helm
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
ci / bump (push) Blocked by required conditions
helm's list gunzips every release payload to build its table. The only fields
either caller reads are name and namespace:
reconciler/main.py live = {(r.name, r.namespace) for r in releases}
worker/handlers.py releases = {r.name for r in await ...list_releases()}
Both live in the release secret's labels and metadata, so nothing needs
decompressing. Measured from a pod on this cluster: 21ms and 31KB, against
helm's 4392ms.
That is the whole fix for a drift check that was timing out at 330s every tick
and OOMKilling the container at 256Mi. The cause of both was decompressing 96
releases to extract two strings each. It also means the container no longer
cares that a cluster policy mutates its CPU request to 0 — at 21ms there is
nothing left to starve.
Three details carry the correctness, each with a test:
- PartialObjectMetadataList in the Accept header asks for metadata only.
Without it every release's gzipped manifest crosses the wire to be thrown
away, which is the cost this exists to avoid.
- The status selector drops superseded revisions server-side: 96 release
secrets here, 25 live. Failed and pending states are kept, matching what
`helm list` shows, because a failed release does exist and calling it
missing would have the reconciler re-provision on top of it.
- helm writes one secret per revision, up to 10 per release here, so the
newest version label wins. Control-tested with a string compare, which
picks "9" over "10".
Out of cluster there is no ServiceAccount, so it falls back to helm and the
e2e suite and laptop runs are unchanged. An API error raises rather than
falling back: the fallback is for a known-absent ServiceAccount, and quietly
retrying through helm would swap a visible error for the timeout this removes.
chart and app_version are empty on this path rather than wrong — they live
only in the compressed payload, and nothing reads them.
|
||
|
|
2356ac4ef3 |
reconciler: scope the drift check to svcforge's own releases
ci / lint (push) Failing after 10s
ci / types (push) Has been skipped
ci / unit (push) Has been skipped
ci / integration (push) Has been skipped
ci / security (push) Has been skipped
ci / dockerfile (push) Has been skipped
ci / chart (push) Has been skipped
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
check_drift diffs helm against the database in both directions, and the second one is `live - known` -> drift.orphan_release at ERROR. `live` was every release in the cluster, so argocd, longhorn, gitea and cert-manager were all reported as orphans svcforge is failing to account for, on every sweep. They are not orphans; they were never svcforge's to know about. install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases() selects on it. Releases provisioned before this see one sweep as missing and get re-provisioned, which is safe by construction — `helm upgrade --install` against a deterministic release name — and that re-provision applies the label. This does NOT fix the timeout, and the docstring says so. Measured, not assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is not pushed down as a server-side selector, so helm still fetches and decompresses every release secret and filters what it already parsed. Around 10%, not the order of magnitude the flag's shape suggests. The 330s timeouts need CPU for the container or a different read path. The two sides of the label are asserted against each other rather than a literal, so a rename that updates only one fails in tests instead of in production as a silently empty drift check. Control-tested: renaming the read side alone fails two of the three. |
||
|
|
c2a27952d1 |
runbook: fix section ordering and a duplicate number
The dind entry landed ahead of the postgres one, and the postgres entry I added earlier was numbered 7 while 'Verify the whole loop' already was. Now 7 postgres, 8 dind, 9 verify. |
||
|
|
0dbb5af1d3 |
runbook: record the dind liveness probe as an open issue
dind is killed by a probe whose only job is to check a socket exists, with a 1s timeout the nodes cannot always meet: 27 failures over 156 minutes. Worth recording because it probably explains build failures already attributed to something else. `DeadlineExceeded: no active session` was blamed on CPU starvation and addressed by dropping runner capacity to 1; the likelier mechanism is kubelet killing dind mid-build and taking the buildkit session with it. Capacity reduced the load that trips the probe, which fits #17 passing and #18-#21 failing anyway. Written as a hypothesis with the command to confirm it, not as a conclusion. The chart exposes no probe knobs, so the candidate fix is a Kyverno mutation in Ansible, following the existing force-best-effort-cpu precedent. |
||
|
|
d4ac3801a3 |
runbook: salvaging a faulted Longhorn volume
The documented scale 0/1 recovery did not work this time: the volume returned detached/faulted and refused to attach, so the pod sat in ContainerCreating. auto-salvage: true cannot rescue it, because salvage happens during attach and a faulted volume never gets that far. The replica's failedAt timestamp is the only thing holding it faulted. Clearing it brought the volume to attached/healthy and postgres to 1/1 in under a minute, with repo data and CI history intact. Adds the backup check first, which matters more now that Longhorn runs at one replica and there is no second copy to fall back on. |
||
|
|
6093c52160 |
argocd: ignore the cpu request this cluster rewrites
ci / lint (push) Failing after 1s
ci / bump (push) Has been skipped
ci / types (push) Has been skipped
ci / unit (push) Has been skipped
ci / integration (push) Has been skipped
ci / security (push) Has been skipped
ci / dockerfile (push) Has been skipped
ci / chart (push) Has been skipped
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
A Kyverno ClusterPolicy, force-best-effort-cpu, rewrites every container's CPU request to "0" at admission. It is deliberate and predates this app by well over a year: the nodes are oversubscribed and BestEffort-on-CPU is how everything gets scheduled. The chart asks for 50m and the cluster writes 0, so all three Deployments sat permanently OutOfSync while perfectly Healthy — the failure mode where a dashboard is always yellow, everyone learns to ignore it, and it stops meaning anything the day it goes yellow for a real reason. Ignored on the Application rather than capitulating in the chart. What the chart asks for is the honest intent; what the cluster does with it is the cluster's business, and a reader of the repo should see the former. jqPathExpressions rather than jsonPointers, which would have to name a container index and this has to hold for every container in all three Deployments. Takes effect only after a hard refresh. |
||
|
|
31fa9165ff |
chart: give the migrate hook its own ServiceAccount
ci / lint (push) Failing after 14m13s
ci / types (push) Has been skipped
ci / unit (push) Has been skipped
ci / integration (push) Has been skipped
ci / security (push) Has been skipped
ci / dockerfile (push) Has been skipped
ci / chart (push) Has been skipped
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
The Job ran as the api ServiceAccount, which is an ordinary chart resource.
Hooks are created before the release's ordinary manifests, so on a first
install that account does not exist and the Job never starts:
Error creating: pods "svcforge-migrate-" is forbidden: error looking up
service account svcforge/svcforge-api: serviceaccount "svcforge-api" not
found
Not an ArgoCD quirk — helm orders hooks the same way, so both deploy paths
failed identically. It survived review because the chart was only ever checked
with `helm template` and `helm install --dry-run=server`, and neither creates a
Job. The pod is what fails, so only a real install can catch it.
The new account is a hook at weight -10, ahead of the Job at -5, and is bound
to no Role: the migration talks to Postgres and wants nothing from the
Kubernetes API. automountServiceAccountToken is off for the same reason.
Verified with a real `helm install` into a scratch namespace: STATUS deployed,
job Complete 1/1, four migrations applied, and the hook ServiceAccount was 27s
old against 10s for the ordinary ones — the ordering the bug turned on.
|
||
|
|
89ad9625f3 |
chart: disable externalSecret on this cluster
ci / bump (push) Blocked by required conditions
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
The block describes a ClusterSecretStore named `vault` with HashiCorp-style key/property refs. This cluster has `oci-vault` — OCI Vault via InstancePrincipal — whose provider addresses a secret by name and takes a JSON property, so those remoteRefs do not translate as written. There are no ExternalSecrets anywhere on the cluster, so the path had never been exercised. svcforge.secretName still resolves through targetName, so the deployments and the migrate hook read a Secret named svcforge-secrets, created out of band from ~/.config/svcforge/secrets.env. This is a deviation and the comment says so. ArgoCD does not manage that Secret, so prune and selfHeal cannot touch it, and it is the one part of the deployment that cannot be read from git. Restoring the intended design means adding oci_vault_secret resources to oci-k8s/infra/vault.tf and repointing secretStoreRef at oci-vault. |
||
|
|
e8b6116a58 |
ci: age-filter the image prune so digest-pulled images survive
ci / lint (push) Successful in 43s
ci / types (push) Successful in 1m14s
ci / unit (push) Successful in 45s
ci / security (push) Successful in 1m24s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 11s
ci / integration (push) Successful in 1m40s
ci / image (api) (push) Successful in 2m32s
ci / image (reconciler) (push) Successful in 3m0s
ci / image (worker) (push) Failing after 49s
ci / bump (push) Has been skipped
Run #17 showed `docker image prune -f` deleting the trivy image, which buys back a 157MB pull on every subsequent run. An image pulled by digest carries no tag, so it is dangling the moment its container exits, and bare dangling-only pruning is not as narrow as it reads. The act runner image survived the same prune only by accident: this step runs inside an act container, so that image happens to be in use exactly while the prune executes. That is luck, not design, and it would not hold if the step ever moved. An until=168h filter on both prunes keeps a week of recently used layers, matching the buildx cache window already in place. Verified in run #17 before this change: trivy logged "Need to update DB" zero times, so the named-volume DB cache works, and registry digests equal the chart's for all three images. |
||
|
|
5f18f9eeeb |
runbook: correct the claim that runner restarts orphan jobs
The previous entry stated that restarting act_runner leaves every in-flight job orphaned. That is wrong: run #16 had its remaining jobs re-dispatched to the new pod and finished normally, while run #14 really was left stuck. Both outcomes happen, so in_progress after a restart is ambiguous and the runner logs are what settle it. Also corrects the recovery advice. Gitea 1.26 has no cancel endpoint anywhere in its swagger, and DELETE on a run returned 204 against a live run without stopping it. The UI button is the only way to cancel. |
||
|
|
1a36f43c69 |
ci: bump image digests to 37b297bf5c
Built and scanned by
|
||
|
|
37b297bf5c |
ci: reclaim dind disk after each image build
ci / lint (push) Successful in 43s
ci / types (push) Successful in 1m29s
ci / unit (push) Successful in 59s
ci / security (push) Successful in 1m10s
ci / dockerfile (push) Successful in 24s
ci / chart (push) Successful in 9s
ci / integration (push) Successful in 1m50s
ci / image (api) (push) Successful in 4m57s
ci / image (reconciler) (push) Successful in 2m55s
ci / image (worker) (push) Successful in 2m58s
ci / bump (push) Successful in 23s
dind's /var/lib/docker is now a hostPath on node2 rather than the container's writable layer, so it survives restarts — and nothing reclaims it. Kubelet's image GC does not manage a nested daemon's store, so left alone it grows every run until node2 hits disk pressure and evicts pods, which reads as a cluster problem rather than a CI one. The step is deliberately narrow. `docker image prune` without -a removes dangling images only; with -a it would delete the act runner image, which no container references between jobs, and buy back a 1.6GB re-pull on the next run. The buildx cache is what actually grows without bound, so it is pruned by age keeping a week, recent enough that --cache-from still hits. Named volumes are never pruned, since that is where the trivy vuln DB lives. always(), because a failed build still leaves layers behind, and that is when disk is most likely to have been the reason it failed. |
||
|
|
5d7f46483e |
runbook: recovering the queue after a runner restart
Restarting act_runner leaves its in-flight jobs in_progress with nothing behind
them, and at capacity 1 one orphan blocks every later run. Gitea 1.26 has no
cancel endpoint; DELETE .../actions/runs/{index} does the job and takes the run
index, not the database id.
|
||
|
|
b02d4e85c6 |
ci: record that act_runner ignores max-parallel
ci / integration (push) Successful in 2m54s
ci / lint (push) Successful in 3m5s
ci / unit (push) Successful in 47s
ci / types (push) Successful in 1m11s
ci / security (push) Successful in 1m15s
ci / dockerfile (push) Successful in 29s
ci / chart (push) Successful in 1m39s
ci / image (reconciler) (push) Has been cancelled
ci / image (worker) (push) Has been cancelled
ci / bump (push) Has been cancelled
ci / image (api) (push) Has been cancelled
The previous commit claimed max-parallel: 1 would stop two image builds from starving each other. It does not — act_runner ignores strategy.max-parallel. Run #14 had it set and still ran the worker leg from 04:25:47 while reconciler was still building, after api had run alone from 04:23:03 to 04:24:04. That run failed a different way: actions/checkout could not reach Gitea at all (`Failed to connect to gitea-http:3000 after 3105 ms`) while two runs were in flight, with node0 at 140% memory. Same root cause, new mask. The setting stays, since it is correct on runners that honour it, but the comment no longer claims it does anything here. The binding lever is the runner's `capacity`, dropped 2 -> 1 in oci-k8s. |
||
|
|
d70a7b622c |
ci: cache the trivy vuln DB in a named volume, not a $PWD bind
ci / lint (push) Successful in 1m54s
ci / types (push) Successful in 1m42s
ci / unit (push) Successful in 2m49s
ci / security (push) Successful in 2m17s
ci / dockerfile (push) Successful in 46s
ci / chart (push) Successful in 2m52s
ci / image (api) (push) Has been cancelled
ci / image (reconciler) (push) Has been cancelled
ci / image (worker) (push) Has been cancelled
ci / bump (push) Has been cancelled
ci / integration (push) Has been cancelled
`-v "$PWD/.trivycache:/root/.cache/trivy"` was the third instance of the bug that made gitleaks scan nothing. $PWD is a path in the job container, but the -v is resolved by the daemon in the dind sidecar, which has no such directory and silently creates an empty one. Every run logged `[vulndb] Need to update DB` and re-downloaded the DB, and the cache it wrote went to a throwaway directory inside dind. This file warns about the same trap in two other places. A named volume lives in the dind daemon's own storage, which is the one thing both sides agree on. Verified in dind rather than assumed: cold pass logs `Need to update DB` / `Downloading vulnerability DB` / `Artifact successfully downloaded` and leaves 1.1G in the volume; warm pass logs none of them. The old comment's "~50MB" was wrong by 20x, so the comment now records the real figure and what reclaims it. The two sibling pipelines on this runner (cicd-smoke, otel-demo) were checked for the same bug and do not have it — they use `docker build <context>`, where the client streams the context to the daemon. |
||
|
|
2098443f31 |
ci: build images one at a time
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / bump (push) Blocked by required conditions
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
`image (api)` and `image (reconciler)` both died in run #13 with: ERROR: failed to solve: DeadlineExceeded: no active session for ylux5s1p4xyb5e5tmnz4gljhq: context deadline exceeded That is the buildkit session heartbeat between buildx in the job container and buildkitd in the dind sidecar, not anything in the Dockerfiles. The runner has capacity 2, so two legs of the matrix built concurrently on a 2-core burstable node already sitting at 81% CPU and 87% memory, and starved each other. The evidence is in the same run: `image (worker)` was still building when the other two failed, had the node to itself afterwards, and went green on an unchanged Dockerfile. max-parallel: 1 trades wall-clock for builds that finish. Every other gate in run #13 — lint, types, unit, integration, security, dockerfile, chart — was already green, so this is the last thing standing between here and a full run. |
||
|
|
b35c551160 |
ci: the digest-guard test depended on state CI itself mutates
ci / lint (push) Successful in 1m56s
ci / unit (push) Successful in 2m2s
ci / types (push) Successful in 2m4s
ci / security (push) Successful in 1m21s
ci / dockerfile (push) Successful in 1m24s
ci / chart (push) Successful in 15s
ci / integration (push) Successful in 1m58s
ci / image (reconciler) (push) Failing after 6m1s
ci / image (api) (push) Failing after 6m7s
ci / image (worker) (push) Successful in 5m33s
ci / bump (push) Has been skipped
The chart job asserted that a bare `helm template` FAILS, on the assumption that values.yaml always holds all-zeros placeholder digests. That assumption dies the first time the bump job runs: bump commits real digests into values.yaml, so the bare render then succeeds and the assertion reports 'the guard is not guarding' about a guard that is fine. A test whose expected result flips depending on whether CI has run before is not a test. Now it feeds the guard four distinct bad digests explicitly with --set — all-zeros, a bare tag, right-prefix-wrong-length, and empty — and requires each to be rejected. Verified locally: all four rejected, and a well-formed digest still renders. |
||
|
|
f87d8d4d78 |
runbook: the 14-minute 'Set up job' failure and its cause
ci / lint (push) Waiting to run
ci / types (push) Blocked by required conditions
ci / unit (push) Blocked by required conditions
ci / integration (push) Blocked by required conditions
ci / security (push) Blocked by required conditions
ci / dockerfile (push) Blocked by required conditions
ci / chart (push) Blocked by required conditions
ci / image (api) (push) Blocked by required conditions
ci / image (reconciler) (push) Blocked by required conditions
ci / image (worker) (push) Blocked by required conditions
ci / bump (push) Blocked by required conditions
Every job was failing at Set up job after ~14 minutes, then the first real step died at 0s. It read like a broken action; it was an empty dind image cache re-pulling the 1.6GB act job image on every run. dind has no volume for /var/lib/docker, so the cache lives in its writable layer and dies with every pod restart. A restart-looping runner therefore never keeps one. Warming it by hand took lint from failure to success with no code change. |
||
|
|
c537073c21 |
deps: everything to latest stable
ci / dockerfile (push) Has been cancelled
ci / types (push) Has been cancelled
ci / lint (push) Has been cancelled
ci / security (push) Has been cancelled
ci / chart (push) Has been cancelled
ci / image (api) (push) Has been cancelled
ci / image (reconciler) (push) Has been cancelled
ci / image (worker) (push) Has been cancelled
ci / integration (push) Has been cancelled
ci / unit (push) Has been cancelled
ci / bump (push) Has been cancelled
Python 3.12 -> 3.14, postgres 16 -> 18, uv 0.5.11 -> 0.11.29, trivy 0.58.1 -> 0.72.0, gitleaks 8.21.2 -> 8.30.1, yq 4.44.6 -> 4.53.3, and every action re-pinned to the SHA of its latest tag (checkout v7, setup-uv v8, buildx v4, login v4, hadolint v3.3.0). helm stays 3.21.3: already current for 3.x, and helm 4 is a breaking change, not a CVE fix. trivy mattered most. A vulnerability scanner fourteen minor versions behind is the one stale pin that hides all the others. ruff target-version is deliberately py313 while the runtime is 3.14. It controls the syntax the formatter may emit, and at py314 it rewrites 'except (A, B):' into PEP 758's unparenthesized form — which reads exactly like Python 2's 'except E, name:' and is a hard SyntaxError below 3.14. No semantic gain, real readability cost, in a repo meant to be read. Verified on 3.14: ruff, ruff format, mypy --strict, 166 tests, helm lint, bandit, pip-audit. The digest guard still rejects placeholder digests. Risk carried knowingly: the bumped actions run on node24. If act_runner only provides node20, every job fails at action startup and this commit is the revert. |
||
|
|
4544765ec5 |
test: scope the process-group assertion to this run's own children
test_timeout_kills_the_whole_process_group asserted that `pgrep -f 'sleep 300'` returns nothing. That is machine-global: it matches a leftover from an earlier run of the same test, any unrelated 'sleep 300' on the box, and the shell running pgrep, whose own command line contains the pattern being searched for. Observed three spurious matches on a dev box, which produced a red run that looked like a Python 3.14 regression in the helm timeout kill. It was not — the same test failed identically on 3.12. A test that fails for reasons unrelated to the code is as useless as one that cannot fail. The sleep duration is now derived from the pid, so it cannot collide with another run, and the test asserts up front that its own pattern matches nothing before it starts. Verified: 3 consecutive passes on each of Python 3.12 and 3.14. |
||
|
|
f2b159ef7e |
ci: bump image digests to ca21b6e70d
Built and scanned by
|
||
|
|
ca21b6e70d |
ci: gitleaks was passing without scanning anything
ci / lint (push) Successful in 18s
ci / dockerfile (push) Successful in 18s
ci / chart (push) Successful in 19s
ci / security (push) Successful in 1m7s
ci / integration (push) Successful in 48s
ci / unit (push) Successful in 37s
ci / types (push) Successful in 50s
ci / image (api) (push) Successful in 2m17s
ci / image (reconciler) (push) Successful in 2m36s
ci / image (worker) (push) Successful in 1m30s
ci / bump (push) Successful in 14s
The secret-scanning gate has been green and meaningless. `docker run -v "$PWD:/repo"`
resolves the bind source on the dind sidecar's daemon, not in the job container, so
gitleaks received an empty mount:
ERR [git] fatal: not a git repository (or any parent up to mount point /)
ERR failed to scan Git repository error="stderr is not empty"
INF scan completed in 35.2ms
INF no leaks found <- exit 0
It logged the failure and exited 0. A scanner that reports success without looking is
worse than no scanner. The 35ms runtime was the tell.
Now a checksum-pinned binary, plus a `git rev-list --count HEAD` assertion so an
unscannable checkout fails the job instead of passing it.
Same root cause and same fix for yq in bump-digests.sh, which failed the bump job with
"stat deploy/chart/values.yaml: no such file or directory". helm was fixed this way
earlier. Nothing on this runner should mount $PWD into a container.
|
||
|
|
dc51bd1d9d |
ci: move the bump job's master guard from job level to step level
ci / lint (push) Successful in 19s
ci / unit (push) Successful in 51s
ci / types (push) Successful in 1m1s
ci / dockerfile (push) Successful in 12s
ci / chart (push) Successful in 9s
ci / security (push) Successful in 57s
ci / integration (push) Successful in 51s
ci / image (api) (push) Successful in 2m13s
ci / image (reconciler) (push) Successful in 2m45s
ci / image (worker) (push) Successful in 2m32s
ci / bump (push) Failing after 28s
bump skipped at 0s on a genuine push to master (event: push, head_branch: master) — before the image job it depends on had even started. The identical expression on the 'push by digest' STEP inside the matrix job evaluates true and runs, so the expression is fine; Gitea does not resolve a job-level if correctly when needs points at a matrix job. needs: [image] still orders it and still gates on all three legs. The steps carry the guard in the form this runner is known to evaluate. A PR now starts the job and no-ops every step, which is a few seconds for a guard that actually fires. |
||
|
|
78a0a6d500 |
runbook: record how the image gate was taken to zero findings
ci / lint (push) Successful in 21s
ci / unit (push) Failing after 41s
ci / integration (push) Has been skipped
ci / types (push) Successful in 1m18s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 7s
ci / security (push) Successful in 54s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
trivy on the worker image: 39 findings (2 CRITICAL) -> 18 -> 5 -> 0. Verified in CI
run #7 on
|
||
|
|
4a426dbe50 |
worker: create namespaces with helm, drop the kubectl binary
ci / lint (push) Successful in 25s
ci / unit (push) Successful in 59s
ci / types (push) Successful in 1m8s
ci / dockerfile (push) Successful in 13s
ci / chart (push) Successful in 8s
ci / security (push) Successful in 1m2s
ci / integration (push) Successful in 54s
ci / image (api) (push) Successful in 1m6s
ci / image (reconciler) (push) Successful in 3m6s
ci / image (worker) (push) Successful in 2m28s
ci / bump (push) Has been cancelled
trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the newest kubectl that exists. No release clears them; upstream has not rebuilt against the patched Go yet. Chasing the version further has no end. kubectl was in that image for exactly one call: ensure_namespace before helm. `helm upgrade --install --create-namespace` does the same thing, idempotently, as part of the install it already runs. So the binary goes, and its vendored CVEs go with it. read_secret had no callers. Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since --create-namespace makes a bare one. Nothing reads that label today. |
||
|
|
a59d3848ae |
ci: kubectl v1.35.3 -> v1.36.2 to clear CVE-2026-35469
ci / lint (push) Successful in 36s
ci / unit (push) Successful in 1m40s
ci / types (push) Successful in 1m49s
ci / dockerfile (push) Successful in 44s
ci / security (push) Successful in 1m50s
ci / chart (push) Successful in 1m10s
ci / integration (push) Successful in 1m39s
ci / image (api) (push) Successful in 6m16s
ci / image (reconciler) (push) Successful in 7m32s
ci / image (worker) (push) Failing after 6m36s
ci / bump (push) Has been skipped
trivy took the worker image from 39 findings (2 CRITICAL) to 18 (0 CRITICAL) after the previous bump. The residue is kubectl's vendored deps: k8s v1.35.x pins github.com/moby/spdystream v0.5.0, and the fix is in 0.5.1. v1.36.2 vendors 0.5.1. One minor ahead of the v1.35.3 API server, which is inside kubectl's supported +/-1 window. reconciler was already green because it ships helm only. |
||
|
|
a51227d952 |
ci: run helm from a checksum-pinned binary, not a container
ci / lint (push) Successful in 23s
ci / types (push) Successful in 1m29s
ci / unit (push) Successful in 1m31s
ci / dockerfile (push) Successful in 38s
ci / security (push) Successful in 1m22s
ci / chart (push) Successful in 45s
ci / integration (push) Successful in 1m21s
ci / image (api) (push) Successful in 5m26s
ci / image (worker) (push) Failing after 7m34s
ci / image (reconciler) (push) Successful in 6m0s
ci / bump (push) Has been skipped
The chart gate failed with 'stat deploy/chart/Chart.yaml: no such file or directory' while the file plainly existed in the checkout. `docker run -v "$PWD:/repo"` is interpreted by the dind sidecar's daemon rather than by the job container, so the bind source has to exist on the daemon's side of that boundary. Downloading the binary removes the boundary entirely. Pinned by sha256 for the same reason every image here is pinned by digest. |
||
|
|
c76154aeaa |
review: fix 26 findings from a 4-agent audit
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files. |
||
|
|
77d560ddae |
ci: move buildx layer cache to the registry
ci / lint (push) Successful in 1m22s
ci / types (push) Successful in 1m35s
ci / unit (push) Successful in 1m41s
ci / dockerfile (push) Successful in 1m14s
ci / security (push) Successful in 1m26s
ci / integration (push) Successful in 1m50s
ci / image (reconciler) (push) Failing after 8m10s
ci / image (api) (push) Successful in 9m49s
ci / image (worker) (push) Failing after 6m12s
ci / bump (push) Has been skipped
act_runner's cache PVC is 1Gi and also holds .runner, the runner's own registration file. --cache-to type=gha,mode=max for three images is several GB; filling that volume breaks the runner, not just the cache. act_runner also evicts by age with no size cap, so it fills whatever it is given. type=registry has no such limit and lives beside the images it caches. The uv cache still uses the runner's cache service, which is a few hundred MB. Registry login is no longer gated to master: the build now reads and writes the cache on every run. Pushing the release image keeps its own master-only gate. |
||
|
|
c9d0176bb3 |
ci: run trivy directly; document CI/CD setup in RUNBOOK
trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache. |
||
|
|
851f8919a8 |
ci: fix coverage target, pip-audit scope, bandit config
ci / lint (push) Successful in 35s
ci / unit (push) Successful in 1m11s
ci / types (push) Successful in 1m22s
ci / dockerfile (push) Successful in 4s
ci / security (push) Successful in 1m28s
ci / integration (push) Successful in 1m36s
ci / image (reconciler) (push) Failing after 17m0s
ci / image (api) (push) Failing after 17m1s
ci / image (worker) (push) Failing after 13m8s
ci / bump (push) Has been skipped
- --cov pointed at libs/svcforge_core/domain, a path that does not exist (the package nests one level deeper). Coverage measured 0.00% of the code. Use the module form, which is layout-independent. - pip-audit --strict cannot audit our own editable, not-on-PyPI packages. Audit the locked dependency set instead and keep --strict. - bandit re-reports B608/B104, which ruff's S ruleset already enforces with justified per-line noqa it cannot see. Skipped in config, with reasons. - registry host was git.oci-oci; it is gitea.oci-oci. - integration job set SVCFORGE_PG_DSN; conftest reads SVCFORGE_TEST_DSN. |
||
|
|
50c2fe2a1e |
svcforge: reference implementation
ci / lint (push) Successful in 1m19s
ci / unit (push) Failing after 1m2s
ci / integration (push) Has been skipped
ci / types (push) Successful in 1m37s
ci / security (push) Failing after 38s
ci / dockerfile (push) Successful in 14s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
Complete working build of the system learn-python/ teaches. 164 tests, mypy --strict clean, domain coverage 99%. |