review: fix 26 findings from a 4-agent audit
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.
This commit is contained in:
@@ -190,12 +190,63 @@ jobs:
|
||||
dockerfile: "services/*/Dockerfile"
|
||||
failure-threshold: warning
|
||||
|
||||
# --- stage 9b: the chart must render --------------------------------------------------
|
||||
# Without this, a chart that does not template reaches ArgoCD and fails in the cluster,
|
||||
# where the error surfaces as a sync failure with no PR attached to it. `helm template`
|
||||
# is the real gate: it is what ArgoCD does, and _helpers.tpl's image helper calls `fail`
|
||||
# on anything that is not a full sha256 digest.
|
||||
chart:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [lint]
|
||||
env:
|
||||
# Pinned by digest like gitleaks and trivy, and the same helm the worker and
|
||||
# reconciler images carry — CI renders with the version that ships.
|
||||
HELM: alpine/helm:3.21.3@sha256:35da09ba0716fc7c3cd63b6b31ee380a9c7662e95f29ab0e4ae962420afd315b
|
||||
steps:
|
||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||
|
||||
- name: helm lint
|
||||
run: |
|
||||
docker run --rm -v "$PWD:/repo" -w /repo "$HELM" lint deploy/chart
|
||||
|
||||
- name: helm template (rejects unbumped digests)
|
||||
# values.yaml ships all-zeros placeholders, so a bare `helm template` MUST fail.
|
||||
# That is the guard working, not a broken chart — asserting the failure here is
|
||||
# what stops the guard silently regressing into a prefix check again.
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if docker run --rm -v "$PWD:/repo" -w /repo "$HELM" \
|
||||
template svcforge deploy/chart >/dev/null 2>&1; then
|
||||
echo "FAIL: chart rendered against the placeholder digests in values.yaml." >&2
|
||||
echo "The digest guard in _helpers.tpl is not guarding." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "ok: placeholder digests rejected"
|
||||
|
||||
- name: helm template (renders with real digests)
|
||||
# Dummy but well-formed digests: this checks the templates themselves render, with
|
||||
# the two values-gated monitoring blocks explicitly on so they are covered too.
|
||||
run: |
|
||||
set -euo pipefail
|
||||
A="sha256:$(printf 'a%.0s' $(seq 64))"
|
||||
B="sha256:$(printf 'b%.0s' $(seq 64))"
|
||||
C="sha256:$(printf 'c%.0s' $(seq 64))"
|
||||
docker run --rm -v "$PWD:/repo" -w /repo "$HELM" \
|
||||
template svcforge deploy/chart \
|
||||
--set image.api.digest="$A" \
|
||||
--set image.worker.digest="$B" \
|
||||
--set image.reconciler.digest="$C" \
|
||||
--set serviceMonitor.enabled=true \
|
||||
--set prometheusRule.enabled=true \
|
||||
>/dev/null
|
||||
echo "ok: chart renders"
|
||||
|
||||
# --- stage 10: build -> trivy -> push by digest --------------------------------------
|
||||
image:
|
||||
runs-on: ubuntu-latest
|
||||
# Every gate above is required. An image is not built until all of them are green,
|
||||
# which is what makes "the digest CI pushed is a digest that passed everything" true.
|
||||
needs: [types, unit, integration, security, dockerfile]
|
||||
needs: [types, unit, integration, security, dockerfile, chart]
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
@@ -314,9 +365,9 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||
with:
|
||||
# A bot token with contents:write on this repo and nothing else. Note what is
|
||||
# absent: no kubeconfig, no cluster credential, no ArgoCD API token. CI's maximum
|
||||
# blast radius is a bad commit, which is revertable.
|
||||
# A bot token with contents:write on this repo and nothing else: no kubeconfig,
|
||||
# no cluster credential, no ArgoCD API token. CI's maximum blast radius is a bad
|
||||
# commit, which is revertable.
|
||||
token: ${{ secrets.CI_BOT_TOKEN }}
|
||||
ref: master
|
||||
- uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
|
||||
|
||||
Reference in New Issue
Block a user