review: fix 26 findings from a 4-agent audit
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped

CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.
This commit is contained in:
Nguyen Minh Phuc
2026-07-18 12:13:49 +00:00
parent 77d560ddae
commit c76154aeaa
45 changed files with 1520 additions and 216 deletions
+55 -4
View File
@@ -190,12 +190,63 @@ jobs:
dockerfile: "services/*/Dockerfile"
failure-threshold: warning
# --- stage 9b: the chart must render --------------------------------------------------
# Without this, a chart that does not template reaches ArgoCD and fails in the cluster,
# where the error surfaces as a sync failure with no PR attached to it. `helm template`
# is the real gate: it is what ArgoCD does, and _helpers.tpl's image helper calls `fail`
# on anything that is not a full sha256 digest.
chart:
runs-on: ubuntu-latest
needs: [lint]
env:
# Pinned by digest like gitleaks and trivy, and the same helm the worker and
# reconciler images carry — CI renders with the version that ships.
HELM: alpine/helm:3.21.3@sha256:35da09ba0716fc7c3cd63b6b31ee380a9c7662e95f29ab0e4ae962420afd315b
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: helm lint
run: |
docker run --rm -v "$PWD:/repo" -w /repo "$HELM" lint deploy/chart
- name: helm template (rejects unbumped digests)
# values.yaml ships all-zeros placeholders, so a bare `helm template` MUST fail.
# That is the guard working, not a broken chart — asserting the failure here is
# what stops the guard silently regressing into a prefix check again.
run: |
set -euo pipefail
if docker run --rm -v "$PWD:/repo" -w /repo "$HELM" \
template svcforge deploy/chart >/dev/null 2>&1; then
echo "FAIL: chart rendered against the placeholder digests in values.yaml." >&2
echo "The digest guard in _helpers.tpl is not guarding." >&2
exit 1
fi
echo "ok: placeholder digests rejected"
- name: helm template (renders with real digests)
# Dummy but well-formed digests: this checks the templates themselves render, with
# the two values-gated monitoring blocks explicitly on so they are covered too.
run: |
set -euo pipefail
A="sha256:$(printf 'a%.0s' $(seq 64))"
B="sha256:$(printf 'b%.0s' $(seq 64))"
C="sha256:$(printf 'c%.0s' $(seq 64))"
docker run --rm -v "$PWD:/repo" -w /repo "$HELM" \
template svcforge deploy/chart \
--set image.api.digest="$A" \
--set image.worker.digest="$B" \
--set image.reconciler.digest="$C" \
--set serviceMonitor.enabled=true \
--set prometheusRule.enabled=true \
>/dev/null
echo "ok: chart renders"
# --- stage 10: build -> trivy -> push by digest --------------------------------------
image:
runs-on: ubuntu-latest
# Every gate above is required. An image is not built until all of them are green,
# which is what makes "the digest CI pushed is a digest that passed everything" true.
needs: [types, unit, integration, security, dockerfile]
needs: [types, unit, integration, security, dockerfile, chart]
permissions:
contents: read
strategy:
@@ -314,9 +365,9 @@ jobs:
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
# A bot token with contents:write on this repo and nothing else. Note what is
# absent: no kubeconfig, no cluster credential, no ArgoCD API token. CI's maximum
# blast radius is a bad commit, which is revertable.
# A bot token with contents:write on this repo and nothing else: no kubeconfig,
# no cluster credential, no ArgoCD API token. CI's maximum blast radius is a bad
# commit, which is revertable.
token: ${{ secrets.CI_BOT_TOKEN }}
ref: master
- uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0