test_timeout_kills_the_whole_process_group asserted that `pgrep -f 'sleep 300'`
returns nothing. That is machine-global: it matches a leftover from an earlier run of
the same test, any unrelated 'sleep 300' on the box, and the shell running pgrep,
whose own command line contains the pattern being searched for.
Observed three spurious matches on a dev box, which produced a red run that looked
like a Python 3.14 regression in the helm timeout kill. It was not — the same test
failed identically on 3.12. A test that fails for reasons unrelated to the code is as
useless as one that cannot fail.
The sleep duration is now derived from the pid, so it cannot collide with another run,
and the test asserts up front that its own pattern matches nothing before it starts.
Verified: 3 consecutive passes on each of Python 3.12 and 3.14.
trivy took the worker image 39 -> 18 -> 5 findings across two version bumps, and
the last 5 (4x golang.org/x/net, 1x Go stdlib) live in kubectl v1.36.2 — the
newest kubectl that exists. No release clears them; upstream has not rebuilt
against the patched Go yet. Chasing the version further has no end.
kubectl was in that image for exactly one call: ensure_namespace before helm.
`helm upgrade --install --create-namespace` does the same thing, idempotently,
as part of the install it already runs. So the binary goes, and its vendored CVEs
go with it. read_secret had no callers.
Tradeoff recorded: the namespace no longer gets an svcforge.io/team label, since
--create-namespace makes a bare one. Nothing reads that label today.
CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
lease expired could mark a task done while another worker was running it, or
requeue a task someone else owned. Reproduced, fixed with a CAS on
(state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.
DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
migrate.py resolved the path relative to the source tree, which only works for
an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.
DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
called from every entrypoint.
OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.
SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
which also closes a four-minor skew against the v1.35.3 cluster.
TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.
DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.
Nguyen Minh Phuc