Files
svcforge/deploy/chart/templates/worker-deployment.yaml
Nguyen Minh Phuc c76154aeaa
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
review: fix 26 findings from a 4-agent audit
CORRECTNESS
- lost-lease race: complete()/fail() did not check ownership, so a worker whose
  lease expired could mark a task done while another worker was running it, or
  requeue a task someone else owned. Reproduced, fixed with a CAS on
  (state, locked_by), pinned by two regression tests.
- worker died on report failure: _run_one's docstring claimed no exception
  escapes the TaskGroup; fail()/complete() were outside the guarded block, so a
  DB blip cancelled every sibling provision on the pod.
- claim query used an INNER join, which could strand a just-claimed task and
  report 'queue empty'. LEFT join.
- InstanceRepo.set_error bypassed the state machine and had no callers. Deleted.
- handle_deprovision ignored its CAS result, so a wrong-state instance kept a
  dangling endpoint and got re-provisioned by the drift check 60s later.
- handle_verify re-notified on every retry: five pages for one halt.

DEPLOY-BREAKING
- the migration Job could never succeed: no Dockerfile copied migrations/, and
  migrate.py resolved the path relative to the source tree, which only works for
  an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR.
- ServiceMonitor selector did not match the Service: API metrics never scraped.
- SvcforgeReconcilerStale fired permanently from every pod, because the gauge is
  module-level and every service exports it as 0. Scoped to the reconciler job.
- SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m].
- the digest guard accepted the all-zeros placeholder.
- worker terminationGracePeriodSeconds was 60s against a 600s helm timeout.

DEAD CODE THAT SHOULD NOT HAVE BEEN
- adapters/k8s.py was never called, so tenant namespaces were never created and
  the first provision for a new team would fail. Wired into handle_provision.
- adapters/redis.py was never imported by any service. Rate limiting is now wired
  into the API, failing open.
- Settings.check_production() had no callers. Given an explicit environment and
  called from every entrypoint.

OBSERVABILITY
- the API never called obs.setup(): no JSON logs, no trace correlation, log_json
  silently inert.
- LogNotifier's structured fields were discarded by the stdlib->structlog bridge.
- bind_task_context cleared the 'service' binding for the life of every task.
- split tasks_failed into task_attempts_failed and tasks_dead_lettered.

SECURITY
- trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl
  1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3,
  which also closes a four-minor skew against the v1.35.3 cluster.

TESTS THAT COULD NOT FAIL
- the concurrency cap test passed on a fully serial worker.
- the alert/metric cross-check asserted a hardcoded list instead of reading the
  chart, so it could not catch a rename on the chart side.
- fixed OTel tracer-provider pollution between test files.

DOCS
- ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD
  guarantee (verified with --dry-run=server).
- AGENTS.md + CLAUDE.md.
- prose sweep for back-and-forth phrasing across 19 files.
2026-07-18 12:13:49 +00:00

95 lines
4.0 KiB
YAML

apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "svcforge.fullname" . }}-worker
labels:
{{- include "svcforge.labels" . | nindent 4 }}
app.kubernetes.io/component: worker
spec:
# Plain replicas, no HPA. Concurrency is bounded twice over — by replicas here and by a
# semaphore inside the claim loop — and the SKIP LOCKED claim makes both safe. The day
# replicas: 2 stops keeping up, add an HPA. Not before.
replicas: {{ .Values.worker.replicas }}
selector:
matchLabels:
{{- include "svcforge.selectorLabels" (dict "ctx" $ "component" "worker") | nindent 6 }}
template:
metadata:
labels:
{{- include "svcforge.labels" . | nindent 8 }}
{{- include "svcforge.selectorLabels" (dict "ctx" $ "component" "worker") | nindent 8 }}
spec:
serviceAccountName: {{ include "svcforge.serviceAccountName" (dict "ctx" $ "component" "worker") }}
{{- with .Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
# The load-bearing one. On SIGTERM the loop stops claiming and finishes the task in
# hand. Chaos experiment 2 asserts the pod exits 0 inside it. Lower this and a
# rolling deploy starts orphaning tasks to lease expiry.
#
# 660s, and the number is derived, not chosen. The task being drained is a
# `helm upgrade --install`, and this MUST exceed the longest that call can run before
# it returns:
# - settings.py helm_timeout_s = 300 (default)
# - HelmProvisioner timeout_s = 600 (default, the ceiling)
# 600s is the value to beat; 660 leaves a 60s margin for the pool to close and the
# process to exit cleanly. At the old 60s the kubelet SIGKILLed the worker mid-helm —
# precisely the orphaning this setting exists to prevent. If either timeout above is
# raised, raise this past it first.
terminationGracePeriodSeconds: 660
securityContext:
{{- include "svcforge.podSecurityContext" . | nindent 8 }}
containers:
- name: worker
image: {{ include "svcforge.image" (dict "ctx" $ "component" "worker") }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
securityContext:
{{- include "svcforge.containerSecurityContext" . | nindent 12 }}
ports:
- name: metrics
containerPort: 9000
envFrom:
- secretRef:
name: {{ include "svcforge.secretName" . }}
env:
{{- include "svcforge.env" . | nindent 12 }}
# The claim loop stamps locked_by with this. Per-pod, so a stuck lease names
# the pod that holds it — runbook entry 1 depends on that.
- name: SVCFORGE_WORKER_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: SVCFORGE_WORKER_CONCURRENCY
value: {{ .Values.worker.concurrency | quote }}
- name: OTEL_SERVICE_NAME
value: svcforge-worker
# No liveness probe. A worker mid-provision is legitimately busy for minutes;
# a probe here is a way to kill a healthy provision and learn nothing.
resources:
{{- toYaml .Values.worker.resources | nindent 12 }}
volumeMounts:
- name: tmp
mountPath: /tmp
# helm writes cache/config/repositories under $HELM_*_HOME, which the image
# points at /tmp/helm. Without this, every helm call fails on a read-only fs.
- name: helm-home
mountPath: /tmp/helm
volumes:
- name: tmp
emptyDir: {}
- name: helm-home
emptyDir: {}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}