c76154aeaa
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.
95 lines
4.0 KiB
YAML
95 lines
4.0 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: {{ include "svcforge.fullname" . }}-worker
|
|
labels:
|
|
{{- include "svcforge.labels" . | nindent 4 }}
|
|
app.kubernetes.io/component: worker
|
|
spec:
|
|
# Plain replicas, no HPA. Concurrency is bounded twice over — by replicas here and by a
|
|
# semaphore inside the claim loop — and the SKIP LOCKED claim makes both safe. The day
|
|
# replicas: 2 stops keeping up, add an HPA. Not before.
|
|
replicas: {{ .Values.worker.replicas }}
|
|
selector:
|
|
matchLabels:
|
|
{{- include "svcforge.selectorLabels" (dict "ctx" $ "component" "worker") | nindent 6 }}
|
|
template:
|
|
metadata:
|
|
labels:
|
|
{{- include "svcforge.labels" . | nindent 8 }}
|
|
{{- include "svcforge.selectorLabels" (dict "ctx" $ "component" "worker") | nindent 8 }}
|
|
spec:
|
|
serviceAccountName: {{ include "svcforge.serviceAccountName" (dict "ctx" $ "component" "worker") }}
|
|
{{- with .Values.image.pullSecrets }}
|
|
imagePullSecrets:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
# The load-bearing one. On SIGTERM the loop stops claiming and finishes the task in
|
|
# hand. Chaos experiment 2 asserts the pod exits 0 inside it. Lower this and a
|
|
# rolling deploy starts orphaning tasks to lease expiry.
|
|
#
|
|
# 660s, and the number is derived, not chosen. The task being drained is a
|
|
# `helm upgrade --install`, and this MUST exceed the longest that call can run before
|
|
# it returns:
|
|
# - settings.py helm_timeout_s = 300 (default)
|
|
# - HelmProvisioner timeout_s = 600 (default, the ceiling)
|
|
# 600s is the value to beat; 660 leaves a 60s margin for the pool to close and the
|
|
# process to exit cleanly. At the old 60s the kubelet SIGKILLed the worker mid-helm —
|
|
# precisely the orphaning this setting exists to prevent. If either timeout above is
|
|
# raised, raise this past it first.
|
|
terminationGracePeriodSeconds: 660
|
|
securityContext:
|
|
{{- include "svcforge.podSecurityContext" . | nindent 8 }}
|
|
containers:
|
|
- name: worker
|
|
image: {{ include "svcforge.image" (dict "ctx" $ "component" "worker") }}
|
|
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
|
securityContext:
|
|
{{- include "svcforge.containerSecurityContext" . | nindent 12 }}
|
|
ports:
|
|
- name: metrics
|
|
containerPort: 9000
|
|
envFrom:
|
|
- secretRef:
|
|
name: {{ include "svcforge.secretName" . }}
|
|
env:
|
|
{{- include "svcforge.env" . | nindent 12 }}
|
|
# The claim loop stamps locked_by with this. Per-pod, so a stuck lease names
|
|
# the pod that holds it — runbook entry 1 depends on that.
|
|
- name: SVCFORGE_WORKER_ID
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: metadata.name
|
|
- name: SVCFORGE_WORKER_CONCURRENCY
|
|
value: {{ .Values.worker.concurrency | quote }}
|
|
- name: OTEL_SERVICE_NAME
|
|
value: svcforge-worker
|
|
# No liveness probe. A worker mid-provision is legitimately busy for minutes;
|
|
# a probe here is a way to kill a healthy provision and learn nothing.
|
|
resources:
|
|
{{- toYaml .Values.worker.resources | nindent 12 }}
|
|
volumeMounts:
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
# helm writes cache/config/repositories under $HELM_*_HOME, which the image
|
|
# points at /tmp/helm. Without this, every helm call fails on a read-only fs.
|
|
- name: helm-home
|
|
mountPath: /tmp/helm
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir: {}
|
|
- name: helm-home
|
|
emptyDir: {}
|
|
{{- with .Values.nodeSelector }}
|
|
nodeSelector:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with .Values.tolerations }}
|
|
tolerations:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|
|
{{- with .Values.affinity }}
|
|
affinity:
|
|
{{- toYaml . | nindent 8 }}
|
|
{{- end }}
|