Nguyen Minh Phuc
c76154aeaa
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.
77 lines
3.1 KiB
YAML
77 lines
3.1 KiB
YAML
{{- if .Values.migrate.enabled }}
|
||
{{/*
|
||
Migrations run here and nowhere else.
|
||
|
||
Not on app startup: three services × N replicas racing the same DDL is how you get a
|
||
half-applied schema and a crash loop, and it makes "which pod migrated?" unanswerable.
|
||
A hook runs once, before any new pod starts, and its exit code gates the release.
|
||
|
||
hook-weight -5 puts it ahead of everything else in the pre-install/pre-upgrade phase.
|
||
hook-delete-policy before-hook-creation keeps the last run's pod around for `kubectl logs`
|
||
after a failure — the one time you actually want it — and clears it on the next attempt.
|
||
|
||
Deliberately no terminationGracePeriodSeconds: 60 here. Three Deployments carry it; a
|
||
migration is not one of them.
|
||
*/}}
|
||
apiVersion: batch/v1
|
||
kind: Job
|
||
metadata:
|
||
name: {{ include "svcforge.fullname" . }}-migrate
|
||
labels:
|
||
{{- include "svcforge.labels" . | nindent 4 }}
|
||
app.kubernetes.io/component: migrate
|
||
annotations:
|
||
"helm.sh/hook": pre-install,pre-upgrade
|
||
"helm.sh/hook-weight": "-5"
|
||
"helm.sh/hook-delete-policy": before-hook-creation
|
||
spec:
|
||
# 0, not 3. A failed migration must fail the release. Retrying a DDL that just failed
|
||
# tends to turn one readable error into three, and then a green release on a schema
|
||
# nobody has looked at.
|
||
backoffLimit: 0
|
||
template:
|
||
metadata:
|
||
labels:
|
||
{{- include "svcforge.labels" . | nindent 8 }}
|
||
app.kubernetes.io/component: migrate
|
||
spec:
|
||
restartPolicy: Never
|
||
serviceAccountName: {{ include "svcforge.serviceAccountName" (dict "ctx" $ "component" "api") }}
|
||
{{- with .Values.image.pullSecrets }}
|
||
imagePullSecrets:
|
||
{{- toYaml . | nindent 8 }}
|
||
{{- end }}
|
||
securityContext:
|
||
{{- include "svcforge.podSecurityContext" . | nindent 8 }}
|
||
containers:
|
||
- name: migrate
|
||
# Same image as the api, by digest. The migrations that ship are the ones the
|
||
# code that is about to run was built against — a separate image could drift.
|
||
image: {{ include "svcforge.image" (dict "ctx" $ "component" "api") }}
|
||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||
securityContext:
|
||
{{- include "svcforge.containerSecurityContext" . | nindent 12 }}
|
||
command: ["python", "-m", "svcforge_core.migrate"]
|
||
envFrom:
|
||
- secretRef:
|
||
name: {{ include "svcforge.secretName" . }}
|
||
env:
|
||
{{- include "svcforge.env" . | nindent 12 }}
|
||
- name: OTEL_SERVICE_NAME
|
||
value: svcforge-migrate
|
||
# Where services/api/Dockerfile copies migrations/ to. The Dockerfile sets the
|
||
# same value as an ENV; this states it in the manifest as well so the path is
|
||
# visible to anyone reading the Job rather than only to whoever opens the
|
||
# image. The two MUST agree — if one moves, move both.
|
||
- name: SVCFORGE_MIGRATIONS_DIR
|
||
value: /app/migrations
|
||
resources:
|
||
{{- toYaml .Values.migrate.resources | nindent 12 }}
|
||
volumeMounts:
|
||
- name: tmp
|
||
mountPath: /tmp
|
||
volumes:
|
||
- name: tmp
|
||
emptyDir: {}
|
||
{{- end }}
|