Files
svcforge/deploy/chart/templates/migrate-job.yaml
T
Nguyen Minh Phuc 31fa9165ff
ci / lint (push) Failing after 14m13s
ci / types (push) Has been skipped
ci / unit (push) Has been skipped
ci / integration (push) Has been skipped
ci / security (push) Has been skipped
ci / dockerfile (push) Has been skipped
ci / chart (push) Has been skipped
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
chart: give the migrate hook its own ServiceAccount
The Job ran as the api ServiceAccount, which is an ordinary chart resource.
Hooks are created before the release's ordinary manifests, so on a first
install that account does not exist and the Job never starts:

    Error creating: pods "svcforge-migrate-" is forbidden: error looking up
    service account svcforge/svcforge-api: serviceaccount "svcforge-api" not
    found

Not an ArgoCD quirk — helm orders hooks the same way, so both deploy paths
failed identically. It survived review because the chart was only ever checked
with `helm template` and `helm install --dry-run=server`, and neither creates a
Job. The pod is what fails, so only a real install can catch it.

The new account is a hook at weight -10, ahead of the Job at -5, and is bound
to no Role: the migration talks to Postgres and wants nothing from the
Kubernetes API. automountServiceAccountToken is off for the same reason.

Verified with a real `helm install` into a scratch namespace: STATUS deployed,
job Complete 1/1, four migrations applied, and the hook ServiceAccount was 27s
old against 10s for the ordinary ones — the ordering the bug turned on.
2026-07-20 06:33:31 +00:00

108 lines
4.4 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{{- if .Values.migrate.enabled }}
{{/*
Migrations run here and nowhere else.
Not on app startup: three services × N replicas racing the same DDL is how you get a
half-applied schema and a crash loop, and it makes "which pod migrated?" unanswerable.
A hook runs once, before any new pod starts, and its exit code gates the release.
hook-weight -5 puts it ahead of everything else in the pre-install/pre-upgrade phase.
hook-delete-policy before-hook-creation keeps the last run's pod around for `kubectl logs`
after a failure — the one time you actually want it — and clears it on the next attempt.
Deliberately no terminationGracePeriodSeconds: 60 here. Three Deployments carry it; a
migration is not one of them.
*/}}
{{/*
The hook needs its own ServiceAccount, and it has to be a hook itself.
It used to run as the api ServiceAccount, which is an ordinary chart resource. Hooks are
created before the release's ordinary manifests, so on a first install that account does
not exist yet and the Job never starts:
Error creating: pods "svcforge-migrate-" is forbidden: error looking up service
account svcforge/svcforge-api: serviceaccount "svcforge-api" not found
This is not an ArgoCD quirk. `helm install` orders hooks the same way, so it failed
identically on both paths. It survived review because the chart was only ever checked with
`helm template` and `helm install --dry-run=server`, and neither creates a Job — the pod is
what fails, so nothing short of a real install can catch it.
Weight -10 so it is created before the Job at -5. Deliberately bound to no Role: the
migration talks to Postgres and needs nothing from the Kubernetes API.
*/}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "svcforge.fullname" . }}-migrate
labels:
{{- include "svcforge.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-10"
"helm.sh/hook-delete-policy": before-hook-creation
automountServiceAccountToken: false
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "svcforge.fullname" . }}-migrate
labels:
{{- include "svcforge.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
# 0, not 3. A failed migration must fail the release. Retrying a DDL that just failed
# tends to turn one readable error into three, and then a green release on a schema
# nobody has looked at.
backoffLimit: 0
template:
metadata:
labels:
{{- include "svcforge.labels" . | nindent 8 }}
app.kubernetes.io/component: migrate
spec:
restartPolicy: Never
serviceAccountName: {{ include "svcforge.fullname" . }}-migrate
{{- with .Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
securityContext:
{{- include "svcforge.podSecurityContext" . | nindent 8 }}
containers:
- name: migrate
# Same image as the api, by digest. The migrations that ship are the ones the
# code that is about to run was built against — a separate image could drift.
image: {{ include "svcforge.image" (dict "ctx" $ "component" "api") }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
securityContext:
{{- include "svcforge.containerSecurityContext" . | nindent 12 }}
command: ["python", "-m", "svcforge_core.migrate"]
envFrom:
- secretRef:
name: {{ include "svcforge.secretName" . }}
env:
{{- include "svcforge.env" . | nindent 12 }}
- name: OTEL_SERVICE_NAME
value: svcforge-migrate
# Where services/api/Dockerfile copies migrations/ to. The Dockerfile sets the
# same value as an ENV; this states it in the manifest as well so the path is
# visible to anyone reading the Job rather than only to whoever opens the
# image. The two MUST agree — if one moves, move both.
- name: SVCFORGE_MIGRATIONS_DIR
value: /app/migrations
resources:
{{- toYaml .Values.migrate.resources | nindent 12 }}
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
{{- end }}