Nguyen Minh Phuc
c76154aeaa
ci / lint (push) Successful in 34s
ci / unit (push) Successful in 1m41s
ci / types (push) Successful in 1m41s
ci / dockerfile (push) Successful in 18s
ci / security (push) Successful in 1m27s
ci / chart (push) Failing after 1m11s
ci / integration (push) Successful in 1m10s
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped
CORRECTNESS - lost-lease race: complete()/fail() did not check ownership, so a worker whose lease expired could mark a task done while another worker was running it, or requeue a task someone else owned. Reproduced, fixed with a CAS on (state, locked_by), pinned by two regression tests. - worker died on report failure: _run_one's docstring claimed no exception escapes the TaskGroup; fail()/complete() were outside the guarded block, so a DB blip cancelled every sibling provision on the pod. - claim query used an INNER join, which could strand a just-claimed task and report 'queue empty'. LEFT join. - InstanceRepo.set_error bypassed the state machine and had no callers. Deleted. - handle_deprovision ignored its CAS result, so a wrong-state instance kept a dangling endpoint and got re-provisioned by the drift check 60s later. - handle_verify re-notified on every retry: five pages for one halt. DEPLOY-BREAKING - the migration Job could never succeed: no Dockerfile copied migrations/, and migrate.py resolved the path relative to the source tree, which only works for an editable install. Added COPY + SVCFORGE_MIGRATIONS_DIR. - ServiceMonitor selector did not match the Service: API metrics never scraped. - SvcforgeReconcilerStale fired permanently from every pod, because the gauge is module-level and every service exports it as 0. Scoped to the reconciler job. - SvcforgeTaskFailed latched forever on a monotonic counter. Now increase()[15m]. - the digest guard accepted the all-zeros placeholder. - worker terminationGracePeriodSeconds was 60s against a 600s helm timeout. DEAD CODE THAT SHOULD NOT HAVE BEEN - adapters/k8s.py was never called, so tenant namespaces were never created and the first provision for a new team would fail. Wired into handle_provision. - adapters/redis.py was never imported by any service. Rate limiting is now wired into the API, failing open. - Settings.check_production() had no callers. Given an explicit environment and called from every entrypoint. OBSERVABILITY - the API never called obs.setup(): no JSON logs, no trace correlation, log_json silently inert. - LogNotifier's structured fields were discarded by the stdlib->structlog bridge. - bind_task_context cleared the 'service' binding for the life of every task. - split tasks_failed into task_attempts_failed and tasks_dead_lettered. SECURITY - trivy correctly blocked the worker/reconciler images: helm 3.16.2 and kubectl 1.31.2 carry CRITICAL Go stdlib CVEs. Bumped to helm 3.21.3 and kubectl 1.35.3, which also closes a four-minor skew against the v1.35.3 cluster. TESTS THAT COULD NOT FAIL - the concurrency cap test passed on a fully serial worker. - the alert/metric cross-check asserted a hardcoded list instead of reading the chart, so it could not catch a rename on the chart side. - fixed OTel tracer-provider pollution between test files. DOCS - ARCHITECTURE.md: mermaid diagrams, user stories, and the helm-vs-ArgoCD guarantee (verified with --dry-run=server). - AGENTS.md + CLAUDE.md. - prose sweep for back-and-forth phrasing across 19 files.
70 lines
3.1 KiB
YAML
70 lines
3.1 KiB
YAML
# The other half of "CI does not deploy".
|
|
#
|
|
# CI's last act is a commit that changes image.*.digest in deploy/chart/values.yaml.
|
|
# ArgoCD notices that commit and syncs. There is no kubeconfig in CI, no `helm upgrade` in
|
|
# a pipeline step, and no human running kubectl. If you want to know what is running in the
|
|
# cluster, you read git — that is the whole property, and a single `kubectl apply` from a
|
|
# laptop is what destroys it (selfHeal below exists to undo exactly that).
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: svcforge
|
|
namespace: argocd
|
|
finalizers:
|
|
# Without this, deleting the Application orphans every resource it created.
|
|
- resources-finalizer.argocd.argoproj.io
|
|
spec:
|
|
project: default
|
|
source:
|
|
repoURL: https://gitea.oci-oci.duckdns.org/gitea_admin/svcforge.git
|
|
targetRevision: master
|
|
path: deploy/chart
|
|
helm:
|
|
releaseName: svcforge
|
|
# No valueFiles override and no `parameters:` block. values.yaml in the repo is the
|
|
# single source of truth — a parameter here would be a second place the deployed
|
|
# digest could come from, invisible in the chart's own diff.
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: svcforge
|
|
syncPolicy:
|
|
automated:
|
|
# Delete resources removed from the chart. Safe because the chart owns only svcforge
|
|
# itself; tenant releases are created by the worker's helm calls and are not part of
|
|
# this Application, so prune cannot reach them.
|
|
prune: true
|
|
# Revert manual edits. A hotfix applied by hand is a lie the next sync tells on.
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
# The migrate Job is a helm pre-install/pre-upgrade hook. ArgoCD maps helm hooks onto
|
|
# its own PreSync phase, so migrations still run before any new pod starts, and a
|
|
# non-zero exit fails the sync instead of rolling out pods onto an unmigrated schema.
|
|
- ApplyOutOfSyncOnly=true
|
|
retry:
|
|
# 0, not 3, and for the same reason migrate-job.yaml sets backoffLimit: 0.
|
|
#
|
|
# An ArgoCD retry re-runs the WHOLE sync including the PreSync phase, and the migrate
|
|
# Job is a helm pre-install/pre-upgrade hook that ArgoCD maps onto PreSync. So
|
|
# `limit: 3` quietly reinstated the retry-a-failed-DDL behaviour that backoffLimit: 0
|
|
# exists to forbid — three attempts at the same failed migration, one readable error
|
|
# turned into three, against a schema that may now be half-applied.
|
|
#
|
|
# There is no recovery path here that a retry helps with. Recovery from a failed
|
|
# migration is a REVERT COMMIT: fix the SQL forward, push, and let ArgoCD sync the
|
|
# new revision. Rolling back the app image does not roll back DDL that already
|
|
# committed.
|
|
limit: 0
|
|
backoff:
|
|
duration: 20s
|
|
factor: 2
|
|
maxDuration: 3m
|
|
# The migrate Job is a hook, so ArgoCD deletes and recreates it every sync; its
|
|
# generated fields would otherwise show as permanent drift and the app would never
|
|
# report Synced.
|
|
ignoreDifferences:
|
|
- group: batch
|
|
kind: Job
|
|
jsonPointers:
|
|
- /spec/template/metadata/labels
|