docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s

The comment pass is prose-only: every distinct "why" is kept, the
narration around it is not. Verified by AST-comparing each changed file
against HEAD with docstrings stripped — only the two files below differ
in executable code.

Two real fixes fell out of the read-through:

* The CLI documented itself as never touching the database, then called
  load_settings(), which requires SVCFORGE_PG_DSN. It refused to start
  without a Postgres URL it never opens. It now has its own two-field
  ClientSettings; the orphaned api_url/api_token are dropped from
  Settings, where nothing else read them.
* repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS
  comment run together above the wrong symbol.

USER_GUIDE.md is the caller-facing guide the README only gestured at:
auth, catalog, every endpoint with curl, the lifecycle, the error table,
rate limiting, the CLI, client generation, an end-to-end poll loop.

It records two facts about the live deployment rather than documenting a
flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP
behind it, so the API logs "JWKS warm-up failed" at startup and every
/v1 request is a 401. And `helm repo list` in the worker returns no
repositories, so the three bitnamilegacy/ catalog entries cannot resolve
at provision time; only the oci:// entries can.

make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 11:18:57 +00:00
parent 7918dc2b37
commit c53734d2bc
22 changed files with 936 additions and 872 deletions
+19 -24
View File
@@ -1,13 +1,12 @@
"""Task handlers.
Every handler here obeys one rule: running it twice must equal running it once.
Every handler obeys one rule: running it twice must equal running it once.
A worker can be SIGKILLed after helm has installed the release but
before the DB row says so; the lease expires; another worker claims the same task and runs
this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
places: a deterministic `release_name`, and adapters that state desired state
(`helm upgrade --install`) instead of issuing imperative commands.
A worker can be SIGKILLed after helm installed the release but before the DB row says so;
the lease expires, another worker claims the same task, and this function runs again. A
handler that is not idempotent gives the tenant two Elasticsearches and you a bill.
Idempotency is bought in two places: a deterministic `release_name`, and adapters that
state desired state (`helm upgrade --install`) instead of issuing imperative commands.
"""
from __future__ import annotations
@@ -85,11 +84,10 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
{"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
)
except Exception:
# The provision succeeded and the row is already READY; the notification is a
# courtesy. Letting a webhook timeout propagate would fail the task, and the
# retry would hit the READY early-return and drop the notification anyway — so a
# flaky notifier would turn every provision into a "failed" task. Same guard the
# reconciler puts around its own notify.
# The provision succeeded and the row is READY; the notification is a courtesy.
# Propagating a webhook timeout would fail the task, and the retry would hit the
# READY early-return and drop the notification anyway — so a flaky notifier
# would turn every provision into a "failed" task.
log.exception("notify.failed", instance_id=str(inst.id))
@@ -104,10 +102,9 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
# swallows not-found, because the desired state — no release — is already true.
await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
# Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
# the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
# and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
# asked to delete. Failing loudly turns a silent ping-pong into one visible error.
# Raise rather than ignore the CAS result. Swallowing it leaves the release gone, the
# row on `state=ready` with a dangling endpoint, the task marked done — and 60 seconds
# later the drift check re-provisions the thing the tenant asked to delete.
if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
raise HandlerError(
f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
@@ -146,10 +143,9 @@ async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
async def handle_verify(task: Task, deps: WorkerDeps) -> None:
"""Post-upgrade health probe. On failure, halt the whole rollout for this service type.
One column decides whether the fleet keeps rolling. The work-list query returns nothing
while `rollout_state='halted'`, so a bad chart stops after the first tenant instead of
after all of them. You clear it with SQL, deliberately: an automatic un-halt would just
resume breaking things.
The work-list query returns nothing while `rollout_state='halted'`, so a bad chart stops
after the first tenant instead of all of them. Clearing it is a deliberate SQL statement:
an automatic un-halt would resume breaking things.
"""
inst = await _load_instance(task, deps)
releases = {r.name for r in await deps.provisioner.list_releases()}
@@ -157,10 +153,9 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
if inst.release_name in releases:
return
# `returning` + a `where` on the update half tells us whether THIS call was the one
# that halted the rollout. The halt itself is idempotent; the page is not. Without the
# distinction, a verify that fails its full retry budget sends five identical
# notifications for one incident, spread across the backoff curve.
# `returning` plus a `where` on the update half says whether THIS call halted the
# rollout. The halt is idempotent; the page is not. Without the distinction, a verify
# that burns its full retry budget sends five identical notifications for one incident.
async with deps.pool.connection() as conn, conn.cursor() as cur:
await cur.execute(
"""insert into catalog_versions (service_type, rollout_state)