docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
+19
-24
@@ -1,13 +1,12 @@
|
||||
"""Task handlers.
|
||||
|
||||
Every handler here obeys one rule: running it twice must equal running it once.
|
||||
Every handler obeys one rule: running it twice must equal running it once.
|
||||
|
||||
A worker can be SIGKILLed after helm has installed the release but
|
||||
before the DB row says so; the lease expires; another worker claims the same task and runs
|
||||
this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
|
||||
and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
|
||||
places: a deterministic `release_name`, and adapters that state desired state
|
||||
(`helm upgrade --install`) instead of issuing imperative commands.
|
||||
A worker can be SIGKILLed after helm installed the release but before the DB row says so;
|
||||
the lease expires, another worker claims the same task, and this function runs again. A
|
||||
handler that is not idempotent gives the tenant two Elasticsearches and you a bill.
|
||||
Idempotency is bought in two places: a deterministic `release_name`, and adapters that
|
||||
state desired state (`helm upgrade --install`) instead of issuing imperative commands.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -85,11 +84,10 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
|
||||
{"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
|
||||
)
|
||||
except Exception:
|
||||
# The provision succeeded and the row is already READY; the notification is a
|
||||
# courtesy. Letting a webhook timeout propagate would fail the task, and the
|
||||
# retry would hit the READY early-return and drop the notification anyway — so a
|
||||
# flaky notifier would turn every provision into a "failed" task. Same guard the
|
||||
# reconciler puts around its own notify.
|
||||
# The provision succeeded and the row is READY; the notification is a courtesy.
|
||||
# Propagating a webhook timeout would fail the task, and the retry would hit the
|
||||
# READY early-return and drop the notification anyway — so a flaky notifier
|
||||
# would turn every provision into a "failed" task.
|
||||
log.exception("notify.failed", instance_id=str(inst.id))
|
||||
|
||||
|
||||
@@ -104,10 +102,9 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
|
||||
# swallows not-found, because the desired state — no release — is already true.
|
||||
await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
|
||||
|
||||
# Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
|
||||
# the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
|
||||
# and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
|
||||
# asked to delete. Failing loudly turns a silent ping-pong into one visible error.
|
||||
# Raise rather than ignore the CAS result. Swallowing it leaves the release gone, the
|
||||
# row on `state=ready` with a dangling endpoint, the task marked done — and 60 seconds
|
||||
# later the drift check re-provisions the thing the tenant asked to delete.
|
||||
if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
|
||||
raise HandlerError(
|
||||
f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
|
||||
@@ -146,10 +143,9 @@ async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
|
||||
async def handle_verify(task: Task, deps: WorkerDeps) -> None:
|
||||
"""Post-upgrade health probe. On failure, halt the whole rollout for this service type.
|
||||
|
||||
One column decides whether the fleet keeps rolling. The work-list query returns nothing
|
||||
while `rollout_state='halted'`, so a bad chart stops after the first tenant instead of
|
||||
after all of them. You clear it with SQL, deliberately: an automatic un-halt would just
|
||||
resume breaking things.
|
||||
The work-list query returns nothing while `rollout_state='halted'`, so a bad chart stops
|
||||
after the first tenant instead of all of them. Clearing it is a deliberate SQL statement:
|
||||
an automatic un-halt would resume breaking things.
|
||||
"""
|
||||
inst = await _load_instance(task, deps)
|
||||
releases = {r.name for r in await deps.provisioner.list_releases()}
|
||||
@@ -157,10 +153,9 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
|
||||
if inst.release_name in releases:
|
||||
return
|
||||
|
||||
# `returning` + a `where` on the update half tells us whether THIS call was the one
|
||||
# that halted the rollout. The halt itself is idempotent; the page is not. Without the
|
||||
# distinction, a verify that fails its full retry budget sends five identical
|
||||
# notifications for one incident, spread across the backoff curve.
|
||||
# `returning` plus a `where` on the update half says whether THIS call halted the
|
||||
# rollout. The halt is idempotent; the page is not. Without the distinction, a verify
|
||||
# that burns its full retry budget sends five identical notifications for one incident.
|
||||
async with deps.pool.connection() as conn, conn.cursor() as cur:
|
||||
await cur.execute(
|
||||
"""insert into catalog_versions (service_type, rollout_state)
|
||||
|
||||
Reference in New Issue
Block a user