docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN

The comment pass is prose-only: every distinct "why" is kept, the
narration around it is not. Verified by AST-comparing each changed file
against HEAD with docstrings stripped — only the two files below differ
in executable code.

Two real fixes fell out of the read-through:

* The CLI documented itself as never touching the database, then called
  load_settings(), which requires SVCFORGE_PG_DSN. It refused to start
  without a Postgres URL it never opens. It now has its own two-field
  ClientSettings; the orphaned api_url/api_token are dropped from
  Settings, where nothing else read them.
* repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS
  comment run together above the wrong symbol.

USER_GUIDE.md is the caller-facing guide the README only gestured at:
auth, catalog, every endpoint with curl, the lifecycle, the error table,
rate limiting, the CLI, client generation, an end-to-end poll loop.

It records two facts about the live deployment rather than documenting a
flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP
behind it, so the API logs "JWKS warm-up failed" at startup and every
/v1 request is a 401. And `helm repo list` in the worker returns no
repositories, so the three bitnamilegacy/ catalog entries cannot resolve
at provision time; only the oci:// entries can.

make lint clean, 76 unit + 111 integration tests pass.

services/worker/handlers.py

@@ -1,13 +1,12 @@
 """Task handlers.
 
-Every handler here obeys one rule: running it twice must equal running it once.
+Every handler obeys one rule: running it twice must equal running it once.
 
-A worker can be SIGKILLed after helm has installed the release but
-before the DB row says so; the lease expires; another worker claims the same task and runs
-this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
-and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
-places: a deterministic `release_name`, and adapters that state desired state
-(`helm upgrade --install`) instead of issuing imperative commands.
+A worker can be SIGKILLed after helm installed the release but before the DB row says so;
+the lease expires, another worker claims the same task, and this function runs again. A
+handler that is not idempotent gives the tenant two Elasticsearches and you a bill.
+Idempotency is bought in two places: a deterministic `release_name`, and adapters that
+state desired state (`helm upgrade --install`) instead of issuing imperative commands.
 """
 
 from __future__ import annotations
@@ -85,11 +84,10 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
                 {"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
             )
         except Exception:
-            # The provision succeeded and the row is already READY; the notification is a
-            # courtesy. Letting a webhook timeout propagate would fail the task, and the
-            # retry would hit the READY early-return and drop the notification anyway — so a
-            # flaky notifier would turn every provision into a "failed" task. Same guard the
-            # reconciler puts around its own notify.
+            # The provision succeeded and the row is READY; the notification is a courtesy.
+            # Propagating a webhook timeout would fail the task, and the retry would hit the
+            # READY early-return and drop the notification anyway — so a flaky notifier
+            # would turn every provision into a "failed" task.
             log.exception("notify.failed", instance_id=str(inst.id))
 
 
@@ -104,10 +102,9 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
     # swallows not-found, because the desired state — no release — is already true.
     await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
 
-    # Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
-    # the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
-    # and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
-    # asked to delete. Failing loudly turns a silent ping-pong into one visible error.
+    # Raise rather than ignore the CAS result. Swallowing it leaves the release gone, the
+    # row on `state=ready` with a dangling endpoint, the task marked done — and 60 seconds
+    # later the drift check re-provisions the thing the tenant asked to delete.
     if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
         raise HandlerError(
             f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
@@ -146,10 +143,9 @@ async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
 async def handle_verify(task: Task, deps: WorkerDeps) -> None:
     """Post-upgrade health probe. On failure, halt the whole rollout for this service type.
 
-    One column decides whether the fleet keeps rolling. The work-list query returns nothing
-    while `rollout_state='halted'`, so a bad chart stops after the first tenant instead of
-    after all of them. You clear it with SQL, deliberately: an automatic un-halt would just
-    resume breaking things.
+    The work-list query returns nothing while `rollout_state='halted'`, so a bad chart stops
+    after the first tenant instead of all of them. Clearing it is a deliberate SQL statement:
+    an automatic un-halt would resume breaking things.
     """
     inst = await _load_instance(task, deps)
     releases = {r.name for r in await deps.provisioner.list_releases()}
@@ -157,10 +153,9 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
     if inst.release_name in releases:
         return
 
-    # `returning` + a `where` on the update half tells us whether THIS call was the one
-    # that halted the rollout. The halt itself is idempotent; the page is not. Without the
-    # distinction, a verify that fails its full retry budget sends five identical
-    # notifications for one incident, spread across the backoff curve.
+    # `returning` plus a `where` on the update half says whether THIS call halted the
+    # rollout. The halt is idempotent; the page is not. Without the distinction, a verify
+    # that burns its full retry budget sends five identical notifications for one incident.
     async with deps.pool.connection() as conn, conn.cursor() as cur:
         await cur.execute(
             """insert into catalog_versions (service_type, rollout_state)

services/worker/main.py

@@ -1,10 +1,9 @@
 """The claim loop.
 
-Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report.
-That is the whole design, and the restraint is the point: LISTEN/NOTIFY would shave the
-latency, is fire-and-forget so it can never replace the poll anyway, is strictly extra
-code, and does not exist on pgbouncer's transaction pooler. The poll is not a placeholder
-for something better.
+Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report. The
+poll is not a placeholder for something better: LISTEN/NOTIFY would shave latency, but it
+is fire-and-forget so it can never replace the poll, and it does not exist on pgbouncer's
+transaction pooler.
 """
 
 from __future__ import annotations
@@ -35,17 +34,16 @@ log = obs.get_logger("svcforge.worker")
 async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
     """Run a terminal report, and never let its failure escape.
 
-    Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
-    TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
-    DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
-    not just this one. The task itself is safe either way: it stays `running` and the
-    reconciler's lease sweep returns it to the queue. Losing the report costs one lease
-    interval; losing the siblings costs their work.
+    `_run_one` runs inside a TaskGroup, which cancels every sibling the moment one child
+    raises — so a DB blip during `tasks.fail()` would abort every other in-flight provision
+    on this pod. The task itself is safe either way: it stays `running` and the lease sweep
+    returns it to the queue. Losing the report costs one lease interval; losing the siblings
+    costs their work.
     """
     try:
         if not await coro:
-            # The lease was stolen while we were working: another worker owns this task
-            # now and is mid-run. Reporting is theirs to do, not ours.
+            # The lease was stolen while we were working: another worker owns this task now
+            # and is mid-run. Reporting is theirs, not ours.
             log.warning("lease lost before report; another worker owns this task", task_id=task_id)
     except Exception:
         log.exception("could not report task %s (%s); lease will expire", task_id, what)
@@ -55,10 +53,8 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
     """Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
     worker_id = deps.settings.worker_id
     try:
-        # Every log line from here carries instance_id/task_id/team. Bound once, at claim,
-        # rather than passed down: the alternative is threading three arguments through
-        # every function that might log, and the first one anyone forgets is the one you
-        # need at 3am.
+        # Every log line from here carries instance_id/task_id/team. Bound once at claim
+        # rather than threaded through every function that might log.
         obs.bind_task_context(task.instance_id, task.id, team=task.team or "unknown")
         log.info("task claimed", kind=task.kind.value, attempt=task.attempts)
         obs.TASKS_CLAIMED.labels(kind=task.kind.value).inc()
@@ -72,9 +68,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
             )
             return
 
-        # Re-parent to the span that enqueued this task. Without the stored traceparent
-        # the worker's span starts a brand-new trace, and the POST that caused the work
-        # is in a different trace to the helm call that did it.
+        # Re-parent to the span that enqueued this task. Without the stored traceparent the
+        # worker's span starts a new trace, putting the POST that caused the work in a
+        # different trace from the helm call that did it.
         ctx = obs.context_from_traceparent(task.traceparent)
         started = time.monotonic()
         with obs.tracer().start_as_current_span(
@@ -108,10 +104,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
                     "fail",
                 )
             else:
-                # Only provisions go in the provision histogram. The buckets run 10s..1800s
-                # because they were sized for helm installs; a sub-second `verify` dropped
-                # into the same series drags the p95 down and quietly stops
-                # SvcforgeProvisionSlow from ever firing.
+                # Only provisions go in the provision histogram. Its buckets run 10s..1800s
+                # for helm installs, so a sub-second `verify` in the same series drags the
+                # p95 down and quietly stops SvcforgeProvisionSlow from ever firing.
                 if task.kind is TaskKind.PROVISION:
                     obs.PROVISION_TIME.observe(time.monotonic() - started)
                 await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
@@ -122,10 +117,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
 async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
     """Claim and run until told to stop, then drain what is in flight.
 
-    Draining is what makes a rolling deploy invisible. Exiting the `async with` block
-    awaits every in-flight handler, so a pod that is being replaced finishes the provision
-    it already started instead of abandoning it half-done for the lease to clean up
-    five minutes later.
+    Draining is what makes a rolling deploy invisible: exiting the `async with` awaits every
+    in-flight handler, so a pod being replaced finishes the provision it started instead of
+    abandoning it for the lease to clean up five minutes later.
     """
     sem = asyncio.Semaphore(deps.settings.worker_concurrency)
     worker_id = deps.settings.worker_id
@@ -158,9 +152,8 @@ async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
 async def _amain() -> None:
     settings: Settings = load_settings()
 
-    # Before anything else: nothing logged above this line is structured, and the metrics
-    # the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
-    # registry is up.
+    # Before anything else: nothing above this line logs structured, and the metrics the
+    # SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until it runs.
     obs.setup("svcforge-worker", settings)
     settings.check_production()
     obs.start_metrics_server(settings.metrics_port)