docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN

The comment pass is prose-only: every distinct "why" is kept, the
narration around it is not. Verified by AST-comparing each changed file
against HEAD with docstrings stripped — only the two files below differ
in executable code.

Two real fixes fell out of the read-through:

* The CLI documented itself as never touching the database, then called
  load_settings(), which requires SVCFORGE_PG_DSN. It refused to start
  without a Postgres URL it never opens. It now has its own two-field
  ClientSettings; the orphaned api_url/api_token are dropped from
  Settings, where nothing else read them.
* repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS
  comment run together above the wrong symbol.

USER_GUIDE.md is the caller-facing guide the README only gestured at:
auth, catalog, every endpoint with curl, the lifecycle, the error table,
rate limiting, the CLI, client generation, an end-to-end poll loop.

It records two facts about the live deployment rather than documenting a
flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP
behind it, so the API logs "JWKS warm-up failed" at startup and every
/v1 request is a 401. And `helm repo list` in the worker returns no
repositories, so the three bitnamilegacy/ catalog entries cannot resolve
at provision time; only the oci:// entries can.

make lint clean, 76 unit + 111 integration tests pass.

services/reconciler/main.py

@@ -1,29 +1,28 @@
 """The control loop.
 
-Every other service in svcforge is edge-triggered: a tenant POSTs, a row appears, a worker
-claims it. Edge-triggered systems are correct exactly as long as nothing is ever missed —
-and things are missed. A worker is SIGKILLed holding a lease. An operator runs
-`helm uninstall` by hand. A pod dies between the CAS and the enqueue. Nobody sends an event
-for any of that, because the thing that would have sent it is the thing that died.
+Every other service here is edge-triggered: a tenant POSTs, a row appears, a worker claims
+it. That is correct only as long as nothing is missed, and things are missed — a worker
+SIGKILLed holding a lease, an operator running `helm uninstall` by hand, a pod dying
+between the CAS and the enqueue. Nothing sends an event for any of it, because the thing
+that would have sent it is the thing that died.
 
 So: level-triggered. Every 60 seconds, compare the world to the database and enqueue what
-is missing. The four checks below do not know or care what went wrong, or whether anything
-did; they are the same code on the happy path and after an outage. That property is the
-entire reason this service exists, and it is why each check is written as a *query for
-work*, never as a reaction to an event.
+is missing. The four checks below do not know what went wrong, or whether anything did;
+they are the same code on the happy path and after an outage. That is why each is written
+as a query for work rather than a reaction to an event.
 
 Three rules hold the design together:
 
-* **Singleton.** `replicas: 1`, `strategy: Recreate` in the chart. Two reconcilers
-  double-enqueue drift and race on TTL. There is no leader election here on purpose: the
-  correct lease for that lives in Postgres next to the data, not in a Redis lock, and
-  until there is a second replica to elect between, an election is a subsystem that can
-  only fail. One pod, and the `SvcforgeReconcilerStale` alert is what notices it is gone.
-* **Each check is independent.** One failing check must not skip the other three. A helm
-  binary that cannot reach the API server must not stop TTLs from expiring.
-* **Enqueue, never act.** The reconciler diagnoses; workers treat. It writes task rows and
-  instance states, and never calls `helm install`. The one exception is reading — the drift
-  check lists the live releases, because seeing reality is the job.
+* **Singleton.** `replicas: 1`, `strategy: Recreate`. Two reconcilers double-enqueue drift
+  and race on TTL. No leader election on purpose — the right lease for that lives in
+  Postgres next to the data, and until there is a second replica to elect between, an
+  election is a subsystem that can only fail. The `SvcforgeReconcilerStale` alert notices
+  when the one pod is gone.
+* **Each check is independent.** A helm binary that cannot reach the API server must not
+  stop TTLs from expiring.
+* **Enqueue, never act.** The reconciler diagnoses and workers treat: it writes task rows
+  and instance states and never calls `helm install`. Reading is the exception, since
+  seeing reality is the job.
 """
 
 from __future__ import annotations
@@ -63,9 +62,9 @@ log = get_logger("svcforge.reconciler")
 class ReconcilerDeps:
     """Everything a check is allowed to touch. Built once in main(), passed down.
 
-    Same shape as `WorkerDeps` for the same reason: the checks take `deps` instead of
-    reaching for globals, so the integration tests below run every check against a real
-    Postgres and a `FakeProvisioner` without a cluster anywhere in sight.
+    Same shape as `WorkerDeps` and for the same reason: checks take `deps` instead of
+    reaching for globals, so the integration tests run every check against a real Postgres
+    and a `FakeProvisioner` with no cluster in sight.
     """
 
     pool: DictPool
@@ -90,21 +89,17 @@ class ReconcilerDeps:
 async def check_drift(deps: ReconcilerDeps) -> None:
     """The live helm releases versus what the database believes.
 
-    This is the only check that looks outside Postgres, and the only one that can catch the
-    failure nothing else can: someone ran `helm uninstall` by hand, or a node was drained
-    and the release never came back. The DB still says `ready` and still hands the tenant an
-    endpoint that resolves to nothing.
+    The only check that looks outside Postgres, and the only one that catches someone
+    running `helm uninstall` by hand or a drained node whose release never came back — where
+    the DB still says `ready` and still hands the tenant an endpoint resolving to nothing.
 
-    Two directions, two very different answers:
+    Two directions, two different answers:
 
-    * **Release gone, DB says `ready`** -> re-enqueue provision. Safe, because provisioning
-      is `helm upgrade --install` against a deterministic release name: converging on
-      desired state, not a blind re-install.
-    * **Release exists, DB knows nothing** -> log at error with release and namespace, and
-      stop. **Never delete in v1.** The reconciler's view of "the DB knows nothing" is one
-      query against one database; the release might belong to another team, another tool,
-      or a migration half-finished. Deleting on that evidence is how an automated system
-      takes down production faster than any human could. A human reads the log and decides.
+    * **Release gone, DB says `ready`** -> re-enqueue provision. Safe because provisioning
+      is `helm upgrade --install` against a deterministic release name.
+    * **Release exists, DB knows nothing** -> log at error and stop. **Never delete in v1.**
+      "The DB knows nothing" is one query against one database, and the release might belong
+      to another team, another tool, or a half-finished migration. A human decides.
     """
     with tracer().start_as_current_span("helm.list"):
         releases = await deps.provisioner.list_releases()
@@ -132,31 +127,30 @@ async def check_drift(deps: ReconcilerDeps) -> None:
                 {"instance_id": str(inst.id), "team": inst.team},
             )
         except Exception:
-            # The task is already committed; the notification is a courtesy. A webhook
-            # timing out must not abandon the rest of the sweep — the instances after this
-            # one in the loop have the same problem and nobody else is coming to find them.
+            # The task is committed; the notification is a courtesy. A webhook timing out
+            # must not abandon the rest of the sweep — the instances after this one have the
+            # same problem and nobody else is coming to find them.
             log.exception("notify.failed", instance_id=str(inst.id))
 
     known = await deps.reconcile.known_releases()
     for name, namespace in sorted(live - known):
-        # error, not warning: this is a resource nobody is billing for and nobody owns.
-        # It will sit here every 60s until a human deletes it or adopts it. That is the
-        # intended pressure.
+        # error, not warning: a resource nobody owns and nobody is billing for. It repeats
+        # every 60s until a human deletes or adopts it, which is the intended pressure.
         log.error("drift.orphan_release", release=name, namespace=namespace, action="none (v1 never deletes)")
 
 
 async def check_lease_expiry(deps: ReconcilerDeps) -> None:
     """Tasks whose worker died -> back to `queued`.
 
-    A lease. No lock survives a power cut: a worker SIGKILLed mid-provision
-    leaves `state='running'` with `locked_by` set and nobody running it, and no amount of
-    cleanup code in the worker helps, because the worker is the part that died. `locked_at`
-    plus a timeout is the only thing that recovers the row, which is why `locked_at` exists.
+    A lease, not a lock: no lock survives a power cut. A worker SIGKILLed mid-provision
+    leaves `state='running'` with `locked_by` set and nobody running it, and cleanup code in
+    the worker cannot help because the worker is what died. `locked_at` plus a timeout is
+    the only thing that recovers the row.
 
-    The 5-minute default must exceed the longest a healthy task can hold a lease, or the
-    reconciler hands a still-running provision to a second worker. Handlers are idempotent,
-    so that is survivable, though it still costs a duplicated helm run — which is why
-    `lease_seconds` sits above helm's `--timeout`.
+    The 5-minute default must exceed the longest a healthy task can hold a lease, or a
+    still-running provision is handed to a second worker. Handlers are idempotent so that is
+    survivable, but it costs a duplicated helm run — hence `lease_seconds` > helm's
+    `--timeout`.
     """
     freed = await deps.tasks.reset_expired_leases(deps.settings.lease_seconds)
     if freed:
@@ -166,14 +160,13 @@ async def check_lease_expiry(deps: ReconcilerDeps) -> None:
 async def check_ttl(deps: ReconcilerDeps) -> None:
     """Expired instances -> `deleting`, plus a deprovision task.
 
-    The line item that stops a demo cluster from becoming a permanent cloud bill. Also the
-    sweep the API's DELETE route depends on: it CASes to `deleting` and enqueues in two
-    statements, and a crash in between lands here on the next tick.
+    What stops a demo cluster becoming a permanent cloud bill, and the sweep the API's
+    DELETE route depends on: DELETE CASes and enqueues in two statements, and a crash
+    between them lands here on the next tick.
 
-    Idempotent by construction — the work list excludes anything that already has a queued
-    or running deprovision, and the CAS and the insert share one transaction. Without that
-    guard, a deprovision that takes longer than 60 seconds gets a second task on the next
-    tick, and a third on the tick after.
+    Idempotent by construction — the work list excludes anything with a queued or running
+    deprovision, and the CAS and insert share one transaction. Without that, a deprovision
+    taking longer than 60 seconds collects a new task every tick.
     """
     for inst in await deps.reconcile.due_for_deprovision():
         task_id = await deps.reconcile.enqueue_deprovision(inst.id)
@@ -192,20 +185,18 @@ async def check_ttl(deps: ReconcilerDeps) -> None:
 async def check_version_drift(deps: ReconcilerDeps) -> None:
     """The day-2 rollout: the work-list query, one service type at a time.
 
-    Everything that makes this safe is somewhere else, which is the point:
+    Everything that makes it safe is somewhere else, which is the point:
 
     * `list_upgradable` limits to `max_in_flight` and returns nothing while
       `rollout_state='halted'`, so a bad chart stops after one tenant.
-    * `schedule_upgrade_at` turns the tenant's maintenance window into a `run_after`; the
-      queue does the waiting, in `where run_after <= now()`. There is no scheduler here and
-      there must not be one — a task parked in Postgres until 03:00 Sunday survives a
-      reconciler restart, and an in-memory timer does not.
-    * `security: true` in the catalog bypasses the window. A CVE with a public exploit does
-      not wait until Sunday.
+    * `schedule_upgrade_at` turns the maintenance window into a `run_after` and the queue
+      does the waiting, in `where run_after <= now()`. No scheduler here, and there must not
+      be one: a task parked in Postgres until 03:00 Sunday survives a restart, a timer does
+      not.
+    * `security: true` in the catalog bypasses the window.
 
-    A bad window spec is this instance's problem, not the fleet's: log it and move to the
-    next one. Failing the whole check would let one tenant's typo freeze everyone's
-    security rollout.
+    A bad window spec is one instance's problem: log it and move on. Failing the check would
+    let one tenant's typo freeze everyone's security rollout.
     """
     now = deps.clock.now()
 
@@ -260,23 +251,20 @@ CHECKS: dict[str, Callable[[ReconcilerDeps], Awaitable[None]]] = {
 async def tick(deps: ReconcilerDeps) -> None:
     """One pass: all four checks, then the gauges, then the heartbeat.
 
-    Checks first, gauges second: `svcforge_queue_depth` is read straight after the checks
-    that add to the queue, so the value scraped is the value the tick left behind rather
-    than one from before its own work.
+    Checks first, gauges second, so `svcforge_queue_depth` reports what this tick left
+    behind rather than what preceded its own work.
 
-    The heartbeat is set unconditionally, and that is deliberate. It answers "is the loop
-    running", not "is everything fine" — the checks have their own alerts. Gating it on
-    success would make `SvcforgeReconcilerStale` fire for a helm blip and mean two things
-    at once, and an alert that means two things gets muted.
+    The heartbeat is set unconditionally. It answers "is the loop running", not "is
+    everything fine" — the checks have their own alerts. Gating it on success would make
+    `SvcforgeReconcilerStale` fire for a helm blip and mean two things at once, and an alert
+    that means two things gets muted.
 
-    The whole tick runs inside one span, which is a considered exception to "manual spans go
-    around helm calls only". That rule exists so the API does not hand-roll spans that
-    `opentelemetry-instrument` already creates for it. Nothing auto-instruments the
-    reconciler: without a span here it emits no traces at all, and — because
-    `inject_traceparent` serialises the *active* context — every task it enqueues would be
-    written with a null `traceparent` and be unjoinable to the tick that decided to create
-    it. One span per tick is what makes "why was this instance re-provisioned at 03:00?" a
-    question the traces can answer.
+    The whole tick runs in one span, a considered exception to "manual spans wrap helm calls
+    only". That rule keeps the API from hand-rolling spans `opentelemetry-instrument`
+    already makes; nothing auto-instruments the reconciler, so without this it emits no
+    traces at all and — since `inject_traceparent` serialises the *active* context — every
+    task it enqueues would carry a null `traceparent` and be unjoinable to the tick that
+    created it.
     """
     with tracer().start_as_current_span("reconciler.tick"):
         await _run_checks(deps)
@@ -291,12 +279,11 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
         try:
             await check(deps)
         except Exception:  # the tick is the error boundary
-            # The swallow is the design. These four checks share nothing but a database
-            # handle, and the value of a level-triggered loop is that it keeps running: an
-            # unreachable cluster must not stop TTLs from expiring, and one tenant's broken
+            # The swallow is the design. The four checks share nothing but a database
+            # handle, and a level-triggered loop is only worth having if it keeps running:
+            # an unreachable cluster must not stop TTLs expiring, and one tenant's broken
             # window spec must not stop drift detection. This means "this check achieved
-            # nothing for 60 seconds", which the log says out loud. It never means "the
-            # reconciler stops".
+            # nothing for 60 seconds", never "the reconciler stops".
             log.exception("check.failed", check=name)
 
     try:
@@ -311,10 +298,10 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
 async def run_reconciler(deps: ReconcilerDeps, stop: asyncio.Event) -> None:
     """Tick, sleep, repeat, until told to stop.
 
-    Tick first, then sleep: a pod that has just been restarted should reconcile now, not in
-    sixty seconds. Fixed interval rather than a fixed period — a tick that overruns simply
-    delays the next one, instead of stacking a second tick on top of the first, which for a
-    singleton would be exactly the concurrent reconciler `replicas: 1` exists to prevent.
+    Tick first, then sleep: a just-restarted pod should reconcile now, not in sixty seconds.
+    Fixed interval rather than fixed period, so a tick that overruns delays the next one
+    instead of stacking a second on top — which for a singleton is exactly the concurrent
+    reconciler `replicas: 1` exists to prevent.
     """
     while not stop.is_set():
         await tick(deps)
@@ -353,13 +340,13 @@ async def _amain(once: bool, own_team: str, max_in_flight: int) -> None:
 
     try:
         if once:
-            # One pass and exit: the acceptance path, and how you drive a reconcile by hand
-            # from a shell. No metrics server — nothing would ever scrape it.
+            # One pass and exit: the acceptance path, and how to drive a reconcile by hand.
+            # No metrics server — nothing would ever scrape it.
             await tick(deps)
             return
 
-        # settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT still overrides it,
-        # through pydantic rather than a second CLI option, so the port has one definition.
+        # settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT overrides it through
+        # pydantic rather than a second CLI option, so the port has one definition.
         start_metrics_server(settings.metrics_port)
 
         stop = asyncio.Event()