docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s

The comment pass is prose-only: every distinct "why" is kept, the
narration around it is not. Verified by AST-comparing each changed file
against HEAD with docstrings stripped — only the two files below differ
in executable code.

Two real fixes fell out of the read-through:

* The CLI documented itself as never touching the database, then called
  load_settings(), which requires SVCFORGE_PG_DSN. It refused to start
  without a Postgres URL it never opens. It now has its own two-field
  ClientSettings; the orphaned api_url/api_token are dropped from
  Settings, where nothing else read them.
* repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS
  comment run together above the wrong symbol.

USER_GUIDE.md is the caller-facing guide the README only gestured at:
auth, catalog, every endpoint with curl, the lifecycle, the error table,
rate limiting, the CLI, client generation, an end-to-end poll loop.

It records two facts about the live deployment rather than documenting a
flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP
behind it, so the API logs "JWKS warm-up failed" at startup and every
/v1 request is a 401. And `helm repo list` in the worker returns no
repositories, so the three bitnamilegacy/ catalog entries cannot resolve
at provision time; only the oci:// entries can.

make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 11:18:57 +00:00
parent 7918dc2b37
commit c53734d2bc
22 changed files with 936 additions and 872 deletions
+18 -20
View File
@@ -3,11 +3,11 @@
The distinction between the first two is the difference between a 30-second blip and a
fleet-wide outage:
* `/healthz` (liveness) answers "is this process wedged?" A failure here gets the
container KILLED. It must therefore touch NOTHING external. Wire it to the DB and a
20-second Postgres failover restarts every pod at once; they come back, find the DB
still down, and CrashLoopBackOff with exponential restart delays — so the fleet is now
down for minutes after the database recovered.
* `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the
container, so it must touch nothing external. Wired to the DB, a 20-second Postgres
failover restarts every pod at once; they come back, find the DB still down, and
CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the
database recovered.
* `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes
it from the Service endpoints. It is allowed to check dependencies, and it recovers by
itself the moment the check passes.
@@ -26,10 +26,10 @@ from services.api.models import ErrorBody
router = APIRouter(tags=["ops"])
# No PROMETHEUS_MULTIPROC_DIR here, deliberately: it exists for prefork servers where each
# worker process holds a slice of the counters. One uvicorn process per container means
# the default in-process registry is already correct, and multiproc mode would add a
# shared temp dir, a cleanup obligation, and a class of stale-file bugs for nothing.
# No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each
# process holds a slice of the counters. One uvicorn process per container makes the
# in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup
# obligation, and a class of stale-file bugs for nothing.
@router.get("/healthz", status_code=status.HTTP_200_OK)
@@ -45,10 +45,9 @@ async def healthz() -> dict[str, str]:
async def readyz(pool: PoolDep) -> dict[str, str]:
"""Readiness. Postgres only.
Postgres-only is the rule, and Redis is the temptation. Redis holds derived state
rate-limit buckets, caches — and everything degrades gracefully without it. Put it in
this check and an Upstash hiccup marks every pod unready, Kubernetes empties the
Service, and a cache outage becomes a total API outage.
Redis is the temptation and stays out: it holds derived state that degrades gracefully,
so checking it here would let an Upstash hiccup mark every pod unready, empty the
Service, and turn a cache outage into a total API outage.
"""
try:
async with pool.connection() as conn, conn.cursor() as cur:
@@ -68,14 +67,13 @@ async def readyz(pool: PoolDep) -> dict[str, str]:
async def metrics(request: Request) -> Response:
"""The Prometheus scrape endpoint.
A route rather than `app.mount("/metrics", make_asgi_app())`, for two reasons. A
Starlette `Mount` compiles to `^/metrics(?P<path>/.*)$`, which does not match a bare
`/metrics` — the exact URL every scrape config uses and a `Mount` is invisible to
OpenAPI, while the deliverable asks for `/metrics` in `openapi.json`.
A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount`
compiles to `^/metrics(?P<path>/.*)$`, which does not match the bare `/metrics` every
scrape config uses, and a Mount is invisible to OpenAPI.
The encoding is still prometheus_client's: `choose_encoder` reads the Accept header and
picks the exposition format (Prometheus text vs OpenMetrics) with its matching content
type. Hand-rolling either is how you end up serving text/plain that a scraper rejects.
The encoding stays prometheus_client's `choose_encoder` reads Accept and picks the
exposition format with its matching content type. Hand-rolling it serves text/plain a
scraper rejects.
"""
encoder, content_type = choose_encoder(request.headers.get("Accept", ""))
return Response(content=encoder(REGISTRY), media_type=content_type)
+27 -31
View File
@@ -1,12 +1,12 @@
"""The tenant-facing API.
Two rules run through every handler here:
Two rules run through every handler:
* **AuthZ is the WHERE clause.** No handler ever compares `inst.team` to the caller's
team, because the repo never returns another team's row to compare. A wrong-team id is
a 404. 403 would confirm the id exists, which is the leak.
* **The instance and its task commit together.** A committed instance with no task is an
instance that never provisions and that nothing will ever retry.
* **AuthZ is the WHERE clause.** No handler compares `inst.team` to the caller's team,
because the repo never returns another team's row to compare. A wrong-team id is a 404;
403 would confirm the id exists.
* **The instance and its task commit together.** A committed instance with no task never
provisions and nothing retries it.
"""
from __future__ import annotations
@@ -30,9 +30,9 @@ from services.api.models import CreateInstanceRequest, ErrorBody, InstanceRespon
from svcforge_core.domain.models import CatalogEntry, Instance, TaskKind
from svcforge_core.domain.states import IllegalTransition, InstanceState, transition
# Declared on the router so every error shape lands in openapi.json under ErrorBody.
# The exception handler already renders this at runtime; without declaring it, generated
# clients see the contract for 2xx only and invent their own guess for the rest.
# Declared on the router so every error shape lands in openapi.json under ErrorBody. The
# exception handler already renders these at runtime; undeclared, a generated client sees
# the contract for 2xx only and guesses the rest.
ERROR_RESPONSES: dict[int | str, dict[str, Any]] = {
401: {"model": ErrorBody, "description": "Missing or invalid credentials"},
404: {"model": ErrorBody, "description": "No such instance, or not this team's"},
@@ -46,14 +46,12 @@ router = APIRouter(prefix="/v1/instances", tags=["instances"], responses=ERROR_R
def release_name_for(team: str, service_type: str, instance_id: UUID) -> str:
"""The helm release name. Deterministic, and `unique` in the schema.
This is the idempotency anchor. A worker that dies after `helm install` but before it
marks the task done will retry, compute the same name, and `helm upgrade --install`
onto the same release instead of creating a second one. Derive it from anything that
is not already durable — a timestamp, a random suffix, the retry count — and a retry
provisions a duplicate.
The idempotency anchor: a worker that dies after `helm install` but before marking the
task done retries, computes the same name, and upgrades the same release instead of
creating a second one. Derive it from anything not already durable — a timestamp, a
random suffix, the retry count — and a retry provisions a duplicate.
Truncated to the uuid's first 8 chars to stay inside the 53-char limit helm imposes
on release names (Kubernetes label values, minus room for chart-generated suffixes).
Truncated to the uuid's first 8 chars to stay inside helm's 53-char release-name limit.
"""
return f"{team}-{service_type}-{str(instance_id)[:8]}"
@@ -66,9 +64,8 @@ def namespace_for(team: str) -> str:
def _resolve(catalog: dict[str, CatalogEntry], service_type: str, size: str) -> CatalogEntry:
"""Look up service_type + size, or raise the right 4xx.
The two failures are different HTTP problems and the spec asks for different codes:
an unknown service_type is a resource that does not exist (404); an unknown size for a
real service_type is a body the server understood and cannot process (422).
Two different HTTP problems: an unknown service_type is a resource that does not exist
(404), an unknown size for a real one is a body understood and unprocessable (422).
"""
entry = catalog.get(service_type)
if entry is None:
@@ -107,9 +104,9 @@ async def create_instance(
) -> Instance:
"""Accept a provisioning request. 202, never 201.
Nothing is provisioned when this returns. The row exists and a task is queued; a
worker will do the work seconds or minutes from now. 201 Created would be a lie about
a resource that does not exist yet, and clients would stop polling.
Nothing is provisioned when this returns: the row exists and a task is queued, and a
worker does the work seconds or minutes later. 201 Created would be a lie about a
resource that does not exist yet, and clients would stop polling.
"""
entry = _resolve(catalog, body.service_type, body.size)
@@ -123,9 +120,9 @@ async def create_instance(
state=InstanceState.REQUESTED,
namespace=namespace_for(team),
release_name=release_name_for(team, body.service_type, instance_id),
# Pinned from the catalog AT CREATION TIME, not read from the catalog later.
# This column records what is actually deployed; bumping catalog.yaml must show up
# as drift the reconciler can see, not silently rewrite history.
# Pinned at creation time, not read from the catalog later. The column records what
# is deployed, so bumping catalog.yaml shows up as drift the reconciler can see
# rather than silently rewriting history.
chart_version=entry.chart_version,
expires_at=now + timedelta(days=body.ttl_days) if body.ttl_days is not None else None,
created_at=now,
@@ -181,12 +178,11 @@ async def delete_instance(
) -> Instance:
"""state -> deleting, enqueue deprovision. 202: the helm uninstall has not happened yet.
Ordering note. `InstanceRepo.update_state` owns its own connection, so the CAS and the
enqueue cannot share one transaction without reaching around the repo. Given two
statements, the order is chosen for its failure mode: CAS first, enqueue second. A
crash in between leaves an instance in `deleting` with no task, which the reconciler's
sweep re-enqueues. The other order leaves a deprovision task pointing at a `ready`
instance, and a worker would tear down a live service nobody asked to delete.
`InstanceRepo.update_state` owns its own connection, so the CAS and the enqueue cannot
share a transaction without reaching around the repo. Given two statements, the order is
chosen for its failure mode: a crash between CAS and enqueue leaves an instance in
`deleting` with no task, which the reconciler's sweep re-enqueues. The reverse would
leave a deprovision task on a `ready` instance and tear down a live service.
"""
inst = await instances.get(instance_id, team)
if inst is None: