docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
+22
-27
@@ -1,8 +1,8 @@
|
||||
"""Dependency injection: how a handler gets a pool, a repo, a catalog, and a team.
|
||||
|
||||
Everything expensive — the pool, the JWKS client, the parsed catalog — is built once in
|
||||
`lifespan` and parked on `app.state`. These functions only hand it out. A `Depends` that
|
||||
does I/O per request is a `Depends` that does that I/O on every request forever.
|
||||
`lifespan` and parked on `app.state`; these functions only hand it out. A `Depends` that
|
||||
does I/O per request does that I/O on every request forever.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -22,10 +22,9 @@ from svcforge_core.repo.instances import InstanceRepo
|
||||
from svcforge_core.repo.tasks import TaskRepo
|
||||
from svcforge_core.settings import Settings
|
||||
|
||||
# The algorithm allow-list is the whole point of naming algorithms explicitly.
|
||||
# `jwt.decode(..., algorithms=...)` without it accepts whatever the *token* claims in its
|
||||
# own header — including `none`, and including HS256 verified with the RSA public key as
|
||||
# an HMAC secret. Both are forgery. The list is not configuration.
|
||||
# The algorithm allow-list is not configuration. Without it, `jwt.decode` accepts whatever
|
||||
# the *token* claims in its own header — including `none`, and including HS256 verified
|
||||
# with the RSA public key as an HMAC secret. Both are forgery.
|
||||
ALLOWED_ALGORITHMS = ["RS256"]
|
||||
|
||||
# What `auth_disabled` returns. Settings.check_production() refuses that flag in prod.
|
||||
@@ -42,9 +41,9 @@ _bearer = HTTPBearer(auto_error=False)
|
||||
def _unauthorized() -> HTTPException:
|
||||
"""One shape for every auth failure.
|
||||
|
||||
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: all the
|
||||
same 401 with the same body. Telling a caller *which* one turns the endpoint into an
|
||||
oracle they can tune a forgery against.
|
||||
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: the same 401
|
||||
with the same body. Naming which one turns the endpoint into an oracle a forger can tune
|
||||
against.
|
||||
"""
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
@@ -68,8 +67,8 @@ async def get_pool(request: Request) -> DictPool:
|
||||
def get_catalog(request: Request) -> dict[str, CatalogEntry]:
|
||||
"""The catalog, parsed once at startup.
|
||||
|
||||
Read from disk per request and a mid-flight edit to catalog.yaml changes the answer
|
||||
between two requests of the same deploy. Load it at startup; a change is a restart.
|
||||
Read per request, a mid-flight edit to catalog.yaml would change the answer between two
|
||||
requests of the same deploy. A catalog change is a restart.
|
||||
"""
|
||||
catalog: dict[str, CatalogEntry] = request.app.state.catalog
|
||||
return catalog
|
||||
@@ -102,16 +101,14 @@ async def get_current_team(
|
||||
|
||||
jwks_client: PyJWKClient | None = getattr(request.app.state, "jwks_client", None)
|
||||
if jwks_client is None:
|
||||
# Auth is on but there is no key source. Fail closed. Answering 500 here would be
|
||||
# honest about the cause and would also let a misconfigured deploy be told apart
|
||||
# from a bad token; 401 is the same answer a forger gets.
|
||||
# Auth is on but there is no key source. Fail closed. A 500 would be honest about
|
||||
# the cause and would also let a forger tell a misconfigured deploy from a bad token.
|
||||
raise _unauthorized()
|
||||
|
||||
try:
|
||||
# PyJWKClient keeps its own TTL cache, so this is a dict lookup on the hot path.
|
||||
# It is only blocking on a cache MISS (key rotation) — hence to_thread, which
|
||||
# costs a thread hop we take a handful of times a day rather than an event loop
|
||||
# stalled on someone else's HTTP call once per rotation.
|
||||
# PyJWKClient keeps a TTL cache, so this is a dict lookup on the hot path and only
|
||||
# blocks on a miss (key rotation) — hence to_thread, a thread hop a few times a day
|
||||
# rather than an event loop stalled on someone else's HTTP call.
|
||||
signing_key = await _signing_key(jwks_client, creds.credentials)
|
||||
claims: dict[str, Any] = jwt.decode(
|
||||
creds.credentials,
|
||||
@@ -139,10 +136,9 @@ async def get_current_team(
|
||||
async def _signing_key(client: PyJWKClient, token: str) -> jwt.PyJWK:
|
||||
"""Fetch the signing key off the event loop.
|
||||
|
||||
PyJWKClient.get_signing_key_from_jwt() does a synchronous urlopen on a cache miss.
|
||||
Called directly from `async def`, that blocks the loop — every other in-flight request
|
||||
on this worker stops until the identity provider answers, and if it hangs, so does the
|
||||
pod, and /readyz keeps saying it is fine.
|
||||
`get_signing_key_from_jwt()` does a synchronous urlopen on a cache miss. Called directly
|
||||
from `async def` it blocks the loop: every other in-flight request stops until the IdP
|
||||
answers, and if the IdP hangs so does the pod, with /readyz still saying it is fine.
|
||||
"""
|
||||
return await asyncio.to_thread(client.get_signing_key_from_jwt, token)
|
||||
|
||||
@@ -159,11 +155,10 @@ async def rate_limit(
|
||||
) -> None:
|
||||
"""Per-team rate limiting. One Redis command per check, and it fails OPEN.
|
||||
|
||||
Failing open is the entire policy. Redis holds derived state; losing it must degrade
|
||||
the platform, never stop it. A limiter that fails closed converts a cache outage into
|
||||
a total outage, which is a strictly worse incident than the burst it was protecting
|
||||
against — so `RateLimiter.check` swallows its own errors and returns `allowed=True`.
|
||||
The 429 below therefore only ever comes from a real, counted overage.
|
||||
Redis holds derived state, so losing it must degrade the platform rather than stop it: a
|
||||
limiter that fails closed turns a cache outage into a total outage, a worse incident
|
||||
than the burst it was guarding against. `RateLimiter.check` swallows its own errors and
|
||||
returns `allowed=True`, so the 429 below only comes from a real, counted overage.
|
||||
"""
|
||||
limiter = get_rate_limiter(request)
|
||||
if limiter is None:
|
||||
|
||||
+43
-52
@@ -1,7 +1,7 @@
|
||||
"""The app factory and its lifespan.
|
||||
|
||||
`create_app(settings)` is a factory, not a module-level `app = FastAPI()`, for one reason:
|
||||
a test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
|
||||
`create_app(settings)` is a factory rather than a module-level `app = FastAPI()` because a
|
||||
test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
|
||||
environment at import time — before any fixture can say otherwise.
|
||||
"""
|
||||
|
||||
@@ -32,23 +32,21 @@ log = obs.get_logger("svcforge.api")
|
||||
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
"""Open the pool, yield, close the pool.
|
||||
|
||||
A lifespan context, not the deprecated startup/shutdown event decorators: those cannot
|
||||
express "this resource lives for exactly as long as the app", and give you no place to
|
||||
put the teardown next to the setup. Closing the pool matters — an unclosed pool means
|
||||
connections linger server-side after SIGTERM, and on a pooled Postgres with a small
|
||||
connection budget a few rolling deploys exhaust it.
|
||||
A lifespan context, not the deprecated startup/shutdown decorators: those cannot express
|
||||
"this resource lives exactly as long as the app" and leave no place to put teardown next
|
||||
to setup. Closing matters — an unclosed pool leaves connections open server-side after
|
||||
SIGTERM, and on a pooled Postgres with a small budget a few rolling deploys exhaust it.
|
||||
|
||||
(The old decorator's name is spelled nowhere in this package on purpose: CI greps for
|
||||
the literal string, and a comment quoting it fails the gate just as loudly as a call.)
|
||||
(The old decorator's name is spelled nowhere here on purpose: CI greps for the literal
|
||||
string, so a comment quoting it fails the gate as loudly as a call would.)
|
||||
"""
|
||||
settings: Settings = app.state.settings
|
||||
|
||||
app.state.catalog = load_catalog(settings.catalog_path)
|
||||
|
||||
# Redis is optional by construction. `make_redis` returns None when no DSN is set, and
|
||||
# every consumer treats None as "skip" — so a deployment without Redis loses rate
|
||||
# limiting and keeps everything else. Built here rather than per request because a
|
||||
# connection pool per request is a connection pool per request.
|
||||
# Redis is optional by construction: `make_redis` returns None when no DSN is set and
|
||||
# every consumer treats None as "skip", so a deployment without Redis loses rate
|
||||
# limiting and keeps everything else. Built once here, not per request.
|
||||
redis = make_redis(settings)
|
||||
app.state.redis = redis
|
||||
app.state.rate_limiter = (
|
||||
@@ -61,16 +59,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
await pool.open(wait=True)
|
||||
app.state.pool = pool
|
||||
|
||||
# The pool is open from here on, so everything below is inside the try: an exception
|
||||
# in JWKS setup must still close it, or a crash-looping pod leaks a connection per
|
||||
# restart until the database refuses new ones.
|
||||
# The pool is open from here, so everything below is inside the try: an exception in
|
||||
# JWKS setup must still close it, or a crash-looping pod leaks a connection per restart
|
||||
# until the database refuses new ones.
|
||||
try:
|
||||
if settings.jwks_url and not settings.auth_disabled:
|
||||
client = PyJWKClient(settings.jwks_url, cache_keys=True, lifespan=300)
|
||||
app.state.jwks_client = client
|
||||
# Warm the cache off the loop so the first authenticated request does not pay
|
||||
# a blocking urlopen. Best-effort: a slow identity provider must not stop the
|
||||
# pod from starting — a cache miss later just costs one to_thread hop.
|
||||
# Warm the cache off the loop so the first authenticated request does not pay a
|
||||
# blocking urlopen. Best-effort: a slow IdP must not stop the pod from starting,
|
||||
# and a miss later costs one to_thread hop.
|
||||
try:
|
||||
await asyncio.to_thread(client.get_signing_keys)
|
||||
except Exception: # deliberate catch-all: startup must not hinge on the IdP being up
|
||||
@@ -87,9 +85,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
|
||||
# --------------------------------------------------------------------------- API docs
|
||||
|
||||
# Everything a caller needs that the generated schema cannot express on its own. Kept next
|
||||
# to create_app rather than in a README because /docs is what someone integrating actually
|
||||
# reads, and a README in this repo is not something they have.
|
||||
# What the generated schema cannot express. Kept next to create_app because /docs is what
|
||||
# someone integrating reads, and they do not have this repo. USER_GUIDE.md is the longer form.
|
||||
API_DESCRIPTION = """
|
||||
Provision managed service instances into Kubernetes. The catalog offers Elasticsearch,
|
||||
Redis and Postgres, plus two deliberately tiny entries — `podinfo` and `nginx` — for
|
||||
@@ -144,21 +141,19 @@ OPENAPI_TAGS = [
|
||||
async def _http_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Render HTTPException bodies as ErrorBody, so every error has one shape.
|
||||
|
||||
Handlers raise `detail={"code": ..., "message": ...}`; FastAPI's default would nest
|
||||
that under `{"detail": {...}}`. Plain-string details (raised by FastAPI itself, e.g.
|
||||
a 405) are wrapped so clients never have to branch on the body's type.
|
||||
Handlers raise `detail={"code": ..., "message": ...}`, which FastAPI's default would
|
||||
nest under `{"detail": {...}}`. Plain-string details (a framework 405, say) are wrapped
|
||||
so clients never branch on the body's type.
|
||||
|
||||
Registered on starlette's HTTPException, not fastapi's. fastapi.HTTPException is a
|
||||
subclass, and Starlette matches handlers by walking type(exc).__mro__, so a handler
|
||||
keyed on the subclass never fires for a framework-raised 404 or 405 — which are
|
||||
starlette.HTTPException instances. Keying on the parent catches both: app handlers
|
||||
raise the FastAPI subclass with a dict detail, the framework raises the parent with a
|
||||
str detail, and the branch below renders each into ErrorBody.
|
||||
Registered on starlette's HTTPException, not fastapi's. The FastAPI class is a subclass
|
||||
and Starlette matches handlers by walking `type(exc).__mro__`, so a handler keyed on the
|
||||
subclass never fires for a framework-raised 404 or 405. Keying on the parent catches
|
||||
both, and the branch below renders each into ErrorBody.
|
||||
"""
|
||||
assert isinstance(exc, HTTPException) # noqa: S101 - registered only for HTTPException
|
||||
# Widened to object deliberately. Starlette types `detail` as str, but FastAPI passes
|
||||
# through whatever a handler raised — our handlers raise dicts. Narrowing off the
|
||||
# declared type would make mypy call the dict branch unreachable and delete it.
|
||||
# Widened to object deliberately: Starlette types `detail` as str, but FastAPI passes
|
||||
# through whatever a handler raised, and ours raise dicts. Narrowing off the declared
|
||||
# type would let mypy call the dict branch unreachable and delete it.
|
||||
detail: object = exc.detail
|
||||
if isinstance(detail, dict) and "code" in detail and "message" in detail:
|
||||
body = ErrorBody(code=str(detail["code"]), message=str(detail["message"]))
|
||||
@@ -170,10 +165,9 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
|
||||
async def _validation_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Render request-validation failures as ErrorBody too.
|
||||
|
||||
A body that fails validation (a forbidden extra field, a bad type, an out-of-range
|
||||
ttl_days) raises RequestValidationError, which the HTTPException handler above never
|
||||
sees. Without this it returns FastAPI's default `{"detail": [...]}` — a second 422 shape
|
||||
alongside the ErrorBody 422s the handlers raise. This gives every 422 one shape.
|
||||
A forbidden extra field, a bad type or an out-of-range ttl_days raises
|
||||
RequestValidationError, which the handler above never sees. Without this, FastAPI's
|
||||
default `{"detail": [...]}` is a second 422 shape alongside the handlers' ErrorBody.
|
||||
"""
|
||||
assert isinstance(exc, RequestValidationError) # noqa: S101 - registered only for this
|
||||
return JSONResponse(
|
||||
@@ -186,22 +180,20 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
"""App factory: lifespan, routers, exception handler, /metrics."""
|
||||
settings = settings or load_settings()
|
||||
|
||||
# FIRST, before any router is built and before any logger is bound. Without this the
|
||||
# API is the one service of three that never configures structlog: its lines go out
|
||||
# through logging.lastResort as bare text on stderr with no service, no trace_id and
|
||||
# no JSON envelope — a parse failure in the collector, and unattributable in Loki.
|
||||
# FIRST, before any router is built and any logger is bound. Without it the API is the
|
||||
# one service of three that never configures structlog, and its lines go out through
|
||||
# logging.lastResort as bare text on stderr — no service, no trace_id, no JSON envelope.
|
||||
# `settings.log_json` was silently inert here for the same reason.
|
||||
obs.setup("svcforge-api", settings)
|
||||
|
||||
# Refuse the dev escape hatches when SVCFORGE_ENVIRONMENT says this is not a laptop.
|
||||
# Called unconditionally and early: a check that only runs from a branch someone
|
||||
# remembered to write is a check that does not run.
|
||||
# Unconditional and early: a check that runs only from a branch someone remembered to
|
||||
# write is a check that does not run.
|
||||
settings.check_production()
|
||||
|
||||
# The description is the API's documentation. FastAPI renders it as markdown at /docs,
|
||||
# and it is the only place a caller who does not have this repo can learn the two things
|
||||
# that are not obvious from the schema: every write is asynchronous, and the instance
|
||||
# lifecycle is a state machine they have to poll.
|
||||
# The description is the API's documentation, rendered as markdown at /docs. It is the
|
||||
# only place a caller without this repo learns the two things the schema cannot say:
|
||||
# every write is asynchronous, and the lifecycle is a state machine they have to poll.
|
||||
app = FastAPI(
|
||||
title="svcforge",
|
||||
version="0.1.0",
|
||||
@@ -227,7 +219,6 @@ def app() -> FastAPI:
|
||||
return create_app()
|
||||
|
||||
|
||||
# There is deliberately no `if __name__ == "__main__"` here. `services/api/__main__.py` is
|
||||
# the single entrypoint, and the image's ENTRYPOINT uses it. A second one in this module
|
||||
# drifted from it — different log_level, different access_log — so `python -m services.api`
|
||||
# and `python services/api/main.py` started the same app two different ways.
|
||||
# No `if __name__ == "__main__"` here on purpose. `services/api/__main__.py` is the single
|
||||
# entrypoint and the image's ENTRYPOINT uses it. A second one in this module drifted from
|
||||
# it — different log_level, different access_log — so the same app started two ways.
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
"""Wire types.
|
||||
"""Wire types, deliberately not the domain models.
|
||||
|
||||
These are deliberately NOT the domain models. `Instance` carries `team`, `namespace` and
|
||||
`release_name` — placement details a tenant has no business seeing and no business
|
||||
setting. The response model is the allow-list that keeps them off the wire, which is why
|
||||
it is written out by hand instead of derived from `Instance`.
|
||||
`Instance` carries `team`, `namespace` and `release_name` — placement details a tenant has
|
||||
no business seeing or setting. The response model is the allow-list that keeps them off the
|
||||
wire, which is why it is written by hand instead of derived from `Instance`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -19,9 +18,8 @@ class CreateInstanceRequest(BaseModel):
|
||||
"""What a tenant may ask for.
|
||||
|
||||
`service_type` and `size` are plain strings, not enums: the catalog is data loaded at
|
||||
runtime, so baking its keys into a type would mean a redeploy to add a service type,
|
||||
and a 422 (schema) where the spec wants a 404 (unknown resource). They are validated
|
||||
against the catalog in the handler.
|
||||
runtime, so baking its keys into a type would mean a redeploy to add a service type and
|
||||
a 422 where the spec wants a 404. The handler validates them against the catalog.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(
|
||||
|
||||
@@ -3,11 +3,11 @@
|
||||
The distinction between the first two is the difference between a 30-second blip and a
|
||||
fleet-wide outage:
|
||||
|
||||
* `/healthz` (liveness) answers "is this process wedged?" A failure here gets the
|
||||
container KILLED. It must therefore touch NOTHING external. Wire it to the DB and a
|
||||
20-second Postgres failover restarts every pod at once; they come back, find the DB
|
||||
still down, and CrashLoopBackOff with exponential restart delays — so the fleet is now
|
||||
down for minutes after the database recovered.
|
||||
* `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the
|
||||
container, so it must touch nothing external. Wired to the DB, a 20-second Postgres
|
||||
failover restarts every pod at once; they come back, find the DB still down, and
|
||||
CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the
|
||||
database recovered.
|
||||
* `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes
|
||||
it from the Service endpoints. It is allowed to check dependencies, and it recovers by
|
||||
itself the moment the check passes.
|
||||
@@ -26,10 +26,10 @@ from services.api.models import ErrorBody
|
||||
|
||||
router = APIRouter(tags=["ops"])
|
||||
|
||||
# No PROMETHEUS_MULTIPROC_DIR here, deliberately: it exists for prefork servers where each
|
||||
# worker process holds a slice of the counters. One uvicorn process per container means
|
||||
# the default in-process registry is already correct, and multiproc mode would add a
|
||||
# shared temp dir, a cleanup obligation, and a class of stale-file bugs for nothing.
|
||||
# No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each
|
||||
# process holds a slice of the counters. One uvicorn process per container makes the
|
||||
# in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup
|
||||
# obligation, and a class of stale-file bugs for nothing.
|
||||
|
||||
|
||||
@router.get("/healthz", status_code=status.HTTP_200_OK)
|
||||
@@ -45,10 +45,9 @@ async def healthz() -> dict[str, str]:
|
||||
async def readyz(pool: PoolDep) -> dict[str, str]:
|
||||
"""Readiness. Postgres only.
|
||||
|
||||
Postgres-only is the rule, and Redis is the temptation. Redis holds derived state —
|
||||
rate-limit buckets, caches — and everything degrades gracefully without it. Put it in
|
||||
this check and an Upstash hiccup marks every pod unready, Kubernetes empties the
|
||||
Service, and a cache outage becomes a total API outage.
|
||||
Redis is the temptation and stays out: it holds derived state that degrades gracefully,
|
||||
so checking it here would let an Upstash hiccup mark every pod unready, empty the
|
||||
Service, and turn a cache outage into a total API outage.
|
||||
"""
|
||||
try:
|
||||
async with pool.connection() as conn, conn.cursor() as cur:
|
||||
@@ -68,14 +67,13 @@ async def readyz(pool: PoolDep) -> dict[str, str]:
|
||||
async def metrics(request: Request) -> Response:
|
||||
"""The Prometheus scrape endpoint.
|
||||
|
||||
A route rather than `app.mount("/metrics", make_asgi_app())`, for two reasons. A
|
||||
Starlette `Mount` compiles to `^/metrics(?P<path>/.*)$`, which does not match a bare
|
||||
`/metrics` — the exact URL every scrape config uses — and a `Mount` is invisible to
|
||||
OpenAPI, while the deliverable asks for `/metrics` in `openapi.json`.
|
||||
A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount`
|
||||
compiles to `^/metrics(?P<path>/.*)$`, which does not match the bare `/metrics` every
|
||||
scrape config uses, and a Mount is invisible to OpenAPI.
|
||||
|
||||
The encoding is still prometheus_client's: `choose_encoder` reads the Accept header and
|
||||
picks the exposition format (Prometheus text vs OpenMetrics) with its matching content
|
||||
type. Hand-rolling either is how you end up serving text/plain that a scraper rejects.
|
||||
The encoding stays prometheus_client's — `choose_encoder` reads Accept and picks the
|
||||
exposition format with its matching content type. Hand-rolling it serves text/plain a
|
||||
scraper rejects.
|
||||
"""
|
||||
encoder, content_type = choose_encoder(request.headers.get("Accept", ""))
|
||||
return Response(content=encoder(REGISTRY), media_type=content_type)
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
"""The tenant-facing API.
|
||||
|
||||
Two rules run through every handler here:
|
||||
Two rules run through every handler:
|
||||
|
||||
* **AuthZ is the WHERE clause.** No handler ever compares `inst.team` to the caller's
|
||||
team, because the repo never returns another team's row to compare. A wrong-team id is
|
||||
a 404. 403 would confirm the id exists, which is the leak.
|
||||
* **The instance and its task commit together.** A committed instance with no task is an
|
||||
instance that never provisions and that nothing will ever retry.
|
||||
* **AuthZ is the WHERE clause.** No handler compares `inst.team` to the caller's team,
|
||||
because the repo never returns another team's row to compare. A wrong-team id is a 404;
|
||||
403 would confirm the id exists.
|
||||
* **The instance and its task commit together.** A committed instance with no task never
|
||||
provisions and nothing retries it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -30,9 +30,9 @@ from services.api.models import CreateInstanceRequest, ErrorBody, InstanceRespon
|
||||
from svcforge_core.domain.models import CatalogEntry, Instance, TaskKind
|
||||
from svcforge_core.domain.states import IllegalTransition, InstanceState, transition
|
||||
|
||||
# Declared on the router so every error shape lands in openapi.json under ErrorBody.
|
||||
# The exception handler already renders this at runtime; without declaring it, generated
|
||||
# clients see the contract for 2xx only and invent their own guess for the rest.
|
||||
# Declared on the router so every error shape lands in openapi.json under ErrorBody. The
|
||||
# exception handler already renders these at runtime; undeclared, a generated client sees
|
||||
# the contract for 2xx only and guesses the rest.
|
||||
ERROR_RESPONSES: dict[int | str, dict[str, Any]] = {
|
||||
401: {"model": ErrorBody, "description": "Missing or invalid credentials"},
|
||||
404: {"model": ErrorBody, "description": "No such instance, or not this team's"},
|
||||
@@ -46,14 +46,12 @@ router = APIRouter(prefix="/v1/instances", tags=["instances"], responses=ERROR_R
|
||||
def release_name_for(team: str, service_type: str, instance_id: UUID) -> str:
|
||||
"""The helm release name. Deterministic, and `unique` in the schema.
|
||||
|
||||
This is the idempotency anchor. A worker that dies after `helm install` but before it
|
||||
marks the task done will retry, compute the same name, and `helm upgrade --install`
|
||||
onto the same release instead of creating a second one. Derive it from anything that
|
||||
is not already durable — a timestamp, a random suffix, the retry count — and a retry
|
||||
provisions a duplicate.
|
||||
The idempotency anchor: a worker that dies after `helm install` but before marking the
|
||||
task done retries, computes the same name, and upgrades the same release instead of
|
||||
creating a second one. Derive it from anything not already durable — a timestamp, a
|
||||
random suffix, the retry count — and a retry provisions a duplicate.
|
||||
|
||||
Truncated to the uuid's first 8 chars to stay inside the 53-char limit helm imposes
|
||||
on release names (Kubernetes label values, minus room for chart-generated suffixes).
|
||||
Truncated to the uuid's first 8 chars to stay inside helm's 53-char release-name limit.
|
||||
"""
|
||||
return f"{team}-{service_type}-{str(instance_id)[:8]}"
|
||||
|
||||
@@ -66,9 +64,8 @@ def namespace_for(team: str) -> str:
|
||||
def _resolve(catalog: dict[str, CatalogEntry], service_type: str, size: str) -> CatalogEntry:
|
||||
"""Look up service_type + size, or raise the right 4xx.
|
||||
|
||||
The two failures are different HTTP problems and the spec asks for different codes:
|
||||
an unknown service_type is a resource that does not exist (404); an unknown size for a
|
||||
real service_type is a body the server understood and cannot process (422).
|
||||
Two different HTTP problems: an unknown service_type is a resource that does not exist
|
||||
(404), an unknown size for a real one is a body understood and unprocessable (422).
|
||||
"""
|
||||
entry = catalog.get(service_type)
|
||||
if entry is None:
|
||||
@@ -107,9 +104,9 @@ async def create_instance(
|
||||
) -> Instance:
|
||||
"""Accept a provisioning request. 202, never 201.
|
||||
|
||||
Nothing is provisioned when this returns. The row exists and a task is queued; a
|
||||
worker will do the work seconds or minutes from now. 201 Created would be a lie about
|
||||
a resource that does not exist yet, and clients would stop polling.
|
||||
Nothing is provisioned when this returns: the row exists and a task is queued, and a
|
||||
worker does the work seconds or minutes later. 201 Created would be a lie about a
|
||||
resource that does not exist yet, and clients would stop polling.
|
||||
"""
|
||||
entry = _resolve(catalog, body.service_type, body.size)
|
||||
|
||||
@@ -123,9 +120,9 @@ async def create_instance(
|
||||
state=InstanceState.REQUESTED,
|
||||
namespace=namespace_for(team),
|
||||
release_name=release_name_for(team, body.service_type, instance_id),
|
||||
# Pinned from the catalog AT CREATION TIME, not read from the catalog later.
|
||||
# This column records what is actually deployed; bumping catalog.yaml must show up
|
||||
# as drift the reconciler can see, not silently rewrite history.
|
||||
# Pinned at creation time, not read from the catalog later. The column records what
|
||||
# is deployed, so bumping catalog.yaml shows up as drift the reconciler can see
|
||||
# rather than silently rewriting history.
|
||||
chart_version=entry.chart_version,
|
||||
expires_at=now + timedelta(days=body.ttl_days) if body.ttl_days is not None else None,
|
||||
created_at=now,
|
||||
@@ -181,12 +178,11 @@ async def delete_instance(
|
||||
) -> Instance:
|
||||
"""state -> deleting, enqueue deprovision. 202: the helm uninstall has not happened yet.
|
||||
|
||||
Ordering note. `InstanceRepo.update_state` owns its own connection, so the CAS and the
|
||||
enqueue cannot share one transaction without reaching around the repo. Given two
|
||||
statements, the order is chosen for its failure mode: CAS first, enqueue second. A
|
||||
crash in between leaves an instance in `deleting` with no task, which the reconciler's
|
||||
sweep re-enqueues. The other order leaves a deprovision task pointing at a `ready`
|
||||
instance, and a worker would tear down a live service nobody asked to delete.
|
||||
`InstanceRepo.update_state` owns its own connection, so the CAS and the enqueue cannot
|
||||
share a transaction without reaching around the repo. Given two statements, the order is
|
||||
chosen for its failure mode: a crash between CAS and enqueue leaves an instance in
|
||||
`deleting` with no task, which the reconciler's sweep re-enqueues. The reverse would
|
||||
leave a deprovision task on a `ready` instance and tear down a live service.
|
||||
"""
|
||||
inst = await instances.get(instance_id, team)
|
||||
if inst is None:
|
||||
|
||||
Reference in New Issue
Block a user