docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
+22
-27
@@ -1,8 +1,8 @@
|
||||
"""Dependency injection: how a handler gets a pool, a repo, a catalog, and a team.
|
||||
|
||||
Everything expensive — the pool, the JWKS client, the parsed catalog — is built once in
|
||||
`lifespan` and parked on `app.state`. These functions only hand it out. A `Depends` that
|
||||
does I/O per request is a `Depends` that does that I/O on every request forever.
|
||||
`lifespan` and parked on `app.state`; these functions only hand it out. A `Depends` that
|
||||
does I/O per request does that I/O on every request forever.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -22,10 +22,9 @@ from svcforge_core.repo.instances import InstanceRepo
|
||||
from svcforge_core.repo.tasks import TaskRepo
|
||||
from svcforge_core.settings import Settings
|
||||
|
||||
# The algorithm allow-list is the whole point of naming algorithms explicitly.
|
||||
# `jwt.decode(..., algorithms=...)` without it accepts whatever the *token* claims in its
|
||||
# own header — including `none`, and including HS256 verified with the RSA public key as
|
||||
# an HMAC secret. Both are forgery. The list is not configuration.
|
||||
# The algorithm allow-list is not configuration. Without it, `jwt.decode` accepts whatever
|
||||
# the *token* claims in its own header — including `none`, and including HS256 verified
|
||||
# with the RSA public key as an HMAC secret. Both are forgery.
|
||||
ALLOWED_ALGORITHMS = ["RS256"]
|
||||
|
||||
# What `auth_disabled` returns. Settings.check_production() refuses that flag in prod.
|
||||
@@ -42,9 +41,9 @@ _bearer = HTTPBearer(auto_error=False)
|
||||
def _unauthorized() -> HTTPException:
|
||||
"""One shape for every auth failure.
|
||||
|
||||
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: all the
|
||||
same 401 with the same body. Telling a caller *which* one turns the endpoint into an
|
||||
oracle they can tune a forgery against.
|
||||
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: the same 401
|
||||
with the same body. Naming which one turns the endpoint into an oracle a forger can tune
|
||||
against.
|
||||
"""
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
@@ -68,8 +67,8 @@ async def get_pool(request: Request) -> DictPool:
|
||||
def get_catalog(request: Request) -> dict[str, CatalogEntry]:
|
||||
"""The catalog, parsed once at startup.
|
||||
|
||||
Read from disk per request and a mid-flight edit to catalog.yaml changes the answer
|
||||
between two requests of the same deploy. Load it at startup; a change is a restart.
|
||||
Read per request, a mid-flight edit to catalog.yaml would change the answer between two
|
||||
requests of the same deploy. A catalog change is a restart.
|
||||
"""
|
||||
catalog: dict[str, CatalogEntry] = request.app.state.catalog
|
||||
return catalog
|
||||
@@ -102,16 +101,14 @@ async def get_current_team(
|
||||
|
||||
jwks_client: PyJWKClient | None = getattr(request.app.state, "jwks_client", None)
|
||||
if jwks_client is None:
|
||||
# Auth is on but there is no key source. Fail closed. Answering 500 here would be
|
||||
# honest about the cause and would also let a misconfigured deploy be told apart
|
||||
# from a bad token; 401 is the same answer a forger gets.
|
||||
# Auth is on but there is no key source. Fail closed. A 500 would be honest about
|
||||
# the cause and would also let a forger tell a misconfigured deploy from a bad token.
|
||||
raise _unauthorized()
|
||||
|
||||
try:
|
||||
# PyJWKClient keeps its own TTL cache, so this is a dict lookup on the hot path.
|
||||
# It is only blocking on a cache MISS (key rotation) — hence to_thread, which
|
||||
# costs a thread hop we take a handful of times a day rather than an event loop
|
||||
# stalled on someone else's HTTP call once per rotation.
|
||||
# PyJWKClient keeps a TTL cache, so this is a dict lookup on the hot path and only
|
||||
# blocks on a miss (key rotation) — hence to_thread, a thread hop a few times a day
|
||||
# rather than an event loop stalled on someone else's HTTP call.
|
||||
signing_key = await _signing_key(jwks_client, creds.credentials)
|
||||
claims: dict[str, Any] = jwt.decode(
|
||||
creds.credentials,
|
||||
@@ -139,10 +136,9 @@ async def get_current_team(
|
||||
async def _signing_key(client: PyJWKClient, token: str) -> jwt.PyJWK:
|
||||
"""Fetch the signing key off the event loop.
|
||||
|
||||
PyJWKClient.get_signing_key_from_jwt() does a synchronous urlopen on a cache miss.
|
||||
Called directly from `async def`, that blocks the loop — every other in-flight request
|
||||
on this worker stops until the identity provider answers, and if it hangs, so does the
|
||||
pod, and /readyz keeps saying it is fine.
|
||||
`get_signing_key_from_jwt()` does a synchronous urlopen on a cache miss. Called directly
|
||||
from `async def` it blocks the loop: every other in-flight request stops until the IdP
|
||||
answers, and if the IdP hangs so does the pod, with /readyz still saying it is fine.
|
||||
"""
|
||||
return await asyncio.to_thread(client.get_signing_key_from_jwt, token)
|
||||
|
||||
@@ -159,11 +155,10 @@ async def rate_limit(
|
||||
) -> None:
|
||||
"""Per-team rate limiting. One Redis command per check, and it fails OPEN.
|
||||
|
||||
Failing open is the entire policy. Redis holds derived state; losing it must degrade
|
||||
the platform, never stop it. A limiter that fails closed converts a cache outage into
|
||||
a total outage, which is a strictly worse incident than the burst it was protecting
|
||||
against — so `RateLimiter.check` swallows its own errors and returns `allowed=True`.
|
||||
The 429 below therefore only ever comes from a real, counted overage.
|
||||
Redis holds derived state, so losing it must degrade the platform rather than stop it: a
|
||||
limiter that fails closed turns a cache outage into a total outage, a worse incident
|
||||
than the burst it was guarding against. `RateLimiter.check` swallows its own errors and
|
||||
returns `allowed=True`, so the 429 below only comes from a real, counted overage.
|
||||
"""
|
||||
limiter = get_rate_limiter(request)
|
||||
if limiter is None:
|
||||
|
||||
+43
-52
@@ -1,7 +1,7 @@
|
||||
"""The app factory and its lifespan.
|
||||
|
||||
`create_app(settings)` is a factory, not a module-level `app = FastAPI()`, for one reason:
|
||||
a test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
|
||||
`create_app(settings)` is a factory rather than a module-level `app = FastAPI()` because a
|
||||
test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
|
||||
environment at import time — before any fixture can say otherwise.
|
||||
"""
|
||||
|
||||
@@ -32,23 +32,21 @@ log = obs.get_logger("svcforge.api")
|
||||
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
"""Open the pool, yield, close the pool.
|
||||
|
||||
A lifespan context, not the deprecated startup/shutdown event decorators: those cannot
|
||||
express "this resource lives for exactly as long as the app", and give you no place to
|
||||
put the teardown next to the setup. Closing the pool matters — an unclosed pool means
|
||||
connections linger server-side after SIGTERM, and on a pooled Postgres with a small
|
||||
connection budget a few rolling deploys exhaust it.
|
||||
A lifespan context, not the deprecated startup/shutdown decorators: those cannot express
|
||||
"this resource lives exactly as long as the app" and leave no place to put teardown next
|
||||
to setup. Closing matters — an unclosed pool leaves connections open server-side after
|
||||
SIGTERM, and on a pooled Postgres with a small budget a few rolling deploys exhaust it.
|
||||
|
||||
(The old decorator's name is spelled nowhere in this package on purpose: CI greps for
|
||||
the literal string, and a comment quoting it fails the gate just as loudly as a call.)
|
||||
(The old decorator's name is spelled nowhere here on purpose: CI greps for the literal
|
||||
string, so a comment quoting it fails the gate as loudly as a call would.)
|
||||
"""
|
||||
settings: Settings = app.state.settings
|
||||
|
||||
app.state.catalog = load_catalog(settings.catalog_path)
|
||||
|
||||
# Redis is optional by construction. `make_redis` returns None when no DSN is set, and
|
||||
# every consumer treats None as "skip" — so a deployment without Redis loses rate
|
||||
# limiting and keeps everything else. Built here rather than per request because a
|
||||
# connection pool per request is a connection pool per request.
|
||||
# Redis is optional by construction: `make_redis` returns None when no DSN is set and
|
||||
# every consumer treats None as "skip", so a deployment without Redis loses rate
|
||||
# limiting and keeps everything else. Built once here, not per request.
|
||||
redis = make_redis(settings)
|
||||
app.state.redis = redis
|
||||
app.state.rate_limiter = (
|
||||
@@ -61,16 +59,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
await pool.open(wait=True)
|
||||
app.state.pool = pool
|
||||
|
||||
# The pool is open from here on, so everything below is inside the try: an exception
|
||||
# in JWKS setup must still close it, or a crash-looping pod leaks a connection per
|
||||
# restart until the database refuses new ones.
|
||||
# The pool is open from here, so everything below is inside the try: an exception in
|
||||
# JWKS setup must still close it, or a crash-looping pod leaks a connection per restart
|
||||
# until the database refuses new ones.
|
||||
try:
|
||||
if settings.jwks_url and not settings.auth_disabled:
|
||||
client = PyJWKClient(settings.jwks_url, cache_keys=True, lifespan=300)
|
||||
app.state.jwks_client = client
|
||||
# Warm the cache off the loop so the first authenticated request does not pay
|
||||
# a blocking urlopen. Best-effort: a slow identity provider must not stop the
|
||||
# pod from starting — a cache miss later just costs one to_thread hop.
|
||||
# Warm the cache off the loop so the first authenticated request does not pay a
|
||||
# blocking urlopen. Best-effort: a slow IdP must not stop the pod from starting,
|
||||
# and a miss later costs one to_thread hop.
|
||||
try:
|
||||
await asyncio.to_thread(client.get_signing_keys)
|
||||
except Exception: # deliberate catch-all: startup must not hinge on the IdP being up
|
||||
@@ -87,9 +85,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
|
||||
# --------------------------------------------------------------------------- API docs
|
||||
|
||||
# Everything a caller needs that the generated schema cannot express on its own. Kept next
|
||||
# to create_app rather than in a README because /docs is what someone integrating actually
|
||||
# reads, and a README in this repo is not something they have.
|
||||
# What the generated schema cannot express. Kept next to create_app because /docs is what
|
||||
# someone integrating reads, and they do not have this repo. USER_GUIDE.md is the longer form.
|
||||
API_DESCRIPTION = """
|
||||
Provision managed service instances into Kubernetes. The catalog offers Elasticsearch,
|
||||
Redis and Postgres, plus two deliberately tiny entries — `podinfo` and `nginx` — for
|
||||
@@ -144,21 +141,19 @@ OPENAPI_TAGS = [
|
||||
async def _http_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Render HTTPException bodies as ErrorBody, so every error has one shape.
|
||||
|
||||
Handlers raise `detail={"code": ..., "message": ...}`; FastAPI's default would nest
|
||||
that under `{"detail": {...}}`. Plain-string details (raised by FastAPI itself, e.g.
|
||||
a 405) are wrapped so clients never have to branch on the body's type.
|
||||
Handlers raise `detail={"code": ..., "message": ...}`, which FastAPI's default would
|
||||
nest under `{"detail": {...}}`. Plain-string details (a framework 405, say) are wrapped
|
||||
so clients never branch on the body's type.
|
||||
|
||||
Registered on starlette's HTTPException, not fastapi's. fastapi.HTTPException is a
|
||||
subclass, and Starlette matches handlers by walking type(exc).__mro__, so a handler
|
||||
keyed on the subclass never fires for a framework-raised 404 or 405 — which are
|
||||
starlette.HTTPException instances. Keying on the parent catches both: app handlers
|
||||
raise the FastAPI subclass with a dict detail, the framework raises the parent with a
|
||||
str detail, and the branch below renders each into ErrorBody.
|
||||
Registered on starlette's HTTPException, not fastapi's. The FastAPI class is a subclass
|
||||
and Starlette matches handlers by walking `type(exc).__mro__`, so a handler keyed on the
|
||||
subclass never fires for a framework-raised 404 or 405. Keying on the parent catches
|
||||
both, and the branch below renders each into ErrorBody.
|
||||
"""
|
||||
assert isinstance(exc, HTTPException) # noqa: S101 - registered only for HTTPException
|
||||
# Widened to object deliberately. Starlette types `detail` as str, but FastAPI passes
|
||||
# through whatever a handler raised — our handlers raise dicts. Narrowing off the
|
||||
# declared type would make mypy call the dict branch unreachable and delete it.
|
||||
# Widened to object deliberately: Starlette types `detail` as str, but FastAPI passes
|
||||
# through whatever a handler raised, and ours raise dicts. Narrowing off the declared
|
||||
# type would let mypy call the dict branch unreachable and delete it.
|
||||
detail: object = exc.detail
|
||||
if isinstance(detail, dict) and "code" in detail and "message" in detail:
|
||||
body = ErrorBody(code=str(detail["code"]), message=str(detail["message"]))
|
||||
@@ -170,10 +165,9 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
|
||||
async def _validation_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Render request-validation failures as ErrorBody too.
|
||||
|
||||
A body that fails validation (a forbidden extra field, a bad type, an out-of-range
|
||||
ttl_days) raises RequestValidationError, which the HTTPException handler above never
|
||||
sees. Without this it returns FastAPI's default `{"detail": [...]}` — a second 422 shape
|
||||
alongside the ErrorBody 422s the handlers raise. This gives every 422 one shape.
|
||||
A forbidden extra field, a bad type or an out-of-range ttl_days raises
|
||||
RequestValidationError, which the handler above never sees. Without this, FastAPI's
|
||||
default `{"detail": [...]}` is a second 422 shape alongside the handlers' ErrorBody.
|
||||
"""
|
||||
assert isinstance(exc, RequestValidationError) # noqa: S101 - registered only for this
|
||||
return JSONResponse(
|
||||
@@ -186,22 +180,20 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
"""App factory: lifespan, routers, exception handler, /metrics."""
|
||||
settings = settings or load_settings()
|
||||
|
||||
# FIRST, before any router is built and before any logger is bound. Without this the
|
||||
# API is the one service of three that never configures structlog: its lines go out
|
||||
# through logging.lastResort as bare text on stderr with no service, no trace_id and
|
||||
# no JSON envelope — a parse failure in the collector, and unattributable in Loki.
|
||||
# FIRST, before any router is built and any logger is bound. Without it the API is the
|
||||
# one service of three that never configures structlog, and its lines go out through
|
||||
# logging.lastResort as bare text on stderr — no service, no trace_id, no JSON envelope.
|
||||
# `settings.log_json` was silently inert here for the same reason.
|
||||
obs.setup("svcforge-api", settings)
|
||||
|
||||
# Refuse the dev escape hatches when SVCFORGE_ENVIRONMENT says this is not a laptop.
|
||||
# Called unconditionally and early: a check that only runs from a branch someone
|
||||
# remembered to write is a check that does not run.
|
||||
# Unconditional and early: a check that runs only from a branch someone remembered to
|
||||
# write is a check that does not run.
|
||||
settings.check_production()
|
||||
|
||||
# The description is the API's documentation. FastAPI renders it as markdown at /docs,
|
||||
# and it is the only place a caller who does not have this repo can learn the two things
|
||||
# that are not obvious from the schema: every write is asynchronous, and the instance
|
||||
# lifecycle is a state machine they have to poll.
|
||||
# The description is the API's documentation, rendered as markdown at /docs. It is the
|
||||
# only place a caller without this repo learns the two things the schema cannot say:
|
||||
# every write is asynchronous, and the lifecycle is a state machine they have to poll.
|
||||
app = FastAPI(
|
||||
title="svcforge",
|
||||
version="0.1.0",
|
||||
@@ -227,7 +219,6 @@ def app() -> FastAPI:
|
||||
return create_app()
|
||||
|
||||
|
||||
# There is deliberately no `if __name__ == "__main__"` here. `services/api/__main__.py` is
|
||||
# the single entrypoint, and the image's ENTRYPOINT uses it. A second one in this module
|
||||
# drifted from it — different log_level, different access_log — so `python -m services.api`
|
||||
# and `python services/api/main.py` started the same app two different ways.
|
||||
# No `if __name__ == "__main__"` here on purpose. `services/api/__main__.py` is the single
|
||||
# entrypoint and the image's ENTRYPOINT uses it. A second one in this module drifted from
|
||||
# it — different log_level, different access_log — so the same app started two ways.
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
"""Wire types.
|
||||
"""Wire types, deliberately not the domain models.
|
||||
|
||||
These are deliberately NOT the domain models. `Instance` carries `team`, `namespace` and
|
||||
`release_name` — placement details a tenant has no business seeing and no business
|
||||
setting. The response model is the allow-list that keeps them off the wire, which is why
|
||||
it is written out by hand instead of derived from `Instance`.
|
||||
`Instance` carries `team`, `namespace` and `release_name` — placement details a tenant has
|
||||
no business seeing or setting. The response model is the allow-list that keeps them off the
|
||||
wire, which is why it is written by hand instead of derived from `Instance`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -19,9 +18,8 @@ class CreateInstanceRequest(BaseModel):
|
||||
"""What a tenant may ask for.
|
||||
|
||||
`service_type` and `size` are plain strings, not enums: the catalog is data loaded at
|
||||
runtime, so baking its keys into a type would mean a redeploy to add a service type,
|
||||
and a 422 (schema) where the spec wants a 404 (unknown resource). They are validated
|
||||
against the catalog in the handler.
|
||||
runtime, so baking its keys into a type would mean a redeploy to add a service type and
|
||||
a 422 where the spec wants a 404. The handler validates them against the catalog.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(
|
||||
|
||||
@@ -3,11 +3,11 @@
|
||||
The distinction between the first two is the difference between a 30-second blip and a
|
||||
fleet-wide outage:
|
||||
|
||||
* `/healthz` (liveness) answers "is this process wedged?" A failure here gets the
|
||||
container KILLED. It must therefore touch NOTHING external. Wire it to the DB and a
|
||||
20-second Postgres failover restarts every pod at once; they come back, find the DB
|
||||
still down, and CrashLoopBackOff with exponential restart delays — so the fleet is now
|
||||
down for minutes after the database recovered.
|
||||
* `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the
|
||||
container, so it must touch nothing external. Wired to the DB, a 20-second Postgres
|
||||
failover restarts every pod at once; they come back, find the DB still down, and
|
||||
CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the
|
||||
database recovered.
|
||||
* `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes
|
||||
it from the Service endpoints. It is allowed to check dependencies, and it recovers by
|
||||
itself the moment the check passes.
|
||||
@@ -26,10 +26,10 @@ from services.api.models import ErrorBody
|
||||
|
||||
router = APIRouter(tags=["ops"])
|
||||
|
||||
# No PROMETHEUS_MULTIPROC_DIR here, deliberately: it exists for prefork servers where each
|
||||
# worker process holds a slice of the counters. One uvicorn process per container means
|
||||
# the default in-process registry is already correct, and multiproc mode would add a
|
||||
# shared temp dir, a cleanup obligation, and a class of stale-file bugs for nothing.
|
||||
# No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each
|
||||
# process holds a slice of the counters. One uvicorn process per container makes the
|
||||
# in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup
|
||||
# obligation, and a class of stale-file bugs for nothing.
|
||||
|
||||
|
||||
@router.get("/healthz", status_code=status.HTTP_200_OK)
|
||||
@@ -45,10 +45,9 @@ async def healthz() -> dict[str, str]:
|
||||
async def readyz(pool: PoolDep) -> dict[str, str]:
|
||||
"""Readiness. Postgres only.
|
||||
|
||||
Postgres-only is the rule, and Redis is the temptation. Redis holds derived state —
|
||||
rate-limit buckets, caches — and everything degrades gracefully without it. Put it in
|
||||
this check and an Upstash hiccup marks every pod unready, Kubernetes empties the
|
||||
Service, and a cache outage becomes a total API outage.
|
||||
Redis is the temptation and stays out: it holds derived state that degrades gracefully,
|
||||
so checking it here would let an Upstash hiccup mark every pod unready, empty the
|
||||
Service, and turn a cache outage into a total API outage.
|
||||
"""
|
||||
try:
|
||||
async with pool.connection() as conn, conn.cursor() as cur:
|
||||
@@ -68,14 +67,13 @@ async def readyz(pool: PoolDep) -> dict[str, str]:
|
||||
async def metrics(request: Request) -> Response:
|
||||
"""The Prometheus scrape endpoint.
|
||||
|
||||
A route rather than `app.mount("/metrics", make_asgi_app())`, for two reasons. A
|
||||
Starlette `Mount` compiles to `^/metrics(?P<path>/.*)$`, which does not match a bare
|
||||
`/metrics` — the exact URL every scrape config uses — and a `Mount` is invisible to
|
||||
OpenAPI, while the deliverable asks for `/metrics` in `openapi.json`.
|
||||
A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount`
|
||||
compiles to `^/metrics(?P<path>/.*)$`, which does not match the bare `/metrics` every
|
||||
scrape config uses, and a Mount is invisible to OpenAPI.
|
||||
|
||||
The encoding is still prometheus_client's: `choose_encoder` reads the Accept header and
|
||||
picks the exposition format (Prometheus text vs OpenMetrics) with its matching content
|
||||
type. Hand-rolling either is how you end up serving text/plain that a scraper rejects.
|
||||
The encoding stays prometheus_client's — `choose_encoder` reads Accept and picks the
|
||||
exposition format with its matching content type. Hand-rolling it serves text/plain a
|
||||
scraper rejects.
|
||||
"""
|
||||
encoder, content_type = choose_encoder(request.headers.get("Accept", ""))
|
||||
return Response(content=encoder(REGISTRY), media_type=content_type)
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
"""The tenant-facing API.
|
||||
|
||||
Two rules run through every handler here:
|
||||
Two rules run through every handler:
|
||||
|
||||
* **AuthZ is the WHERE clause.** No handler ever compares `inst.team` to the caller's
|
||||
team, because the repo never returns another team's row to compare. A wrong-team id is
|
||||
a 404. 403 would confirm the id exists, which is the leak.
|
||||
* **The instance and its task commit together.** A committed instance with no task is an
|
||||
instance that never provisions and that nothing will ever retry.
|
||||
* **AuthZ is the WHERE clause.** No handler compares `inst.team` to the caller's team,
|
||||
because the repo never returns another team's row to compare. A wrong-team id is a 404;
|
||||
403 would confirm the id exists.
|
||||
* **The instance and its task commit together.** A committed instance with no task never
|
||||
provisions and nothing retries it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -30,9 +30,9 @@ from services.api.models import CreateInstanceRequest, ErrorBody, InstanceRespon
|
||||
from svcforge_core.domain.models import CatalogEntry, Instance, TaskKind
|
||||
from svcforge_core.domain.states import IllegalTransition, InstanceState, transition
|
||||
|
||||
# Declared on the router so every error shape lands in openapi.json under ErrorBody.
|
||||
# The exception handler already renders this at runtime; without declaring it, generated
|
||||
# clients see the contract for 2xx only and invent their own guess for the rest.
|
||||
# Declared on the router so every error shape lands in openapi.json under ErrorBody. The
|
||||
# exception handler already renders these at runtime; undeclared, a generated client sees
|
||||
# the contract for 2xx only and guesses the rest.
|
||||
ERROR_RESPONSES: dict[int | str, dict[str, Any]] = {
|
||||
401: {"model": ErrorBody, "description": "Missing or invalid credentials"},
|
||||
404: {"model": ErrorBody, "description": "No such instance, or not this team's"},
|
||||
@@ -46,14 +46,12 @@ router = APIRouter(prefix="/v1/instances", tags=["instances"], responses=ERROR_R
|
||||
def release_name_for(team: str, service_type: str, instance_id: UUID) -> str:
|
||||
"""The helm release name. Deterministic, and `unique` in the schema.
|
||||
|
||||
This is the idempotency anchor. A worker that dies after `helm install` but before it
|
||||
marks the task done will retry, compute the same name, and `helm upgrade --install`
|
||||
onto the same release instead of creating a second one. Derive it from anything that
|
||||
is not already durable — a timestamp, a random suffix, the retry count — and a retry
|
||||
provisions a duplicate.
|
||||
The idempotency anchor: a worker that dies after `helm install` but before marking the
|
||||
task done retries, computes the same name, and upgrades the same release instead of
|
||||
creating a second one. Derive it from anything not already durable — a timestamp, a
|
||||
random suffix, the retry count — and a retry provisions a duplicate.
|
||||
|
||||
Truncated to the uuid's first 8 chars to stay inside the 53-char limit helm imposes
|
||||
on release names (Kubernetes label values, minus room for chart-generated suffixes).
|
||||
Truncated to the uuid's first 8 chars to stay inside helm's 53-char release-name limit.
|
||||
"""
|
||||
return f"{team}-{service_type}-{str(instance_id)[:8]}"
|
||||
|
||||
@@ -66,9 +64,8 @@ def namespace_for(team: str) -> str:
|
||||
def _resolve(catalog: dict[str, CatalogEntry], service_type: str, size: str) -> CatalogEntry:
|
||||
"""Look up service_type + size, or raise the right 4xx.
|
||||
|
||||
The two failures are different HTTP problems and the spec asks for different codes:
|
||||
an unknown service_type is a resource that does not exist (404); an unknown size for a
|
||||
real service_type is a body the server understood and cannot process (422).
|
||||
Two different HTTP problems: an unknown service_type is a resource that does not exist
|
||||
(404), an unknown size for a real one is a body understood and unprocessable (422).
|
||||
"""
|
||||
entry = catalog.get(service_type)
|
||||
if entry is None:
|
||||
@@ -107,9 +104,9 @@ async def create_instance(
|
||||
) -> Instance:
|
||||
"""Accept a provisioning request. 202, never 201.
|
||||
|
||||
Nothing is provisioned when this returns. The row exists and a task is queued; a
|
||||
worker will do the work seconds or minutes from now. 201 Created would be a lie about
|
||||
a resource that does not exist yet, and clients would stop polling.
|
||||
Nothing is provisioned when this returns: the row exists and a task is queued, and a
|
||||
worker does the work seconds or minutes later. 201 Created would be a lie about a
|
||||
resource that does not exist yet, and clients would stop polling.
|
||||
"""
|
||||
entry = _resolve(catalog, body.service_type, body.size)
|
||||
|
||||
@@ -123,9 +120,9 @@ async def create_instance(
|
||||
state=InstanceState.REQUESTED,
|
||||
namespace=namespace_for(team),
|
||||
release_name=release_name_for(team, body.service_type, instance_id),
|
||||
# Pinned from the catalog AT CREATION TIME, not read from the catalog later.
|
||||
# This column records what is actually deployed; bumping catalog.yaml must show up
|
||||
# as drift the reconciler can see, not silently rewrite history.
|
||||
# Pinned at creation time, not read from the catalog later. The column records what
|
||||
# is deployed, so bumping catalog.yaml shows up as drift the reconciler can see
|
||||
# rather than silently rewriting history.
|
||||
chart_version=entry.chart_version,
|
||||
expires_at=now + timedelta(days=body.ttl_days) if body.ttl_days is not None else None,
|
||||
created_at=now,
|
||||
@@ -181,12 +178,11 @@ async def delete_instance(
|
||||
) -> Instance:
|
||||
"""state -> deleting, enqueue deprovision. 202: the helm uninstall has not happened yet.
|
||||
|
||||
Ordering note. `InstanceRepo.update_state` owns its own connection, so the CAS and the
|
||||
enqueue cannot share one transaction without reaching around the repo. Given two
|
||||
statements, the order is chosen for its failure mode: CAS first, enqueue second. A
|
||||
crash in between leaves an instance in `deleting` with no task, which the reconciler's
|
||||
sweep re-enqueues. The other order leaves a deprovision task pointing at a `ready`
|
||||
instance, and a worker would tear down a live service nobody asked to delete.
|
||||
`InstanceRepo.update_state` owns its own connection, so the CAS and the enqueue cannot
|
||||
share a transaction without reaching around the repo. Given two statements, the order is
|
||||
chosen for its failure mode: a crash between CAS and enqueue leaves an instance in
|
||||
`deleting` with no task, which the reconciler's sweep re-enqueues. The reverse would
|
||||
leave a deprovision task on a `ready` instance and tear down a live service.
|
||||
"""
|
||||
inst = await instances.get(instance_id, team)
|
||||
if inst is None:
|
||||
|
||||
+22
-6
@@ -1,9 +1,9 @@
|
||||
"""svcforge — the control plane client.
|
||||
|
||||
This talks to the API over HTTP and never touches the database. That restraint is the
|
||||
whole design: if the CLI could write to Postgres, every invariant the API enforces
|
||||
(the state machine, the one-transaction create, AuthZ in the WHERE clause) would have a
|
||||
back door, and the first 3am incident would go through it.
|
||||
Talks to the API over HTTP and never touches the database. If the CLI could write to
|
||||
Postgres, every invariant the API enforces — the state machine, the one-transaction create,
|
||||
AuthZ in the WHERE clause — would have a back door, and the first 3am incident would go
|
||||
through it. `ClientSettings` below is what keeps that true in practice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -16,9 +16,9 @@ from typing import Annotated, Any
|
||||
|
||||
import httpx
|
||||
import typer
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
from svcforge_core.domain.states import InstanceState
|
||||
from svcforge_core.settings import load_settings
|
||||
|
||||
app = typer.Typer(help="svcforge control plane client", no_args_is_help=True)
|
||||
|
||||
@@ -41,8 +41,24 @@ class Size(StrEnum):
|
||||
MEDIUM = "medium"
|
||||
|
||||
|
||||
class ClientSettings(BaseSettings):
|
||||
"""The two values the CLI needs, and nothing else.
|
||||
|
||||
Its own model rather than `svcforge_core.settings.Settings`, which requires
|
||||
`SVCFORGE_PG_DSN`: loading that here would refuse to run the CLI without a database URL
|
||||
it then never opens, on a laptop that has no reason to hold one.
|
||||
"""
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_prefix="SVCFORGE_", env_file=".env", env_file_encoding="utf-8", extra="ignore", frozen=True
|
||||
)
|
||||
|
||||
api_url: str = "http://localhost:8000"
|
||||
api_token: str | None = None
|
||||
|
||||
|
||||
def _client() -> httpx.Client:
|
||||
settings = load_settings()
|
||||
settings = ClientSettings()
|
||||
headers = {"authorization": f"Bearer {settings.api_token}"} if settings.api_token else {}
|
||||
return httpx.Client(base_url=settings.api_url, headers=headers, timeout=10.0)
|
||||
|
||||
|
||||
+81
-94
@@ -1,29 +1,28 @@
|
||||
"""The control loop.
|
||||
|
||||
Every other service in svcforge is edge-triggered: a tenant POSTs, a row appears, a worker
|
||||
claims it. Edge-triggered systems are correct exactly as long as nothing is ever missed —
|
||||
and things are missed. A worker is SIGKILLed holding a lease. An operator runs
|
||||
`helm uninstall` by hand. A pod dies between the CAS and the enqueue. Nobody sends an event
|
||||
for any of that, because the thing that would have sent it is the thing that died.
|
||||
Every other service here is edge-triggered: a tenant POSTs, a row appears, a worker claims
|
||||
it. That is correct only as long as nothing is missed, and things are missed — a worker
|
||||
SIGKILLed holding a lease, an operator running `helm uninstall` by hand, a pod dying
|
||||
between the CAS and the enqueue. Nothing sends an event for any of it, because the thing
|
||||
that would have sent it is the thing that died.
|
||||
|
||||
So: level-triggered. Every 60 seconds, compare the world to the database and enqueue what
|
||||
is missing. The four checks below do not know or care what went wrong, or whether anything
|
||||
did; they are the same code on the happy path and after an outage. That property is the
|
||||
entire reason this service exists, and it is why each check is written as a *query for
|
||||
work*, never as a reaction to an event.
|
||||
is missing. The four checks below do not know what went wrong, or whether anything did;
|
||||
they are the same code on the happy path and after an outage. That is why each is written
|
||||
as a query for work rather than a reaction to an event.
|
||||
|
||||
Three rules hold the design together:
|
||||
|
||||
* **Singleton.** `replicas: 1`, `strategy: Recreate` in the chart. Two reconcilers
|
||||
double-enqueue drift and race on TTL. There is no leader election here on purpose: the
|
||||
correct lease for that lives in Postgres next to the data, not in a Redis lock, and
|
||||
until there is a second replica to elect between, an election is a subsystem that can
|
||||
only fail. One pod, and the `SvcforgeReconcilerStale` alert is what notices it is gone.
|
||||
* **Each check is independent.** One failing check must not skip the other three. A helm
|
||||
binary that cannot reach the API server must not stop TTLs from expiring.
|
||||
* **Enqueue, never act.** The reconciler diagnoses; workers treat. It writes task rows and
|
||||
instance states, and never calls `helm install`. The one exception is reading — the drift
|
||||
check lists the live releases, because seeing reality is the job.
|
||||
* **Singleton.** `replicas: 1`, `strategy: Recreate`. Two reconcilers double-enqueue drift
|
||||
and race on TTL. No leader election on purpose — the right lease for that lives in
|
||||
Postgres next to the data, and until there is a second replica to elect between, an
|
||||
election is a subsystem that can only fail. The `SvcforgeReconcilerStale` alert notices
|
||||
when the one pod is gone.
|
||||
* **Each check is independent.** A helm binary that cannot reach the API server must not
|
||||
stop TTLs from expiring.
|
||||
* **Enqueue, never act.** The reconciler diagnoses and workers treat: it writes task rows
|
||||
and instance states and never calls `helm install`. Reading is the exception, since
|
||||
seeing reality is the job.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -63,9 +62,9 @@ log = get_logger("svcforge.reconciler")
|
||||
class ReconcilerDeps:
|
||||
"""Everything a check is allowed to touch. Built once in main(), passed down.
|
||||
|
||||
Same shape as `WorkerDeps` for the same reason: the checks take `deps` instead of
|
||||
reaching for globals, so the integration tests below run every check against a real
|
||||
Postgres and a `FakeProvisioner` without a cluster anywhere in sight.
|
||||
Same shape as `WorkerDeps` and for the same reason: checks take `deps` instead of
|
||||
reaching for globals, so the integration tests run every check against a real Postgres
|
||||
and a `FakeProvisioner` with no cluster in sight.
|
||||
"""
|
||||
|
||||
pool: DictPool
|
||||
@@ -90,21 +89,17 @@ class ReconcilerDeps:
|
||||
async def check_drift(deps: ReconcilerDeps) -> None:
|
||||
"""The live helm releases versus what the database believes.
|
||||
|
||||
This is the only check that looks outside Postgres, and the only one that can catch the
|
||||
failure nothing else can: someone ran `helm uninstall` by hand, or a node was drained
|
||||
and the release never came back. The DB still says `ready` and still hands the tenant an
|
||||
endpoint that resolves to nothing.
|
||||
The only check that looks outside Postgres, and the only one that catches someone
|
||||
running `helm uninstall` by hand or a drained node whose release never came back — where
|
||||
the DB still says `ready` and still hands the tenant an endpoint resolving to nothing.
|
||||
|
||||
Two directions, two very different answers:
|
||||
Two directions, two different answers:
|
||||
|
||||
* **Release gone, DB says `ready`** -> re-enqueue provision. Safe, because provisioning
|
||||
is `helm upgrade --install` against a deterministic release name: converging on
|
||||
desired state, not a blind re-install.
|
||||
* **Release exists, DB knows nothing** -> log at error with release and namespace, and
|
||||
stop. **Never delete in v1.** The reconciler's view of "the DB knows nothing" is one
|
||||
query against one database; the release might belong to another team, another tool,
|
||||
or a migration half-finished. Deleting on that evidence is how an automated system
|
||||
takes down production faster than any human could. A human reads the log and decides.
|
||||
* **Release gone, DB says `ready`** -> re-enqueue provision. Safe because provisioning
|
||||
is `helm upgrade --install` against a deterministic release name.
|
||||
* **Release exists, DB knows nothing** -> log at error and stop. **Never delete in v1.**
|
||||
"The DB knows nothing" is one query against one database, and the release might belong
|
||||
to another team, another tool, or a half-finished migration. A human decides.
|
||||
"""
|
||||
with tracer().start_as_current_span("helm.list"):
|
||||
releases = await deps.provisioner.list_releases()
|
||||
@@ -132,31 +127,30 @@ async def check_drift(deps: ReconcilerDeps) -> None:
|
||||
{"instance_id": str(inst.id), "team": inst.team},
|
||||
)
|
||||
except Exception:
|
||||
# The task is already committed; the notification is a courtesy. A webhook
|
||||
# timing out must not abandon the rest of the sweep — the instances after this
|
||||
# one in the loop have the same problem and nobody else is coming to find them.
|
||||
# The task is committed; the notification is a courtesy. A webhook timing out
|
||||
# must not abandon the rest of the sweep — the instances after this one have the
|
||||
# same problem and nobody else is coming to find them.
|
||||
log.exception("notify.failed", instance_id=str(inst.id))
|
||||
|
||||
known = await deps.reconcile.known_releases()
|
||||
for name, namespace in sorted(live - known):
|
||||
# error, not warning: this is a resource nobody is billing for and nobody owns.
|
||||
# It will sit here every 60s until a human deletes it or adopts it. That is the
|
||||
# intended pressure.
|
||||
# error, not warning: a resource nobody owns and nobody is billing for. It repeats
|
||||
# every 60s until a human deletes or adopts it, which is the intended pressure.
|
||||
log.error("drift.orphan_release", release=name, namespace=namespace, action="none (v1 never deletes)")
|
||||
|
||||
|
||||
async def check_lease_expiry(deps: ReconcilerDeps) -> None:
|
||||
"""Tasks whose worker died -> back to `queued`.
|
||||
|
||||
A lease. No lock survives a power cut: a worker SIGKILLed mid-provision
|
||||
leaves `state='running'` with `locked_by` set and nobody running it, and no amount of
|
||||
cleanup code in the worker helps, because the worker is the part that died. `locked_at`
|
||||
plus a timeout is the only thing that recovers the row, which is why `locked_at` exists.
|
||||
A lease, not a lock: no lock survives a power cut. A worker SIGKILLed mid-provision
|
||||
leaves `state='running'` with `locked_by` set and nobody running it, and cleanup code in
|
||||
the worker cannot help because the worker is what died. `locked_at` plus a timeout is
|
||||
the only thing that recovers the row.
|
||||
|
||||
The 5-minute default must exceed the longest a healthy task can hold a lease, or the
|
||||
reconciler hands a still-running provision to a second worker. Handlers are idempotent,
|
||||
so that is survivable, though it still costs a duplicated helm run — which is why
|
||||
`lease_seconds` sits above helm's `--timeout`.
|
||||
The 5-minute default must exceed the longest a healthy task can hold a lease, or a
|
||||
still-running provision is handed to a second worker. Handlers are idempotent so that is
|
||||
survivable, but it costs a duplicated helm run — hence `lease_seconds` > helm's
|
||||
`--timeout`.
|
||||
"""
|
||||
freed = await deps.tasks.reset_expired_leases(deps.settings.lease_seconds)
|
||||
if freed:
|
||||
@@ -166,14 +160,13 @@ async def check_lease_expiry(deps: ReconcilerDeps) -> None:
|
||||
async def check_ttl(deps: ReconcilerDeps) -> None:
|
||||
"""Expired instances -> `deleting`, plus a deprovision task.
|
||||
|
||||
The line item that stops a demo cluster from becoming a permanent cloud bill. Also the
|
||||
sweep the API's DELETE route depends on: it CASes to `deleting` and enqueues in two
|
||||
statements, and a crash in between lands here on the next tick.
|
||||
What stops a demo cluster becoming a permanent cloud bill, and the sweep the API's
|
||||
DELETE route depends on: DELETE CASes and enqueues in two statements, and a crash
|
||||
between them lands here on the next tick.
|
||||
|
||||
Idempotent by construction — the work list excludes anything that already has a queued
|
||||
or running deprovision, and the CAS and the insert share one transaction. Without that
|
||||
guard, a deprovision that takes longer than 60 seconds gets a second task on the next
|
||||
tick, and a third on the tick after.
|
||||
Idempotent by construction — the work list excludes anything with a queued or running
|
||||
deprovision, and the CAS and insert share one transaction. Without that, a deprovision
|
||||
taking longer than 60 seconds collects a new task every tick.
|
||||
"""
|
||||
for inst in await deps.reconcile.due_for_deprovision():
|
||||
task_id = await deps.reconcile.enqueue_deprovision(inst.id)
|
||||
@@ -192,20 +185,18 @@ async def check_ttl(deps: ReconcilerDeps) -> None:
|
||||
async def check_version_drift(deps: ReconcilerDeps) -> None:
|
||||
"""The day-2 rollout: the work-list query, one service type at a time.
|
||||
|
||||
Everything that makes this safe is somewhere else, which is the point:
|
||||
Everything that makes it safe is somewhere else, which is the point:
|
||||
|
||||
* `list_upgradable` limits to `max_in_flight` and returns nothing while
|
||||
`rollout_state='halted'`, so a bad chart stops after one tenant.
|
||||
* `schedule_upgrade_at` turns the tenant's maintenance window into a `run_after`; the
|
||||
queue does the waiting, in `where run_after <= now()`. There is no scheduler here and
|
||||
there must not be one — a task parked in Postgres until 03:00 Sunday survives a
|
||||
reconciler restart, and an in-memory timer does not.
|
||||
* `security: true` in the catalog bypasses the window. A CVE with a public exploit does
|
||||
not wait until Sunday.
|
||||
* `schedule_upgrade_at` turns the maintenance window into a `run_after` and the queue
|
||||
does the waiting, in `where run_after <= now()`. No scheduler here, and there must not
|
||||
be one: a task parked in Postgres until 03:00 Sunday survives a restart, a timer does
|
||||
not.
|
||||
* `security: true` in the catalog bypasses the window.
|
||||
|
||||
A bad window spec is this instance's problem, not the fleet's: log it and move to the
|
||||
next one. Failing the whole check would let one tenant's typo freeze everyone's
|
||||
security rollout.
|
||||
A bad window spec is one instance's problem: log it and move on. Failing the check would
|
||||
let one tenant's typo freeze everyone's security rollout.
|
||||
"""
|
||||
now = deps.clock.now()
|
||||
|
||||
@@ -260,23 +251,20 @@ CHECKS: dict[str, Callable[[ReconcilerDeps], Awaitable[None]]] = {
|
||||
async def tick(deps: ReconcilerDeps) -> None:
|
||||
"""One pass: all four checks, then the gauges, then the heartbeat.
|
||||
|
||||
Checks first, gauges second: `svcforge_queue_depth` is read straight after the checks
|
||||
that add to the queue, so the value scraped is the value the tick left behind rather
|
||||
than one from before its own work.
|
||||
Checks first, gauges second, so `svcforge_queue_depth` reports what this tick left
|
||||
behind rather than what preceded its own work.
|
||||
|
||||
The heartbeat is set unconditionally, and that is deliberate. It answers "is the loop
|
||||
running", not "is everything fine" — the checks have their own alerts. Gating it on
|
||||
success would make `SvcforgeReconcilerStale` fire for a helm blip and mean two things
|
||||
at once, and an alert that means two things gets muted.
|
||||
The heartbeat is set unconditionally. It answers "is the loop running", not "is
|
||||
everything fine" — the checks have their own alerts. Gating it on success would make
|
||||
`SvcforgeReconcilerStale` fire for a helm blip and mean two things at once, and an alert
|
||||
that means two things gets muted.
|
||||
|
||||
The whole tick runs inside one span, which is a considered exception to "manual spans go
|
||||
around helm calls only". That rule exists so the API does not hand-roll spans that
|
||||
`opentelemetry-instrument` already creates for it. Nothing auto-instruments the
|
||||
reconciler: without a span here it emits no traces at all, and — because
|
||||
`inject_traceparent` serialises the *active* context — every task it enqueues would be
|
||||
written with a null `traceparent` and be unjoinable to the tick that decided to create
|
||||
it. One span per tick is what makes "why was this instance re-provisioned at 03:00?" a
|
||||
question the traces can answer.
|
||||
The whole tick runs in one span, a considered exception to "manual spans wrap helm calls
|
||||
only". That rule keeps the API from hand-rolling spans `opentelemetry-instrument`
|
||||
already makes; nothing auto-instruments the reconciler, so without this it emits no
|
||||
traces at all and — since `inject_traceparent` serialises the *active* context — every
|
||||
task it enqueues would carry a null `traceparent` and be unjoinable to the tick that
|
||||
created it.
|
||||
"""
|
||||
with tracer().start_as_current_span("reconciler.tick"):
|
||||
await _run_checks(deps)
|
||||
@@ -291,12 +279,11 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
|
||||
try:
|
||||
await check(deps)
|
||||
except Exception: # the tick is the error boundary
|
||||
# The swallow is the design. These four checks share nothing but a database
|
||||
# handle, and the value of a level-triggered loop is that it keeps running: an
|
||||
# unreachable cluster must not stop TTLs from expiring, and one tenant's broken
|
||||
# The swallow is the design. The four checks share nothing but a database
|
||||
# handle, and a level-triggered loop is only worth having if it keeps running:
|
||||
# an unreachable cluster must not stop TTLs expiring, and one tenant's broken
|
||||
# window spec must not stop drift detection. This means "this check achieved
|
||||
# nothing for 60 seconds", which the log says out loud. It never means "the
|
||||
# reconciler stops".
|
||||
# nothing for 60 seconds", never "the reconciler stops".
|
||||
log.exception("check.failed", check=name)
|
||||
|
||||
try:
|
||||
@@ -311,10 +298,10 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
|
||||
async def run_reconciler(deps: ReconcilerDeps, stop: asyncio.Event) -> None:
|
||||
"""Tick, sleep, repeat, until told to stop.
|
||||
|
||||
Tick first, then sleep: a pod that has just been restarted should reconcile now, not in
|
||||
sixty seconds. Fixed interval rather than a fixed period — a tick that overruns simply
|
||||
delays the next one, instead of stacking a second tick on top of the first, which for a
|
||||
singleton would be exactly the concurrent reconciler `replicas: 1` exists to prevent.
|
||||
Tick first, then sleep: a just-restarted pod should reconcile now, not in sixty seconds.
|
||||
Fixed interval rather than fixed period, so a tick that overruns delays the next one
|
||||
instead of stacking a second on top — which for a singleton is exactly the concurrent
|
||||
reconciler `replicas: 1` exists to prevent.
|
||||
"""
|
||||
while not stop.is_set():
|
||||
await tick(deps)
|
||||
@@ -353,13 +340,13 @@ async def _amain(once: bool, own_team: str, max_in_flight: int) -> None:
|
||||
|
||||
try:
|
||||
if once:
|
||||
# One pass and exit: the acceptance path, and how you drive a reconcile by hand
|
||||
# from a shell. No metrics server — nothing would ever scrape it.
|
||||
# One pass and exit: the acceptance path, and how to drive a reconcile by hand.
|
||||
# No metrics server — nothing would ever scrape it.
|
||||
await tick(deps)
|
||||
return
|
||||
|
||||
# settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT still overrides it,
|
||||
# through pydantic rather than a second CLI option, so the port has one definition.
|
||||
# settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT overrides it through
|
||||
# pydantic rather than a second CLI option, so the port has one definition.
|
||||
start_metrics_server(settings.metrics_port)
|
||||
|
||||
stop = asyncio.Event()
|
||||
|
||||
+19
-24
@@ -1,13 +1,12 @@
|
||||
"""Task handlers.
|
||||
|
||||
Every handler here obeys one rule: running it twice must equal running it once.
|
||||
Every handler obeys one rule: running it twice must equal running it once.
|
||||
|
||||
A worker can be SIGKILLed after helm has installed the release but
|
||||
before the DB row says so; the lease expires; another worker claims the same task and runs
|
||||
this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
|
||||
and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
|
||||
places: a deterministic `release_name`, and adapters that state desired state
|
||||
(`helm upgrade --install`) instead of issuing imperative commands.
|
||||
A worker can be SIGKILLed after helm installed the release but before the DB row says so;
|
||||
the lease expires, another worker claims the same task, and this function runs again. A
|
||||
handler that is not idempotent gives the tenant two Elasticsearches and you a bill.
|
||||
Idempotency is bought in two places: a deterministic `release_name`, and adapters that
|
||||
state desired state (`helm upgrade --install`) instead of issuing imperative commands.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -85,11 +84,10 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
|
||||
{"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
|
||||
)
|
||||
except Exception:
|
||||
# The provision succeeded and the row is already READY; the notification is a
|
||||
# courtesy. Letting a webhook timeout propagate would fail the task, and the
|
||||
# retry would hit the READY early-return and drop the notification anyway — so a
|
||||
# flaky notifier would turn every provision into a "failed" task. Same guard the
|
||||
# reconciler puts around its own notify.
|
||||
# The provision succeeded and the row is READY; the notification is a courtesy.
|
||||
# Propagating a webhook timeout would fail the task, and the retry would hit the
|
||||
# READY early-return and drop the notification anyway — so a flaky notifier
|
||||
# would turn every provision into a "failed" task.
|
||||
log.exception("notify.failed", instance_id=str(inst.id))
|
||||
|
||||
|
||||
@@ -104,10 +102,9 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
|
||||
# swallows not-found, because the desired state — no release — is already true.
|
||||
await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
|
||||
|
||||
# Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
|
||||
# the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
|
||||
# and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
|
||||
# asked to delete. Failing loudly turns a silent ping-pong into one visible error.
|
||||
# Raise rather than ignore the CAS result. Swallowing it leaves the release gone, the
|
||||
# row on `state=ready` with a dangling endpoint, the task marked done — and 60 seconds
|
||||
# later the drift check re-provisions the thing the tenant asked to delete.
|
||||
if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
|
||||
raise HandlerError(
|
||||
f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
|
||||
@@ -146,10 +143,9 @@ async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
|
||||
async def handle_verify(task: Task, deps: WorkerDeps) -> None:
|
||||
"""Post-upgrade health probe. On failure, halt the whole rollout for this service type.
|
||||
|
||||
One column decides whether the fleet keeps rolling. The work-list query returns nothing
|
||||
while `rollout_state='halted'`, so a bad chart stops after the first tenant instead of
|
||||
after all of them. You clear it with SQL, deliberately: an automatic un-halt would just
|
||||
resume breaking things.
|
||||
The work-list query returns nothing while `rollout_state='halted'`, so a bad chart stops
|
||||
after the first tenant instead of all of them. Clearing it is a deliberate SQL statement:
|
||||
an automatic un-halt would resume breaking things.
|
||||
"""
|
||||
inst = await _load_instance(task, deps)
|
||||
releases = {r.name for r in await deps.provisioner.list_releases()}
|
||||
@@ -157,10 +153,9 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
|
||||
if inst.release_name in releases:
|
||||
return
|
||||
|
||||
# `returning` + a `where` on the update half tells us whether THIS call was the one
|
||||
# that halted the rollout. The halt itself is idempotent; the page is not. Without the
|
||||
# distinction, a verify that fails its full retry budget sends five identical
|
||||
# notifications for one incident, spread across the backoff curve.
|
||||
# `returning` plus a `where` on the update half says whether THIS call halted the
|
||||
# rollout. The halt is idempotent; the page is not. Without the distinction, a verify
|
||||
# that burns its full retry budget sends five identical notifications for one incident.
|
||||
async with deps.pool.connection() as conn, conn.cursor() as cur:
|
||||
await cur.execute(
|
||||
"""insert into catalog_versions (service_type, rollout_state)
|
||||
|
||||
+24
-31
@@ -1,10 +1,9 @@
|
||||
"""The claim loop.
|
||||
|
||||
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report.
|
||||
That is the whole design, and the restraint is the point: LISTEN/NOTIFY would shave the
|
||||
latency, is fire-and-forget so it can never replace the poll anyway, is strictly extra
|
||||
code, and does not exist on pgbouncer's transaction pooler. The poll is not a placeholder
|
||||
for something better.
|
||||
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report. The
|
||||
poll is not a placeholder for something better: LISTEN/NOTIFY would shave latency, but it
|
||||
is fire-and-forget so it can never replace the poll, and it does not exist on pgbouncer's
|
||||
transaction pooler.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,17 +34,16 @@ log = obs.get_logger("svcforge.worker")
|
||||
async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
|
||||
"""Run a terminal report, and never let its failure escape.
|
||||
|
||||
Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
|
||||
TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
|
||||
DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
|
||||
not just this one. The task itself is safe either way: it stays `running` and the
|
||||
reconciler's lease sweep returns it to the queue. Losing the report costs one lease
|
||||
interval; losing the siblings costs their work.
|
||||
`_run_one` runs inside a TaskGroup, which cancels every sibling the moment one child
|
||||
raises — so a DB blip during `tasks.fail()` would abort every other in-flight provision
|
||||
on this pod. The task itself is safe either way: it stays `running` and the lease sweep
|
||||
returns it to the queue. Losing the report costs one lease interval; losing the siblings
|
||||
costs their work.
|
||||
"""
|
||||
try:
|
||||
if not await coro:
|
||||
# The lease was stolen while we were working: another worker owns this task
|
||||
# now and is mid-run. Reporting is theirs to do, not ours.
|
||||
# The lease was stolen while we were working: another worker owns this task now
|
||||
# and is mid-run. Reporting is theirs, not ours.
|
||||
log.warning("lease lost before report; another worker owns this task", task_id=task_id)
|
||||
except Exception:
|
||||
log.exception("could not report task %s (%s); lease will expire", task_id, what)
|
||||
@@ -55,10 +53,8 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
"""Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
|
||||
worker_id = deps.settings.worker_id
|
||||
try:
|
||||
# Every log line from here carries instance_id/task_id/team. Bound once, at claim,
|
||||
# rather than passed down: the alternative is threading three arguments through
|
||||
# every function that might log, and the first one anyone forgets is the one you
|
||||
# need at 3am.
|
||||
# Every log line from here carries instance_id/task_id/team. Bound once at claim
|
||||
# rather than threaded through every function that might log.
|
||||
obs.bind_task_context(task.instance_id, task.id, team=task.team or "unknown")
|
||||
log.info("task claimed", kind=task.kind.value, attempt=task.attempts)
|
||||
obs.TASKS_CLAIMED.labels(kind=task.kind.value).inc()
|
||||
@@ -72,9 +68,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
)
|
||||
return
|
||||
|
||||
# Re-parent to the span that enqueued this task. Without the stored traceparent
|
||||
# the worker's span starts a brand-new trace, and the POST that caused the work
|
||||
# is in a different trace to the helm call that did it.
|
||||
# Re-parent to the span that enqueued this task. Without the stored traceparent the
|
||||
# worker's span starts a new trace, putting the POST that caused the work in a
|
||||
# different trace from the helm call that did it.
|
||||
ctx = obs.context_from_traceparent(task.traceparent)
|
||||
started = time.monotonic()
|
||||
with obs.tracer().start_as_current_span(
|
||||
@@ -108,10 +104,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
"fail",
|
||||
)
|
||||
else:
|
||||
# Only provisions go in the provision histogram. The buckets run 10s..1800s
|
||||
# because they were sized for helm installs; a sub-second `verify` dropped
|
||||
# into the same series drags the p95 down and quietly stops
|
||||
# SvcforgeProvisionSlow from ever firing.
|
||||
# Only provisions go in the provision histogram. Its buckets run 10s..1800s
|
||||
# for helm installs, so a sub-second `verify` in the same series drags the
|
||||
# p95 down and quietly stops SvcforgeProvisionSlow from ever firing.
|
||||
if task.kind is TaskKind.PROVISION:
|
||||
obs.PROVISION_TIME.observe(time.monotonic() - started)
|
||||
await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
|
||||
@@ -122,10 +117,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
|
||||
"""Claim and run until told to stop, then drain what is in flight.
|
||||
|
||||
Draining is what makes a rolling deploy invisible. Exiting the `async with` block
|
||||
awaits every in-flight handler, so a pod that is being replaced finishes the provision
|
||||
it already started instead of abandoning it half-done for the lease to clean up
|
||||
five minutes later.
|
||||
Draining is what makes a rolling deploy invisible: exiting the `async with` awaits every
|
||||
in-flight handler, so a pod being replaced finishes the provision it started instead of
|
||||
abandoning it for the lease to clean up five minutes later.
|
||||
"""
|
||||
sem = asyncio.Semaphore(deps.settings.worker_concurrency)
|
||||
worker_id = deps.settings.worker_id
|
||||
@@ -158,9 +152,8 @@ async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
|
||||
async def _amain() -> None:
|
||||
settings: Settings = load_settings()
|
||||
|
||||
# Before anything else: nothing logged above this line is structured, and the metrics
|
||||
# the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
|
||||
# registry is up.
|
||||
# Before anything else: nothing above this line logs structured, and the metrics the
|
||||
# SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until it runs.
|
||||
obs.setup("svcforge-worker", settings)
|
||||
settings.check_production()
|
||||
obs.start_metrics_server(settings.metrics_port)
|
||||
|
||||
Reference in New Issue
Block a user