docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s

The comment pass is prose-only: every distinct "why" is kept, the
narration around it is not. Verified by AST-comparing each changed file
against HEAD with docstrings stripped — only the two files below differ
in executable code.

Two real fixes fell out of the read-through:

* The CLI documented itself as never touching the database, then called
  load_settings(), which requires SVCFORGE_PG_DSN. It refused to start
  without a Postgres URL it never opens. It now has its own two-field
  ClientSettings; the orphaned api_url/api_token are dropped from
  Settings, where nothing else read them.
* repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS
  comment run together above the wrong symbol.

USER_GUIDE.md is the caller-facing guide the README only gestured at:
auth, catalog, every endpoint with curl, the lifecycle, the error table,
rate limiting, the CLI, client generation, an end-to-end poll loop.

It records two facts about the live deployment rather than documenting a
flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP
behind it, so the API logs "JWKS warm-up failed" at startup and every
/v1 request is a 401. And `helm repo list` in the worker returns no
repositories, so the three bitnamilegacy/ catalog entries cannot resolve
at provision time; only the oci:// entries can.

make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 11:18:57 +00:00
parent 7918dc2b37
commit c53734d2bc
22 changed files with 936 additions and 872 deletions
+22 -27
View File
@@ -1,8 +1,8 @@
"""Dependency injection: how a handler gets a pool, a repo, a catalog, and a team.
Everything expensive — the pool, the JWKS client, the parsed catalog — is built once in
`lifespan` and parked on `app.state`. These functions only hand it out. A `Depends` that
does I/O per request is a `Depends` that does that I/O on every request forever.
`lifespan` and parked on `app.state`; these functions only hand it out. A `Depends` that
does I/O per request does that I/O on every request forever.
"""
from __future__ import annotations
@@ -22,10 +22,9 @@ from svcforge_core.repo.instances import InstanceRepo
from svcforge_core.repo.tasks import TaskRepo
from svcforge_core.settings import Settings
# The algorithm allow-list is the whole point of naming algorithms explicitly.
# `jwt.decode(..., algorithms=...)` without it accepts whatever the *token* claims in its
# own header — including `none`, and including HS256 verified with the RSA public key as
# an HMAC secret. Both are forgery. The list is not configuration.
# The algorithm allow-list is not configuration. Without it, `jwt.decode` accepts whatever
# the *token* claims in its own header — including `none`, and including HS256 verified
# with the RSA public key as an HMAC secret. Both are forgery.
ALLOWED_ALGORITHMS = ["RS256"]
# What `auth_disabled` returns. Settings.check_production() refuses that flag in prod.
@@ -42,9 +41,9 @@ _bearer = HTTPBearer(auto_error=False)
def _unauthorized() -> HTTPException:
"""One shape for every auth failure.
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: all the
same 401 with the same body. Telling a caller *which* one turns the endpoint into an
oracle they can tune a forgery against.
Expired, wrong issuer, wrong audience, bad signature, malformed, no header: the same 401
with the same body. Naming which one turns the endpoint into an oracle a forger can tune
against.
"""
return HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
@@ -68,8 +67,8 @@ async def get_pool(request: Request) -> DictPool:
def get_catalog(request: Request) -> dict[str, CatalogEntry]:
"""The catalog, parsed once at startup.
Read from disk per request and a mid-flight edit to catalog.yaml changes the answer
between two requests of the same deploy. Load it at startup; a change is a restart.
Read per request, a mid-flight edit to catalog.yaml would change the answer between two
requests of the same deploy. A catalog change is a restart.
"""
catalog: dict[str, CatalogEntry] = request.app.state.catalog
return catalog
@@ -102,16 +101,14 @@ async def get_current_team(
jwks_client: PyJWKClient | None = getattr(request.app.state, "jwks_client", None)
if jwks_client is None:
# Auth is on but there is no key source. Fail closed. Answering 500 here would be
# honest about the cause and would also let a misconfigured deploy be told apart
# from a bad token; 401 is the same answer a forger gets.
# Auth is on but there is no key source. Fail closed. A 500 would be honest about
# the cause and would also let a forger tell a misconfigured deploy from a bad token.
raise _unauthorized()
try:
# PyJWKClient keeps its own TTL cache, so this is a dict lookup on the hot path.
# It is only blocking on a cache MISS (key rotation) — hence to_thread, which
# costs a thread hop we take a handful of times a day rather than an event loop
# stalled on someone else's HTTP call once per rotation.
# PyJWKClient keeps a TTL cache, so this is a dict lookup on the hot path and only
# blocks on a miss (key rotation) — hence to_thread, a thread hop a few times a day
# rather than an event loop stalled on someone else's HTTP call.
signing_key = await _signing_key(jwks_client, creds.credentials)
claims: dict[str, Any] = jwt.decode(
creds.credentials,
@@ -139,10 +136,9 @@ async def get_current_team(
async def _signing_key(client: PyJWKClient, token: str) -> jwt.PyJWK:
"""Fetch the signing key off the event loop.
PyJWKClient.get_signing_key_from_jwt() does a synchronous urlopen on a cache miss.
Called directly from `async def`, that blocks the loop every other in-flight request
on this worker stops until the identity provider answers, and if it hangs, so does the
pod, and /readyz keeps saying it is fine.
`get_signing_key_from_jwt()` does a synchronous urlopen on a cache miss. Called directly
from `async def` it blocks the loop: every other in-flight request stops until the IdP
answers, and if the IdP hangs so does the pod, with /readyz still saying it is fine.
"""
return await asyncio.to_thread(client.get_signing_key_from_jwt, token)
@@ -159,11 +155,10 @@ async def rate_limit(
) -> None:
"""Per-team rate limiting. One Redis command per check, and it fails OPEN.
Failing open is the entire policy. Redis holds derived state; losing it must degrade
the platform, never stop it. A limiter that fails closed converts a cache outage into
a total outage, which is a strictly worse incident than the burst it was protecting
against — so `RateLimiter.check` swallows its own errors and returns `allowed=True`.
The 429 below therefore only ever comes from a real, counted overage.
Redis holds derived state, so losing it must degrade the platform rather than stop it: a
limiter that fails closed turns a cache outage into a total outage, a worse incident
than the burst it was guarding against. `RateLimiter.check` swallows its own errors and
returns `allowed=True`, so the 429 below only comes from a real, counted overage.
"""
limiter = get_rate_limiter(request)
if limiter is None:
+43 -52
View File
@@ -1,7 +1,7 @@
"""The app factory and its lifespan.
`create_app(settings)` is a factory, not a module-level `app = FastAPI()`, for one reason:
a test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
`create_app(settings)` is a factory rather than a module-level `app = FastAPI()` because a
test needs an app pointed at a throwaway Postgres, and an import-time app reads the real
environment at import time — before any fixture can say otherwise.
"""
@@ -32,23 +32,21 @@ log = obs.get_logger("svcforge.api")
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
"""Open the pool, yield, close the pool.
A lifespan context, not the deprecated startup/shutdown event decorators: those cannot
express "this resource lives for exactly as long as the app", and give you no place to
put the teardown next to the setup. Closing the pool matters — an unclosed pool means
connections linger server-side after SIGTERM, and on a pooled Postgres with a small
connection budget a few rolling deploys exhaust it.
A lifespan context, not the deprecated startup/shutdown decorators: those cannot express
"this resource lives exactly as long as the app" and leave no place to put teardown next
to setup. Closing matters — an unclosed pool leaves connections open server-side after
SIGTERM, and on a pooled Postgres with a small budget a few rolling deploys exhaust it.
(The old decorator's name is spelled nowhere in this package on purpose: CI greps for
the literal string, and a comment quoting it fails the gate just as loudly as a call.)
(The old decorator's name is spelled nowhere here on purpose: CI greps for the literal
string, so a comment quoting it fails the gate as loudly as a call would.)
"""
settings: Settings = app.state.settings
app.state.catalog = load_catalog(settings.catalog_path)
# Redis is optional by construction. `make_redis` returns None when no DSN is set, and
# every consumer treats None as "skip" so a deployment without Redis loses rate
# limiting and keeps everything else. Built here rather than per request because a
# connection pool per request is a connection pool per request.
# Redis is optional by construction: `make_redis` returns None when no DSN is set and
# every consumer treats None as "skip", so a deployment without Redis loses rate
# limiting and keeps everything else. Built once here, not per request.
redis = make_redis(settings)
app.state.redis = redis
app.state.rate_limiter = (
@@ -61,16 +59,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
await pool.open(wait=True)
app.state.pool = pool
# The pool is open from here on, so everything below is inside the try: an exception
# in JWKS setup must still close it, or a crash-looping pod leaks a connection per
# restart until the database refuses new ones.
# The pool is open from here, so everything below is inside the try: an exception in
# JWKS setup must still close it, or a crash-looping pod leaks a connection per restart
# until the database refuses new ones.
try:
if settings.jwks_url and not settings.auth_disabled:
client = PyJWKClient(settings.jwks_url, cache_keys=True, lifespan=300)
app.state.jwks_client = client
# Warm the cache off the loop so the first authenticated request does not pay
# a blocking urlopen. Best-effort: a slow identity provider must not stop the
# pod from starting — a cache miss later just costs one to_thread hop.
# Warm the cache off the loop so the first authenticated request does not pay a
# blocking urlopen. Best-effort: a slow IdP must not stop the pod from starting,
# and a miss later costs one to_thread hop.
try:
await asyncio.to_thread(client.get_signing_keys)
except Exception: # deliberate catch-all: startup must not hinge on the IdP being up
@@ -87,9 +85,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# --------------------------------------------------------------------------- API docs
# Everything a caller needs that the generated schema cannot express on its own. Kept next
# to create_app rather than in a README because /docs is what someone integrating actually
# reads, and a README in this repo is not something they have.
# What the generated schema cannot express. Kept next to create_app because /docs is what
# someone integrating reads, and they do not have this repo. USER_GUIDE.md is the longer form.
API_DESCRIPTION = """
Provision managed service instances into Kubernetes. The catalog offers Elasticsearch,
Redis and Postgres, plus two deliberately tiny entries — `podinfo` and `nginx` — for
@@ -144,21 +141,19 @@ OPENAPI_TAGS = [
async def _http_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Render HTTPException bodies as ErrorBody, so every error has one shape.
Handlers raise `detail={"code": ..., "message": ...}`; FastAPI's default would nest
that under `{"detail": {...}}`. Plain-string details (raised by FastAPI itself, e.g.
a 405) are wrapped so clients never have to branch on the body's type.
Handlers raise `detail={"code": ..., "message": ...}`, which FastAPI's default would
nest under `{"detail": {...}}`. Plain-string details (a framework 405, say) are wrapped
so clients never branch on the body's type.
Registered on starlette's HTTPException, not fastapi's. fastapi.HTTPException is a
subclass, and Starlette matches handlers by walking type(exc).__mro__, so a handler
keyed on the subclass never fires for a framework-raised 404 or 405 — which are
starlette.HTTPException instances. Keying on the parent catches both: app handlers
raise the FastAPI subclass with a dict detail, the framework raises the parent with a
str detail, and the branch below renders each into ErrorBody.
Registered on starlette's HTTPException, not fastapi's. The FastAPI class is a subclass
and Starlette matches handlers by walking `type(exc).__mro__`, so a handler keyed on the
subclass never fires for a framework-raised 404 or 405. Keying on the parent catches
both, and the branch below renders each into ErrorBody.
"""
assert isinstance(exc, HTTPException) # noqa: S101 - registered only for HTTPException
# Widened to object deliberately. Starlette types `detail` as str, but FastAPI passes
# through whatever a handler raised — our handlers raise dicts. Narrowing off the
# declared type would make mypy call the dict branch unreachable and delete it.
# Widened to object deliberately: Starlette types `detail` as str, but FastAPI passes
# through whatever a handler raised, and ours raise dicts. Narrowing off the declared
# type would let mypy call the dict branch unreachable and delete it.
detail: object = exc.detail
if isinstance(detail, dict) and "code" in detail and "message" in detail:
body = ErrorBody(code=str(detail["code"]), message=str(detail["message"]))
@@ -170,10 +165,9 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
async def _validation_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Render request-validation failures as ErrorBody too.
A body that fails validation (a forbidden extra field, a bad type, an out-of-range
ttl_days) raises RequestValidationError, which the HTTPException handler above never
sees. Without this it returns FastAPI's default `{"detail": [...]}` a second 422 shape
alongside the ErrorBody 422s the handlers raise. This gives every 422 one shape.
A forbidden extra field, a bad type or an out-of-range ttl_days raises
RequestValidationError, which the handler above never sees. Without this, FastAPI's
default `{"detail": [...]}` is a second 422 shape alongside the handlers' ErrorBody.
"""
assert isinstance(exc, RequestValidationError) # noqa: S101 - registered only for this
return JSONResponse(
@@ -186,22 +180,20 @@ def create_app(settings: Settings | None = None) -> FastAPI:
"""App factory: lifespan, routers, exception handler, /metrics."""
settings = settings or load_settings()
# FIRST, before any router is built and before any logger is bound. Without this the
# API is the one service of three that never configures structlog: its lines go out
# through logging.lastResort as bare text on stderr with no service, no trace_id and
# no JSON envelope — a parse failure in the collector, and unattributable in Loki.
# FIRST, before any router is built and any logger is bound. Without it the API is the
# one service of three that never configures structlog, and its lines go out through
# logging.lastResort as bare text on stderr no service, no trace_id, no JSON envelope.
# `settings.log_json` was silently inert here for the same reason.
obs.setup("svcforge-api", settings)
# Refuse the dev escape hatches when SVCFORGE_ENVIRONMENT says this is not a laptop.
# Called unconditionally and early: a check that only runs from a branch someone
# remembered to write is a check that does not run.
# Unconditional and early: a check that runs only from a branch someone remembered to
# write is a check that does not run.
settings.check_production()
# The description is the API's documentation. FastAPI renders it as markdown at /docs,
# and it is the only place a caller who does not have this repo can learn the two things
# that are not obvious from the schema: every write is asynchronous, and the instance
# lifecycle is a state machine they have to poll.
# The description is the API's documentation, rendered as markdown at /docs. It is the
# only place a caller without this repo learns the two things the schema cannot say:
# every write is asynchronous, and the lifecycle is a state machine they have to poll.
app = FastAPI(
title="svcforge",
version="0.1.0",
@@ -227,7 +219,6 @@ def app() -> FastAPI:
return create_app()
# There is deliberately no `if __name__ == "__main__"` here. `services/api/__main__.py` is
# the single entrypoint, and the image's ENTRYPOINT uses it. A second one in this module
# drifted from it — different log_level, different access_log — so `python -m services.api`
# and `python services/api/main.py` started the same app two different ways.
# No `if __name__ == "__main__"` here on purpose. `services/api/__main__.py` is the single
# entrypoint and the image's ENTRYPOINT uses it. A second one in this module drifted from
# it — different log_level, different access_log — so the same app started two ways.
+6 -8
View File
@@ -1,9 +1,8 @@
"""Wire types.
"""Wire types, deliberately not the domain models.
These are deliberately NOT the domain models. `Instance` carries `team`, `namespace` and
`release_name` — placement details a tenant has no business seeing and no business
setting. The response model is the allow-list that keeps them off the wire, which is why
it is written out by hand instead of derived from `Instance`.
`Instance` carries `team`, `namespace` and `release_name` — placement details a tenant has
no business seeing or setting. The response model is the allow-list that keeps them off the
wire, which is why it is written by hand instead of derived from `Instance`.
"""
from __future__ import annotations
@@ -19,9 +18,8 @@ class CreateInstanceRequest(BaseModel):
"""What a tenant may ask for.
`service_type` and `size` are plain strings, not enums: the catalog is data loaded at
runtime, so baking its keys into a type would mean a redeploy to add a service type,
and a 422 (schema) where the spec wants a 404 (unknown resource). They are validated
against the catalog in the handler.
runtime, so baking its keys into a type would mean a redeploy to add a service type and
a 422 where the spec wants a 404. The handler validates them against the catalog.
"""
model_config = ConfigDict(
+18 -20
View File
@@ -3,11 +3,11 @@
The distinction between the first two is the difference between a 30-second blip and a
fleet-wide outage:
* `/healthz` (liveness) answers "is this process wedged?" A failure here gets the
container KILLED. It must therefore touch NOTHING external. Wire it to the DB and a
20-second Postgres failover restarts every pod at once; they come back, find the DB
still down, and CrashLoopBackOff with exponential restart delays — so the fleet is now
down for minutes after the database recovered.
* `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the
container, so it must touch nothing external. Wired to the DB, a 20-second Postgres
failover restarts every pod at once; they come back, find the DB still down, and
CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the
database recovered.
* `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes
it from the Service endpoints. It is allowed to check dependencies, and it recovers by
itself the moment the check passes.
@@ -26,10 +26,10 @@ from services.api.models import ErrorBody
router = APIRouter(tags=["ops"])
# No PROMETHEUS_MULTIPROC_DIR here, deliberately: it exists for prefork servers where each
# worker process holds a slice of the counters. One uvicorn process per container means
# the default in-process registry is already correct, and multiproc mode would add a
# shared temp dir, a cleanup obligation, and a class of stale-file bugs for nothing.
# No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each
# process holds a slice of the counters. One uvicorn process per container makes the
# in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup
# obligation, and a class of stale-file bugs for nothing.
@router.get("/healthz", status_code=status.HTTP_200_OK)
@@ -45,10 +45,9 @@ async def healthz() -> dict[str, str]:
async def readyz(pool: PoolDep) -> dict[str, str]:
"""Readiness. Postgres only.
Postgres-only is the rule, and Redis is the temptation. Redis holds derived state
rate-limit buckets, caches — and everything degrades gracefully without it. Put it in
this check and an Upstash hiccup marks every pod unready, Kubernetes empties the
Service, and a cache outage becomes a total API outage.
Redis is the temptation and stays out: it holds derived state that degrades gracefully,
so checking it here would let an Upstash hiccup mark every pod unready, empty the
Service, and turn a cache outage into a total API outage.
"""
try:
async with pool.connection() as conn, conn.cursor() as cur:
@@ -68,14 +67,13 @@ async def readyz(pool: PoolDep) -> dict[str, str]:
async def metrics(request: Request) -> Response:
"""The Prometheus scrape endpoint.
A route rather than `app.mount("/metrics", make_asgi_app())`, for two reasons. A
Starlette `Mount` compiles to `^/metrics(?P<path>/.*)$`, which does not match a bare
`/metrics` — the exact URL every scrape config uses and a `Mount` is invisible to
OpenAPI, while the deliverable asks for `/metrics` in `openapi.json`.
A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount`
compiles to `^/metrics(?P<path>/.*)$`, which does not match the bare `/metrics` every
scrape config uses, and a Mount is invisible to OpenAPI.
The encoding is still prometheus_client's: `choose_encoder` reads the Accept header and
picks the exposition format (Prometheus text vs OpenMetrics) with its matching content
type. Hand-rolling either is how you end up serving text/plain that a scraper rejects.
The encoding stays prometheus_client's `choose_encoder` reads Accept and picks the
exposition format with its matching content type. Hand-rolling it serves text/plain a
scraper rejects.
"""
encoder, content_type = choose_encoder(request.headers.get("Accept", ""))
return Response(content=encoder(REGISTRY), media_type=content_type)
+27 -31
View File
@@ -1,12 +1,12 @@
"""The tenant-facing API.
Two rules run through every handler here:
Two rules run through every handler:
* **AuthZ is the WHERE clause.** No handler ever compares `inst.team` to the caller's
team, because the repo never returns another team's row to compare. A wrong-team id is
a 404. 403 would confirm the id exists, which is the leak.
* **The instance and its task commit together.** A committed instance with no task is an
instance that never provisions and that nothing will ever retry.
* **AuthZ is the WHERE clause.** No handler compares `inst.team` to the caller's team,
because the repo never returns another team's row to compare. A wrong-team id is a 404;
403 would confirm the id exists.
* **The instance and its task commit together.** A committed instance with no task never
provisions and nothing retries it.
"""
from __future__ import annotations
@@ -30,9 +30,9 @@ from services.api.models import CreateInstanceRequest, ErrorBody, InstanceRespon
from svcforge_core.domain.models import CatalogEntry, Instance, TaskKind
from svcforge_core.domain.states import IllegalTransition, InstanceState, transition
# Declared on the router so every error shape lands in openapi.json under ErrorBody.
# The exception handler already renders this at runtime; without declaring it, generated
# clients see the contract for 2xx only and invent their own guess for the rest.
# Declared on the router so every error shape lands in openapi.json under ErrorBody. The
# exception handler already renders these at runtime; undeclared, a generated client sees
# the contract for 2xx only and guesses the rest.
ERROR_RESPONSES: dict[int | str, dict[str, Any]] = {
401: {"model": ErrorBody, "description": "Missing or invalid credentials"},
404: {"model": ErrorBody, "description": "No such instance, or not this team's"},
@@ -46,14 +46,12 @@ router = APIRouter(prefix="/v1/instances", tags=["instances"], responses=ERROR_R
def release_name_for(team: str, service_type: str, instance_id: UUID) -> str:
"""The helm release name. Deterministic, and `unique` in the schema.
This is the idempotency anchor. A worker that dies after `helm install` but before it
marks the task done will retry, compute the same name, and `helm upgrade --install`
onto the same release instead of creating a second one. Derive it from anything that
is not already durable — a timestamp, a random suffix, the retry count — and a retry
provisions a duplicate.
The idempotency anchor: a worker that dies after `helm install` but before marking the
task done retries, computes the same name, and upgrades the same release instead of
creating a second one. Derive it from anything not already durable — a timestamp, a
random suffix, the retry count — and a retry provisions a duplicate.
Truncated to the uuid's first 8 chars to stay inside the 53-char limit helm imposes
on release names (Kubernetes label values, minus room for chart-generated suffixes).
Truncated to the uuid's first 8 chars to stay inside helm's 53-char release-name limit.
"""
return f"{team}-{service_type}-{str(instance_id)[:8]}"
@@ -66,9 +64,8 @@ def namespace_for(team: str) -> str:
def _resolve(catalog: dict[str, CatalogEntry], service_type: str, size: str) -> CatalogEntry:
"""Look up service_type + size, or raise the right 4xx.
The two failures are different HTTP problems and the spec asks for different codes:
an unknown service_type is a resource that does not exist (404); an unknown size for a
real service_type is a body the server understood and cannot process (422).
Two different HTTP problems: an unknown service_type is a resource that does not exist
(404), an unknown size for a real one is a body understood and unprocessable (422).
"""
entry = catalog.get(service_type)
if entry is None:
@@ -107,9 +104,9 @@ async def create_instance(
) -> Instance:
"""Accept a provisioning request. 202, never 201.
Nothing is provisioned when this returns. The row exists and a task is queued; a
worker will do the work seconds or minutes from now. 201 Created would be a lie about
a resource that does not exist yet, and clients would stop polling.
Nothing is provisioned when this returns: the row exists and a task is queued, and a
worker does the work seconds or minutes later. 201 Created would be a lie about a
resource that does not exist yet, and clients would stop polling.
"""
entry = _resolve(catalog, body.service_type, body.size)
@@ -123,9 +120,9 @@ async def create_instance(
state=InstanceState.REQUESTED,
namespace=namespace_for(team),
release_name=release_name_for(team, body.service_type, instance_id),
# Pinned from the catalog AT CREATION TIME, not read from the catalog later.
# This column records what is actually deployed; bumping catalog.yaml must show up
# as drift the reconciler can see, not silently rewrite history.
# Pinned at creation time, not read from the catalog later. The column records what
# is deployed, so bumping catalog.yaml shows up as drift the reconciler can see
# rather than silently rewriting history.
chart_version=entry.chart_version,
expires_at=now + timedelta(days=body.ttl_days) if body.ttl_days is not None else None,
created_at=now,
@@ -181,12 +178,11 @@ async def delete_instance(
) -> Instance:
"""state -> deleting, enqueue deprovision. 202: the helm uninstall has not happened yet.
Ordering note. `InstanceRepo.update_state` owns its own connection, so the CAS and the
enqueue cannot share one transaction without reaching around the repo. Given two
statements, the order is chosen for its failure mode: CAS first, enqueue second. A
crash in between leaves an instance in `deleting` with no task, which the reconciler's
sweep re-enqueues. The other order leaves a deprovision task pointing at a `ready`
instance, and a worker would tear down a live service nobody asked to delete.
`InstanceRepo.update_state` owns its own connection, so the CAS and the enqueue cannot
share a transaction without reaching around the repo. Given two statements, the order is
chosen for its failure mode: a crash between CAS and enqueue leaves an instance in
`deleting` with no task, which the reconciler's sweep re-enqueues. The reverse would
leave a deprovision task on a `ready` instance and tear down a live service.
"""
inst = await instances.get(instance_id, team)
if inst is None:
+22 -6
View File
@@ -1,9 +1,9 @@
"""svcforge — the control plane client.
This talks to the API over HTTP and never touches the database. That restraint is the
whole design: if the CLI could write to Postgres, every invariant the API enforces
(the state machine, the one-transaction create, AuthZ in the WHERE clause) would have a
back door, and the first 3am incident would go through it.
Talks to the API over HTTP and never touches the database. If the CLI could write to
Postgres, every invariant the API enforces — the state machine, the one-transaction create,
AuthZ in the WHERE clause would have a back door, and the first 3am incident would go
through it. `ClientSettings` below is what keeps that true in practice.
"""
from __future__ import annotations
@@ -16,9 +16,9 @@ from typing import Annotated, Any
import httpx
import typer
from pydantic_settings import BaseSettings, SettingsConfigDict
from svcforge_core.domain.states import InstanceState
from svcforge_core.settings import load_settings
app = typer.Typer(help="svcforge control plane client", no_args_is_help=True)
@@ -41,8 +41,24 @@ class Size(StrEnum):
MEDIUM = "medium"
class ClientSettings(BaseSettings):
"""The two values the CLI needs, and nothing else.
Its own model rather than `svcforge_core.settings.Settings`, which requires
`SVCFORGE_PG_DSN`: loading that here would refuse to run the CLI without a database URL
it then never opens, on a laptop that has no reason to hold one.
"""
model_config = SettingsConfigDict(
env_prefix="SVCFORGE_", env_file=".env", env_file_encoding="utf-8", extra="ignore", frozen=True
)
api_url: str = "http://localhost:8000"
api_token: str | None = None
def _client() -> httpx.Client:
settings = load_settings()
settings = ClientSettings()
headers = {"authorization": f"Bearer {settings.api_token}"} if settings.api_token else {}
return httpx.Client(base_url=settings.api_url, headers=headers, timeout=10.0)
+81 -94
View File
@@ -1,29 +1,28 @@
"""The control loop.
Every other service in svcforge is edge-triggered: a tenant POSTs, a row appears, a worker
claims it. Edge-triggered systems are correct exactly as long as nothing is ever missed —
and things are missed. A worker is SIGKILLed holding a lease. An operator runs
`helm uninstall` by hand. A pod dies between the CAS and the enqueue. Nobody sends an event
for any of that, because the thing that would have sent it is the thing that died.
Every other service here is edge-triggered: a tenant POSTs, a row appears, a worker claims
it. That is correct only as long as nothing is missed, and things are missed — a worker
SIGKILLed holding a lease, an operator running `helm uninstall` by hand, a pod dying
between the CAS and the enqueue. Nothing sends an event for any of it, because the thing
that would have sent it is the thing that died.
So: level-triggered. Every 60 seconds, compare the world to the database and enqueue what
is missing. The four checks below do not know or care what went wrong, or whether anything
did; they are the same code on the happy path and after an outage. That property is the
entire reason this service exists, and it is why each check is written as a *query for
work*, never as a reaction to an event.
is missing. The four checks below do not know what went wrong, or whether anything did;
they are the same code on the happy path and after an outage. That is why each is written
as a query for work rather than a reaction to an event.
Three rules hold the design together:
* **Singleton.** `replicas: 1`, `strategy: Recreate` in the chart. Two reconcilers
double-enqueue drift and race on TTL. There is no leader election here on purpose: the
correct lease for that lives in Postgres next to the data, not in a Redis lock, and
until there is a second replica to elect between, an election is a subsystem that can
only fail. One pod, and the `SvcforgeReconcilerStale` alert is what notices it is gone.
* **Each check is independent.** One failing check must not skip the other three. A helm
binary that cannot reach the API server must not stop TTLs from expiring.
* **Enqueue, never act.** The reconciler diagnoses; workers treat. It writes task rows and
instance states, and never calls `helm install`. The one exception is reading — the drift
check lists the live releases, because seeing reality is the job.
* **Singleton.** `replicas: 1`, `strategy: Recreate`. Two reconcilers double-enqueue drift
and race on TTL. No leader election on purpose — the right lease for that lives in
Postgres next to the data, and until there is a second replica to elect between, an
election is a subsystem that can only fail. The `SvcforgeReconcilerStale` alert notices
when the one pod is gone.
* **Each check is independent.** A helm binary that cannot reach the API server must not
stop TTLs from expiring.
* **Enqueue, never act.** The reconciler diagnoses and workers treat: it writes task rows
and instance states and never calls `helm install`. Reading is the exception, since
seeing reality is the job.
"""
from __future__ import annotations
@@ -63,9 +62,9 @@ log = get_logger("svcforge.reconciler")
class ReconcilerDeps:
"""Everything a check is allowed to touch. Built once in main(), passed down.
Same shape as `WorkerDeps` for the same reason: the checks take `deps` instead of
reaching for globals, so the integration tests below run every check against a real
Postgres and a `FakeProvisioner` without a cluster anywhere in sight.
Same shape as `WorkerDeps` and for the same reason: checks take `deps` instead of
reaching for globals, so the integration tests run every check against a real Postgres
and a `FakeProvisioner` with no cluster in sight.
"""
pool: DictPool
@@ -90,21 +89,17 @@ class ReconcilerDeps:
async def check_drift(deps: ReconcilerDeps) -> None:
"""The live helm releases versus what the database believes.
This is the only check that looks outside Postgres, and the only one that can catch the
failure nothing else can: someone ran `helm uninstall` by hand, or a node was drained
and the release never came back. The DB still says `ready` and still hands the tenant an
endpoint that resolves to nothing.
The only check that looks outside Postgres, and the only one that catches someone
running `helm uninstall` by hand or a drained node whose release never came back — where
the DB still says `ready` and still hands the tenant an endpoint resolving to nothing.
Two directions, two very different answers:
Two directions, two different answers:
* **Release gone, DB says `ready`** -> re-enqueue provision. Safe, because provisioning
is `helm upgrade --install` against a deterministic release name: converging on
desired state, not a blind re-install.
* **Release exists, DB knows nothing** -> log at error with release and namespace, and
stop. **Never delete in v1.** The reconciler's view of "the DB knows nothing" is one
query against one database; the release might belong to another team, another tool,
or a migration half-finished. Deleting on that evidence is how an automated system
takes down production faster than any human could. A human reads the log and decides.
* **Release gone, DB says `ready`** -> re-enqueue provision. Safe because provisioning
is `helm upgrade --install` against a deterministic release name.
* **Release exists, DB knows nothing** -> log at error and stop. **Never delete in v1.**
"The DB knows nothing" is one query against one database, and the release might belong
to another team, another tool, or a half-finished migration. A human decides.
"""
with tracer().start_as_current_span("helm.list"):
releases = await deps.provisioner.list_releases()
@@ -132,31 +127,30 @@ async def check_drift(deps: ReconcilerDeps) -> None:
{"instance_id": str(inst.id), "team": inst.team},
)
except Exception:
# The task is already committed; the notification is a courtesy. A webhook
# timing out must not abandon the rest of the sweep — the instances after this
# one in the loop have the same problem and nobody else is coming to find them.
# The task is committed; the notification is a courtesy. A webhook timing out
# must not abandon the rest of the sweep — the instances after this one have the
# same problem and nobody else is coming to find them.
log.exception("notify.failed", instance_id=str(inst.id))
known = await deps.reconcile.known_releases()
for name, namespace in sorted(live - known):
# error, not warning: this is a resource nobody is billing for and nobody owns.
# It will sit here every 60s until a human deletes it or adopts it. That is the
# intended pressure.
# error, not warning: a resource nobody owns and nobody is billing for. It repeats
# every 60s until a human deletes or adopts it, which is the intended pressure.
log.error("drift.orphan_release", release=name, namespace=namespace, action="none (v1 never deletes)")
async def check_lease_expiry(deps: ReconcilerDeps) -> None:
"""Tasks whose worker died -> back to `queued`.
A lease. No lock survives a power cut: a worker SIGKILLed mid-provision
leaves `state='running'` with `locked_by` set and nobody running it, and no amount of
cleanup code in the worker helps, because the worker is the part that died. `locked_at`
plus a timeout is the only thing that recovers the row, which is why `locked_at` exists.
A lease, not a lock: no lock survives a power cut. A worker SIGKILLed mid-provision
leaves `state='running'` with `locked_by` set and nobody running it, and cleanup code in
the worker cannot help because the worker is what died. `locked_at` plus a timeout is
the only thing that recovers the row.
The 5-minute default must exceed the longest a healthy task can hold a lease, or the
reconciler hands a still-running provision to a second worker. Handlers are idempotent,
so that is survivable, though it still costs a duplicated helm run — which is why
`lease_seconds` sits above helm's `--timeout`.
The 5-minute default must exceed the longest a healthy task can hold a lease, or a
still-running provision is handed to a second worker. Handlers are idempotent so that is
survivable, but it costs a duplicated helm run — hence `lease_seconds` > helm's
`--timeout`.
"""
freed = await deps.tasks.reset_expired_leases(deps.settings.lease_seconds)
if freed:
@@ -166,14 +160,13 @@ async def check_lease_expiry(deps: ReconcilerDeps) -> None:
async def check_ttl(deps: ReconcilerDeps) -> None:
"""Expired instances -> `deleting`, plus a deprovision task.
The line item that stops a demo cluster from becoming a permanent cloud bill. Also the
sweep the API's DELETE route depends on: it CASes to `deleting` and enqueues in two
statements, and a crash in between lands here on the next tick.
What stops a demo cluster becoming a permanent cloud bill, and the sweep the API's
DELETE route depends on: DELETE CASes and enqueues in two statements, and a crash
between them lands here on the next tick.
Idempotent by construction — the work list excludes anything that already has a queued
or running deprovision, and the CAS and the insert share one transaction. Without that
guard, a deprovision that takes longer than 60 seconds gets a second task on the next
tick, and a third on the tick after.
Idempotent by construction — the work list excludes anything with a queued or running
deprovision, and the CAS and insert share one transaction. Without that, a deprovision
taking longer than 60 seconds collects a new task every tick.
"""
for inst in await deps.reconcile.due_for_deprovision():
task_id = await deps.reconcile.enqueue_deprovision(inst.id)
@@ -192,20 +185,18 @@ async def check_ttl(deps: ReconcilerDeps) -> None:
async def check_version_drift(deps: ReconcilerDeps) -> None:
"""The day-2 rollout: the work-list query, one service type at a time.
Everything that makes this safe is somewhere else, which is the point:
Everything that makes it safe is somewhere else, which is the point:
* `list_upgradable` limits to `max_in_flight` and returns nothing while
`rollout_state='halted'`, so a bad chart stops after one tenant.
* `schedule_upgrade_at` turns the tenant's maintenance window into a `run_after`; the
queue does the waiting, in `where run_after <= now()`. There is no scheduler here and
there must not be one a task parked in Postgres until 03:00 Sunday survives a
reconciler restart, and an in-memory timer does not.
* `security: true` in the catalog bypasses the window. A CVE with a public exploit does
not wait until Sunday.
* `schedule_upgrade_at` turns the maintenance window into a `run_after` and the queue
does the waiting, in `where run_after <= now()`. No scheduler here, and there must not
be one: a task parked in Postgres until 03:00 Sunday survives a restart, a timer does
not.
* `security: true` in the catalog bypasses the window.
A bad window spec is this instance's problem, not the fleet's: log it and move to the
next one. Failing the whole check would let one tenant's typo freeze everyone's
security rollout.
A bad window spec is one instance's problem: log it and move on. Failing the check would
let one tenant's typo freeze everyone's security rollout.
"""
now = deps.clock.now()
@@ -260,23 +251,20 @@ CHECKS: dict[str, Callable[[ReconcilerDeps], Awaitable[None]]] = {
async def tick(deps: ReconcilerDeps) -> None:
"""One pass: all four checks, then the gauges, then the heartbeat.
Checks first, gauges second: `svcforge_queue_depth` is read straight after the checks
that add to the queue, so the value scraped is the value the tick left behind rather
than one from before its own work.
Checks first, gauges second, so `svcforge_queue_depth` reports what this tick left
behind rather than what preceded its own work.
The heartbeat is set unconditionally, and that is deliberate. It answers "is the loop
running", not "is everything fine" — the checks have their own alerts. Gating it on
success would make `SvcforgeReconcilerStale` fire for a helm blip and mean two things
at once, and an alert that means two things gets muted.
The heartbeat is set unconditionally. It answers "is the loop running", not "is
everything fine" — the checks have their own alerts. Gating it on success would make
`SvcforgeReconcilerStale` fire for a helm blip and mean two things at once, and an alert
that means two things gets muted.
The whole tick runs inside one span, which is a considered exception to "manual spans go
around helm calls only". That rule exists so the API does not hand-roll spans that
`opentelemetry-instrument` already creates for it. Nothing auto-instruments the
reconciler: without a span here it emits no traces at all, and — because
`inject_traceparent` serialises the *active* context — every task it enqueues would be
written with a null `traceparent` and be unjoinable to the tick that decided to create
it. One span per tick is what makes "why was this instance re-provisioned at 03:00?" a
question the traces can answer.
The whole tick runs in one span, a considered exception to "manual spans wrap helm calls
only". That rule keeps the API from hand-rolling spans `opentelemetry-instrument`
already makes; nothing auto-instruments the reconciler, so without this it emits no
traces at all and — since `inject_traceparent` serialises the *active* context — every
task it enqueues would carry a null `traceparent` and be unjoinable to the tick that
created it.
"""
with tracer().start_as_current_span("reconciler.tick"):
await _run_checks(deps)
@@ -291,12 +279,11 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
try:
await check(deps)
except Exception: # the tick is the error boundary
# The swallow is the design. These four checks share nothing but a database
# handle, and the value of a level-triggered loop is that it keeps running: an
# unreachable cluster must not stop TTLs from expiring, and one tenant's broken
# The swallow is the design. The four checks share nothing but a database
# handle, and a level-triggered loop is only worth having if it keeps running:
# an unreachable cluster must not stop TTLs expiring, and one tenant's broken
# window spec must not stop drift detection. This means "this check achieved
# nothing for 60 seconds", which the log says out loud. It never means "the
# reconciler stops".
# nothing for 60 seconds", never "the reconciler stops".
log.exception("check.failed", check=name)
try:
@@ -311,10 +298,10 @@ async def _run_checks(deps: ReconcilerDeps) -> None:
async def run_reconciler(deps: ReconcilerDeps, stop: asyncio.Event) -> None:
"""Tick, sleep, repeat, until told to stop.
Tick first, then sleep: a pod that has just been restarted should reconcile now, not in
sixty seconds. Fixed interval rather than a fixed period a tick that overruns simply
delays the next one, instead of stacking a second tick on top of the first, which for a
singleton would be exactly the concurrent reconciler `replicas: 1` exists to prevent.
Tick first, then sleep: a just-restarted pod should reconcile now, not in sixty seconds.
Fixed interval rather than fixed period, so a tick that overruns delays the next one
instead of stacking a second on top — which for a singleton is exactly the concurrent
reconciler `replicas: 1` exists to prevent.
"""
while not stop.is_set():
await tick(deps)
@@ -353,13 +340,13 @@ async def _amain(once: bool, own_team: str, max_in_flight: int) -> None:
try:
if once:
# One pass and exit: the acceptance path, and how you drive a reconcile by hand
# from a shell. No metrics server — nothing would ever scrape it.
# One pass and exit: the acceptance path, and how to drive a reconcile by hand.
# No metrics server — nothing would ever scrape it.
await tick(deps)
return
# settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT still overrides it,
# through pydantic rather than a second CLI option, so the port has one definition.
# settings.metrics_port, like the worker. SVCFORGE_METRICS_PORT overrides it through
# pydantic rather than a second CLI option, so the port has one definition.
start_metrics_server(settings.metrics_port)
stop = asyncio.Event()
+19 -24
View File
@@ -1,13 +1,12 @@
"""Task handlers.
Every handler here obeys one rule: running it twice must equal running it once.
Every handler obeys one rule: running it twice must equal running it once.
A worker can be SIGKILLed after helm has installed the release but
before the DB row says so; the lease expires; another worker claims the same task and runs
this function again. If the handler is not idempotent, the tenant gets two Elasticsearches
and you get a bill. Idempotency is what makes the crash safe, and it is bought in two
places: a deterministic `release_name`, and adapters that state desired state
(`helm upgrade --install`) instead of issuing imperative commands.
A worker can be SIGKILLed after helm installed the release but before the DB row says so;
the lease expires, another worker claims the same task, and this function runs again. A
handler that is not idempotent gives the tenant two Elasticsearches and you a bill.
Idempotency is bought in two places: a deterministic `release_name`, and adapters that
state desired state (`helm upgrade --install`) instead of issuing imperative commands.
"""
from __future__ import annotations
@@ -85,11 +84,10 @@ async def handle_provision(task: Task, deps: WorkerDeps) -> None:
{"instance_id": str(inst.id), "team": inst.team, "service_type": inst.service_type},
)
except Exception:
# The provision succeeded and the row is already READY; the notification is a
# courtesy. Letting a webhook timeout propagate would fail the task, and the
# retry would hit the READY early-return and drop the notification anyway — so a
# flaky notifier would turn every provision into a "failed" task. Same guard the
# reconciler puts around its own notify.
# The provision succeeded and the row is READY; the notification is a courtesy.
# Propagating a webhook timeout would fail the task, and the retry would hit the
# READY early-return and drop the notification anyway — so a flaky notifier
# would turn every provision into a "failed" task.
log.exception("notify.failed", instance_id=str(inst.id))
@@ -104,10 +102,9 @@ async def handle_deprovision(task: Task, deps: WorkerDeps) -> None:
# swallows not-found, because the desired state — no release — is already true.
await deps.provisioner.uninstall(release=inst.release_name, ns=inst.namespace)
# Raise rather than ignore the CAS result. Swallowing it means: the release is gone,
# the row keeps `state=ready` and its now-dangling endpoint, the task is marked done,
# and 60 seconds later the reconciler's drift check re-provisions the thing the tenant
# asked to delete. Failing loudly turns a silent ping-pong into one visible error.
# Raise rather than ignore the CAS result. Swallowing it leaves the release gone, the
# row on `state=ready` with a dangling endpoint, the task marked done — and 60 seconds
# later the drift check re-provisions the thing the tenant asked to delete.
if not await deps.instances.update_state(inst.id, InstanceState.DELETING, InstanceState.DELETED):
raise HandlerError(
f"instance {inst.id} was {inst.state.value}, expected {InstanceState.DELETING.value}"
@@ -146,10 +143,9 @@ async def handle_upgrade(task: Task, deps: WorkerDeps) -> None:
async def handle_verify(task: Task, deps: WorkerDeps) -> None:
"""Post-upgrade health probe. On failure, halt the whole rollout for this service type.
One column decides whether the fleet keeps rolling. The work-list query returns nothing
while `rollout_state='halted'`, so a bad chart stops after the first tenant instead of
after all of them. You clear it with SQL, deliberately: an automatic un-halt would just
resume breaking things.
The work-list query returns nothing while `rollout_state='halted'`, so a bad chart stops
after the first tenant instead of all of them. Clearing it is a deliberate SQL statement:
an automatic un-halt would resume breaking things.
"""
inst = await _load_instance(task, deps)
releases = {r.name for r in await deps.provisioner.list_releases()}
@@ -157,10 +153,9 @@ async def handle_verify(task: Task, deps: WorkerDeps) -> None:
if inst.release_name in releases:
return
# `returning` + a `where` on the update half tells us whether THIS call was the one
# that halted the rollout. The halt itself is idempotent; the page is not. Without the
# distinction, a verify that fails its full retry budget sends five identical
# notifications for one incident, spread across the backoff curve.
# `returning` plus a `where` on the update half says whether THIS call halted the
# rollout. The halt is idempotent; the page is not. Without the distinction, a verify
# that burns its full retry budget sends five identical notifications for one incident.
async with deps.pool.connection() as conn, conn.cursor() as cur:
await cur.execute(
"""insert into catalog_versions (service_type, rollout_state)
+24 -31
View File
@@ -1,10 +1,9 @@
"""The claim loop.
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report.
That is the whole design, and the restraint is the point: LISTEN/NOTIFY would shave the
latency, is fire-and-forget so it can never replace the poll anyway, is strictly extra
code, and does not exist on pgbouncer's transaction pooler. The poll is not a placeholder
for something better.
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report. The
poll is not a placeholder for something better: LISTEN/NOTIFY would shave latency, but it
is fire-and-forget so it can never replace the poll, and it does not exist on pgbouncer's
transaction pooler.
"""
from __future__ import annotations
@@ -35,17 +34,16 @@ log = obs.get_logger("svcforge.worker")
async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
"""Run a terminal report, and never let its failure escape.
Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
not just this one. The task itself is safe either way: it stays `running` and the
reconciler's lease sweep returns it to the queue. Losing the report costs one lease
interval; losing the siblings costs their work.
`_run_one` runs inside a TaskGroup, which cancels every sibling the moment one child
raises — so a DB blip during `tasks.fail()` would abort every other in-flight provision
on this pod. The task itself is safe either way: it stays `running` and the lease sweep
returns it to the queue. Losing the report costs one lease interval; losing the siblings
costs their work.
"""
try:
if not await coro:
# The lease was stolen while we were working: another worker owns this task
# now and is mid-run. Reporting is theirs to do, not ours.
# The lease was stolen while we were working: another worker owns this task now
# and is mid-run. Reporting is theirs, not ours.
log.warning("lease lost before report; another worker owns this task", task_id=task_id)
except Exception:
log.exception("could not report task %s (%s); lease will expire", task_id, what)
@@ -55,10 +53,8 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
"""Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
worker_id = deps.settings.worker_id
try:
# Every log line from here carries instance_id/task_id/team. Bound once, at claim,
# rather than passed down: the alternative is threading three arguments through
# every function that might log, and the first one anyone forgets is the one you
# need at 3am.
# Every log line from here carries instance_id/task_id/team. Bound once at claim
# rather than threaded through every function that might log.
obs.bind_task_context(task.instance_id, task.id, team=task.team or "unknown")
log.info("task claimed", kind=task.kind.value, attempt=task.attempts)
obs.TASKS_CLAIMED.labels(kind=task.kind.value).inc()
@@ -72,9 +68,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
)
return
# Re-parent to the span that enqueued this task. Without the stored traceparent
# the worker's span starts a brand-new trace, and the POST that caused the work
# is in a different trace to the helm call that did it.
# Re-parent to the span that enqueued this task. Without the stored traceparent the
# worker's span starts a new trace, putting the POST that caused the work in a
# different trace from the helm call that did it.
ctx = obs.context_from_traceparent(task.traceparent)
started = time.monotonic()
with obs.tracer().start_as_current_span(
@@ -108,10 +104,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
"fail",
)
else:
# Only provisions go in the provision histogram. The buckets run 10s..1800s
# because they were sized for helm installs; a sub-second `verify` dropped
# into the same series drags the p95 down and quietly stops
# SvcforgeProvisionSlow from ever firing.
# Only provisions go in the provision histogram. Its buckets run 10s..1800s
# for helm installs, so a sub-second `verify` in the same series drags the
# p95 down and quietly stops SvcforgeProvisionSlow from ever firing.
if task.kind is TaskKind.PROVISION:
obs.PROVISION_TIME.observe(time.monotonic() - started)
await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
@@ -122,10 +117,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
"""Claim and run until told to stop, then drain what is in flight.
Draining is what makes a rolling deploy invisible. Exiting the `async with` block
awaits every in-flight handler, so a pod that is being replaced finishes the provision
it already started instead of abandoning it half-done for the lease to clean up
five minutes later.
Draining is what makes a rolling deploy invisible: exiting the `async with` awaits every
in-flight handler, so a pod being replaced finishes the provision it started instead of
abandoning it for the lease to clean up five minutes later.
"""
sem = asyncio.Semaphore(deps.settings.worker_concurrency)
worker_id = deps.settings.worker_id
@@ -158,9 +152,8 @@ async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
async def _amain() -> None:
settings: Settings = load_settings()
# Before anything else: nothing logged above this line is structured, and the metrics
# the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
# registry is up.
# Before anything else: nothing above this line logs structured, and the metrics the
# SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until it runs.
obs.setup("svcforge-worker", settings)
settings.check_production()
obs.start_metrics_server(settings.metrics_port)