fix: three correctness bugs found in review + a dropped-log
fail() blanket-marked the instance `failed` for every dead-lettered task kind.
Only provision is correct. The others each left the instance in a state the
UPDATE then corrupted:
- deprovision: `deleting` -> `failed` stranded the instance, because
due_for_deprovision only re-selects ready/deleting, leaking the release
reconcile.py promises to reclaim.
- upgrade: helm --atomic rolled back, so the instance was still `ready` and
serving the old version; `failed` mislabelled a healthy service and dropped
it off the upgrade work-list.
- verify: handle_verify already halted the rollout; the instance was `ready`.
Now gated on kind == provision, with a regression test per kind (control-tested
against the blanket UPDATE, which fails all three).
The API exception handler was registered on fastapi.HTTPException, a subclass
of starlette's. Starlette matches handlers by walking type(exc).__mro__, so
framework-raised 404/405 never hit it and returned {"detail": ...} instead of
ErrorBody. Registered on the starlette parent, and added a
RequestValidationError handler so body-validation 422s share the shape too.
Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field.
helm._list_releases_via_api built the httpx client with verify=<ca path>, which
loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an
error the except clause did not catch, crashing the reconciler tick as a bare
bug. Now the CA is checked for readability alongside the token, and a missing
one means "not in-cluster" and falls back to helm. Also documents the no-limit
pagination invariant and pins it with a test.
redis.py logged through stdlib logging with extra={"team": team}, which the
structlog bridge drops on the floor — the trap notify.py documents. Switched to
a bound logger with team as a kwarg.
This commit is contained in:
+26
-1
@@ -11,9 +11,11 @@ import asyncio
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from jwt import PyJWKClient
|
||||
from starlette.exceptions import HTTPException
|
||||
|
||||
from services.api.models import ErrorBody
|
||||
from services.api.routes import health, instances
|
||||
@@ -89,6 +91,13 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
|
||||
Handlers raise `detail={"code": ..., "message": ...}`; FastAPI's default would nest
|
||||
that under `{"detail": {...}}`. Plain-string details (raised by FastAPI itself, e.g.
|
||||
a 405) are wrapped so clients never have to branch on the body's type.
|
||||
|
||||
Registered on starlette's HTTPException, not fastapi's. fastapi.HTTPException is a
|
||||
subclass, and Starlette matches handlers by walking type(exc).__mro__, so a handler
|
||||
keyed on the subclass never fires for a framework-raised 404 or 405 — which are
|
||||
starlette.HTTPException instances. Keying on the parent catches both: app handlers
|
||||
raise the FastAPI subclass with a dict detail, the framework raises the parent with a
|
||||
str detail, and the branch below renders each into ErrorBody.
|
||||
"""
|
||||
assert isinstance(exc, HTTPException) # noqa: S101 - registered only for HTTPException
|
||||
# Widened to object deliberately. Starlette types `detail` as str, but FastAPI passes
|
||||
@@ -102,6 +111,21 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
|
||||
return JSONResponse(status_code=exc.status_code, content=body.model_dump(), headers=exc.headers)
|
||||
|
||||
|
||||
async def _validation_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
"""Render request-validation failures as ErrorBody too.
|
||||
|
||||
A body that fails validation (a forbidden extra field, a bad type, an out-of-range
|
||||
ttl_days) raises RequestValidationError, which the HTTPException handler above never
|
||||
sees. Without this it returns FastAPI's default `{"detail": [...]}` — a second 422 shape
|
||||
alongside the ErrorBody 422s the handlers raise. This gives every 422 one shape.
|
||||
"""
|
||||
assert isinstance(exc, RequestValidationError) # noqa: S101 - registered only for this
|
||||
return JSONResponse(
|
||||
status_code=422,
|
||||
content=ErrorBody(code="validation_error", message=str(exc.errors())).model_dump(),
|
||||
)
|
||||
|
||||
|
||||
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
"""App factory: lifespan, routers, exception handler, /metrics."""
|
||||
settings = settings or load_settings()
|
||||
@@ -132,6 +156,7 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
app.include_router(instances.router)
|
||||
|
||||
app.add_exception_handler(HTTPException, _http_exception_handler)
|
||||
app.add_exception_handler(RequestValidationError, _validation_exception_handler)
|
||||
return app
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user