fix: three correctness bugs found in review + a dropped-log

fail() blanket-marked the instance `failed` for every dead-lettered task kind.
Only provision is correct. The others each left the instance in a state the
UPDATE then corrupted:
  - deprovision: `deleting` -> `failed` stranded the instance, because
    due_for_deprovision only re-selects ready/deleting, leaking the release
    reconcile.py promises to reclaim.
  - upgrade: helm --atomic rolled back, so the instance was still `ready` and
    serving the old version; `failed` mislabelled a healthy service and dropped
    it off the upgrade work-list.
  - verify: handle_verify already halted the rollout; the instance was `ready`.
Now gated on kind == provision, with a regression test per kind (control-tested
against the blanket UPDATE, which fails all three).

The API exception handler was registered on fastapi.HTTPException, a subclass
of starlette's. Starlette matches handlers by walking type(exc).__mro__, so
framework-raised 404/405 never hit it and returned {"detail": ...} instead of
ErrorBody. Registered on the starlette parent, and added a
RequestValidationError handler so body-validation 422s share the shape too.
Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field.

helm._list_releases_via_api built the httpx client with verify=<ca path>, which
loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an
error the except clause did not catch, crashing the reconciler tick as a bare
bug. Now the CA is checked for readability alongside the token, and a missing
one means "not in-cluster" and falls back to helm. Also documents the no-limit
pagination invariant and pins it with a test.

redis.py logged through stdlib logging with extra={"team": team}, which the
structlog bridge drops on the floor — the trap notify.py documents. Switched to
a bound logger with team as a kwarg.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 01:31:11 +00:00
parent d6c1b64512
commit d64c3c9f39
7 changed files with 299 additions and 31 deletions
+26 -1
View File
@@ -11,9 +11,11 @@ import asyncio
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from jwt import PyJWKClient
from starlette.exceptions import HTTPException
from services.api.models import ErrorBody
from services.api.routes import health, instances
@@ -89,6 +91,13 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
Handlers raise `detail={"code": ..., "message": ...}`; FastAPI's default would nest
that under `{"detail": {...}}`. Plain-string details (raised by FastAPI itself, e.g.
a 405) are wrapped so clients never have to branch on the body's type.
Registered on starlette's HTTPException, not fastapi's. fastapi.HTTPException is a
subclass, and Starlette matches handlers by walking type(exc).__mro__, so a handler
keyed on the subclass never fires for a framework-raised 404 or 405 — which are
starlette.HTTPException instances. Keying on the parent catches both: app handlers
raise the FastAPI subclass with a dict detail, the framework raises the parent with a
str detail, and the branch below renders each into ErrorBody.
"""
assert isinstance(exc, HTTPException) # noqa: S101 - registered only for HTTPException
# Widened to object deliberately. Starlette types `detail` as str, but FastAPI passes
@@ -102,6 +111,21 @@ async def _http_exception_handler(request: Request, exc: Exception) -> JSONRespo
return JSONResponse(status_code=exc.status_code, content=body.model_dump(), headers=exc.headers)
async def _validation_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Render request-validation failures as ErrorBody too.
A body that fails validation (a forbidden extra field, a bad type, an out-of-range
ttl_days) raises RequestValidationError, which the HTTPException handler above never
sees. Without this it returns FastAPI's default `{"detail": [...]}` — a second 422 shape
alongside the ErrorBody 422s the handlers raise. This gives every 422 one shape.
"""
assert isinstance(exc, RequestValidationError) # noqa: S101 - registered only for this
return JSONResponse(
status_code=422,
content=ErrorBody(code="validation_error", message=str(exc.errors())).model_dump(),
)
def create_app(settings: Settings | None = None) -> FastAPI:
"""App factory: lifespan, routers, exception handler, /metrics."""
settings = settings or load_settings()
@@ -132,6 +156,7 @@ def create_app(settings: Settings | None = None) -> FastAPI:
app.include_router(instances.router)
app.add_exception_handler(HTTPException, _http_exception_handler)
app.add_exception_handler(RequestValidationError, _validation_exception_handler)
return app