fix: three correctness bugs found in review + a dropped-log

fail() blanket-marked the instance `failed` for every dead-lettered task kind.
Only provision is correct. The others each left the instance in a state the
UPDATE then corrupted:
  - deprovision: `deleting` -> `failed` stranded the instance, because
    due_for_deprovision only re-selects ready/deleting, leaking the release
    reconcile.py promises to reclaim.
  - upgrade: helm --atomic rolled back, so the instance was still `ready` and
    serving the old version; `failed` mislabelled a healthy service and dropped
    it off the upgrade work-list.
  - verify: handle_verify already halted the rollout; the instance was `ready`.
Now gated on kind == provision, with a regression test per kind (control-tested
against the blanket UPDATE, which fails all three).

The API exception handler was registered on fastapi.HTTPException, a subclass
of starlette's. Starlette matches handlers by walking type(exc).__mro__, so
framework-raised 404/405 never hit it and returned {"detail": ...} instead of
ErrorBody. Registered on the starlette parent, and added a
RequestValidationError handler so body-validation 422s share the shape too.
Tests assert the ErrorBody shape for framework 404, 405, and a forbidden field.

helm._list_releases_via_api built the httpx client with verify=<ca path>, which
loads the CA eagerly and raises OSError on a half-mounted ServiceAccount — an
error the except clause did not catch, crashing the reconciler tick as a bare
bug. Now the CA is checked for readability alongside the token, and a missing
one means "not in-cluster" and falls back to helm. Also documents the no-limit
pagination invariant and pins it with a test.

redis.py logged through stdlib logging with extra={"team": team}, which the
structlog bridge drops on the floor — the trap notify.py documents. Switched to
a bound logger with team as a kwarg.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 01:31:11 +00:00
parent d6c1b64512
commit d64c3c9f39
7 changed files with 299 additions and 31 deletions
+40
View File
@@ -298,6 +298,46 @@ async def test_ttl_out_of_range_is_422(client: httpx.AsyncClient, token: str) ->
assert resp.status_code == 422
def _is_error_body(payload: object) -> bool:
"""The uniform error shape: a dict with `code` and `message`, and no default `detail`."""
return isinstance(payload, dict) and "code" in payload and "message" in payload
async def test_framework_404_uses_the_error_body_shape(client: httpx.AsyncClient) -> None:
"""A 404 raised by the router, not a handler, must still be ErrorBody.
Starlette raises its own HTTPException for an unknown route. The exception handler is
registered on that parent class precisely so this body is ErrorBody and not FastAPI's
default `{"detail": "Not Found"}` — one shape for every error.
"""
resp = await client.get("/v1/no-such-route")
assert resp.status_code == 404
assert _is_error_body(resp.json()), resp.text
async def test_framework_405_uses_the_error_body_shape(client: httpx.AsyncClient) -> None:
"""A wrong-method 405 comes from the router too, and must be ErrorBody."""
resp = await client.delete("/v1/instances") # collection route has no DELETE
assert resp.status_code == 405
assert _is_error_body(resp.json()), resp.text
async def test_body_validation_422_uses_the_error_body_shape(client: httpx.AsyncClient, token: str) -> None:
"""A RequestValidationError 422 must match the handler-raised 422 shape.
A forbidden extra field trips pydantic's `extra="forbid"` and raises
RequestValidationError, which the dedicated handler renders as ErrorBody rather than the
default `{"detail": [...]}`.
"""
resp = await client.post(
"/v1/instances",
headers=auth(token),
json={"service_type": "redis", "size": "small", "surprise": "field"},
)
assert resp.status_code == 422
assert _is_error_body(resp.json()), resp.text
# --------------------------------------------------------------------------- authn