"""Liveness, readiness, metrics. The distinction between the first two is the difference between a 30-second blip and a fleet-wide outage: * `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the container, so it must touch nothing external. Wired to the DB, a 20-second Postgres failover restarts every pod at once; they come back, find the DB still down, and CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the database recovered. * `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes it from the Service endpoints. It is allowed to check dependencies, and it recovers by itself the moment the check passes. """ from __future__ import annotations from typing import Any from fastapi import APIRouter, HTTPException, Request, Response, status from prometheus_client import REGISTRY from prometheus_client.exposition import choose_encoder from services.api.deps import PoolDep from services.api.models import ErrorBody router = APIRouter(tags=["ops"]) # No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each # process holds a slice of the counters. One uvicorn process per container makes the # in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup # obligation, and a class of stale-file bugs for nothing. @router.get("/healthz", status_code=status.HTTP_200_OK) async def healthz() -> dict[str, str]: """Liveness. No I/O. If the event loop can run this, the process is alive.""" return {"status": "ok"} @router.get( "/readyz", responses={503: {"model": ErrorBody, "description": "A dependency is unavailable"}}, ) async def readyz(pool: PoolDep) -> dict[str, str]: """Readiness. Postgres only. Redis is the temptation and stays out: it holds derived state that degrades gracefully, so checking it here would let an Upstash hiccup mark every pod unready, empty the Service, and turn a cache outage into a total API outage. """ try: async with pool.connection() as conn, conn.cursor() as cur: await cur.execute("select 1") row: Any = await cur.fetchone() if row is None: raise RuntimeError("select 1 returned no row") except Exception as exc: # closed pool, timeout, dead DB — all mean the same 'not ready' raise HTTPException( status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail={"code": "not_ready", "message": "database unavailable"}, ) from exc return {"status": "ready"} @router.get("/metrics", response_class=Response) async def metrics(request: Request) -> Response: """The Prometheus scrape endpoint. A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount` compiles to `^/metrics(?P/.*)$`, which does not match the bare `/metrics` every scrape config uses, and a Mount is invisible to OpenAPI. The encoding stays prometheus_client's — `choose_encoder` reads Accept and picks the exposition format with its matching content type. Hand-rolling it serves text/plain a scraper rejects. """ encoder, content_type = choose_encoder(request.headers.get("Accept", "")) return Response(content=encoder(REGISTRY), media_type=content_type)