Nguyen Minh Phuc
c53734d2bc
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.
80 lines
3.3 KiB
Python
80 lines
3.3 KiB
Python
"""Liveness, readiness, metrics.
|
|
|
|
The distinction between the first two is the difference between a 30-second blip and a
|
|
fleet-wide outage:
|
|
|
|
* `/healthz` (liveness) answers "is this process wedged?" A failure here KILLS the
|
|
container, so it must touch nothing external. Wired to the DB, a 20-second Postgres
|
|
failover restarts every pod at once; they come back, find the DB still down, and
|
|
CrashLoopBackOff with exponential delays — the fleet stays down for minutes after the
|
|
database recovered.
|
|
* `/readyz` (readiness) answers "should this pod get traffic?" A failure here only removes
|
|
it from the Service endpoints. It is allowed to check dependencies, and it recovers by
|
|
itself the moment the check passes.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from fastapi import APIRouter, HTTPException, Request, Response, status
|
|
from prometheus_client import REGISTRY
|
|
from prometheus_client.exposition import choose_encoder
|
|
|
|
from services.api.deps import PoolDep
|
|
from services.api.models import ErrorBody
|
|
|
|
router = APIRouter(tags=["ops"])
|
|
|
|
# No PROMETHEUS_MULTIPROC_DIR, deliberately: it exists for prefork servers where each
|
|
# process holds a slice of the counters. One uvicorn process per container makes the
|
|
# in-process registry correct, and multiproc mode would add a shared temp dir, a cleanup
|
|
# obligation, and a class of stale-file bugs for nothing.
|
|
|
|
|
|
@router.get("/healthz", status_code=status.HTTP_200_OK)
|
|
async def healthz() -> dict[str, str]:
|
|
"""Liveness. No I/O. If the event loop can run this, the process is alive."""
|
|
return {"status": "ok"}
|
|
|
|
|
|
@router.get(
|
|
"/readyz",
|
|
responses={503: {"model": ErrorBody, "description": "A dependency is unavailable"}},
|
|
)
|
|
async def readyz(pool: PoolDep) -> dict[str, str]:
|
|
"""Readiness. Postgres only.
|
|
|
|
Redis is the temptation and stays out: it holds derived state that degrades gracefully,
|
|
so checking it here would let an Upstash hiccup mark every pod unready, empty the
|
|
Service, and turn a cache outage into a total API outage.
|
|
"""
|
|
try:
|
|
async with pool.connection() as conn, conn.cursor() as cur:
|
|
await cur.execute("select 1")
|
|
row: Any = await cur.fetchone()
|
|
if row is None:
|
|
raise RuntimeError("select 1 returned no row")
|
|
except Exception as exc: # closed pool, timeout, dead DB — all mean the same 'not ready'
|
|
raise HTTPException(
|
|
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
|
detail={"code": "not_ready", "message": "database unavailable"},
|
|
) from exc
|
|
return {"status": "ready"}
|
|
|
|
|
|
@router.get("/metrics", response_class=Response)
|
|
async def metrics(request: Request) -> Response:
|
|
"""The Prometheus scrape endpoint.
|
|
|
|
A route rather than `app.mount("/metrics", make_asgi_app())`: a Starlette `Mount`
|
|
compiles to `^/metrics(?P<path>/.*)$`, which does not match the bare `/metrics` every
|
|
scrape config uses, and a Mount is invisible to OpenAPI.
|
|
|
|
The encoding stays prometheus_client's — `choose_encoder` reads Accept and picks the
|
|
exposition format with its matching content type. Hand-rolling it serves text/plain a
|
|
scraper rejects.
|
|
"""
|
|
encoder, content_type = choose_encoder(request.headers.get("Accept", ""))
|
|
return Response(content=encoder(REGISTRY), media_type=content_type)
|