docs: add USER_GUIDE.md, tighten comments, fix CLI needing a DSN
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
ci / lint (push) Successful in 33s
ci / types (push) Successful in 43s
ci / unit (push) Successful in 32s
ci / security (push) Successful in 57s
ci / dockerfile (push) Successful in 7s
ci / chart (push) Successful in 8s
ci / integration (push) Successful in 55s
ci / image (api) (push) Successful in 3m39s
ci / image (reconciler) (push) Successful in 2m53s
ci / image (worker) (push) Successful in 2m14s
ci / bump (push) Successful in 16s
The comment pass is prose-only: every distinct "why" is kept, the narration around it is not. Verified by AST-comparing each changed file against HEAD with docstrings stripped — only the two files below differ in executable code. Two real fixes fell out of the read-through: * The CLI documented itself as never touching the database, then called load_settings(), which requires SVCFORGE_PG_DSN. It refused to start without a Postgres URL it never opens. It now has its own two-field ClientSettings; the orphaned api_url/api_token are dropped from Settings, where nothing else read them. * repo/db.py had the DictRow alias comment and the ERROR_MAX_CHARS comment run together above the wrong symbol. USER_GUIDE.md is the caller-facing guide the README only gestured at: auth, catalog, every endpoint with curl, the lifecycle, the error table, rate limiting, the CLI, client generation, an end-to-end poll loop. It records two facts about the live deployment rather than documenting a flow nobody can run. SVCFORGE_JWKS_URL points at a realm with no IdP behind it, so the API logs "JWKS warm-up failed" at startup and every /v1 request is a 401. And `helm repo list` in the worker returns no repositories, so the three bitnamilegacy/ catalog entries cannot resolve at provision time; only the oci:// entries can. make lint clean, 76 unit + 111 integration tests pass.
This commit is contained in:
+24
-31
@@ -1,10 +1,9 @@
|
||||
"""The claim loop.
|
||||
|
||||
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report.
|
||||
That is the whole design, and the restraint is the point: LISTEN/NOTIFY would shave the
|
||||
latency, is fire-and-forget so it can never replace the poll anyway, is strictly extra
|
||||
code, and does not exist on pgbouncer's transaction pooler. The poll is not a placeholder
|
||||
for something better.
|
||||
Poll every 5 seconds. Claim while a semaphore slot is free. Run the handler. Report. The
|
||||
poll is not a placeholder for something better: LISTEN/NOTIFY would shave latency, but it
|
||||
is fire-and-forget so it can never replace the poll, and it does not exist on pgbouncer's
|
||||
transaction pooler.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,17 +34,16 @@ log = obs.get_logger("svcforge.worker")
|
||||
async def _report(coro: Awaitable[bool], task_id: int, what: str) -> None:
|
||||
"""Run a terminal report, and never let its failure escape.
|
||||
|
||||
Reporting is the one thing that must not kill the worker. `_run_one` runs inside a
|
||||
TaskGroup, and a TaskGroup cancels every sibling the moment one child raises — so a
|
||||
DB blip during `tasks.fail()` would abort every other in-flight provision on this pod,
|
||||
not just this one. The task itself is safe either way: it stays `running` and the
|
||||
reconciler's lease sweep returns it to the queue. Losing the report costs one lease
|
||||
interval; losing the siblings costs their work.
|
||||
`_run_one` runs inside a TaskGroup, which cancels every sibling the moment one child
|
||||
raises — so a DB blip during `tasks.fail()` would abort every other in-flight provision
|
||||
on this pod. The task itself is safe either way: it stays `running` and the lease sweep
|
||||
returns it to the queue. Losing the report costs one lease interval; losing the siblings
|
||||
costs their work.
|
||||
"""
|
||||
try:
|
||||
if not await coro:
|
||||
# The lease was stolen while we were working: another worker owns this task
|
||||
# now and is mid-run. Reporting is theirs to do, not ours.
|
||||
# The lease was stolen while we were working: another worker owns this task now
|
||||
# and is mid-run. Reporting is theirs, not ours.
|
||||
log.warning("lease lost before report; another worker owns this task", task_id=task_id)
|
||||
except Exception:
|
||||
log.exception("could not report task %s (%s); lease will expire", task_id, what)
|
||||
@@ -55,10 +53,8 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
"""Run one task to a terminal report. Never lets an exception escape the TaskGroup."""
|
||||
worker_id = deps.settings.worker_id
|
||||
try:
|
||||
# Every log line from here carries instance_id/task_id/team. Bound once, at claim,
|
||||
# rather than passed down: the alternative is threading three arguments through
|
||||
# every function that might log, and the first one anyone forgets is the one you
|
||||
# need at 3am.
|
||||
# Every log line from here carries instance_id/task_id/team. Bound once at claim
|
||||
# rather than threaded through every function that might log.
|
||||
obs.bind_task_context(task.instance_id, task.id, team=task.team or "unknown")
|
||||
log.info("task claimed", kind=task.kind.value, attempt=task.attempts)
|
||||
obs.TASKS_CLAIMED.labels(kind=task.kind.value).inc()
|
||||
@@ -72,9 +68,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
)
|
||||
return
|
||||
|
||||
# Re-parent to the span that enqueued this task. Without the stored traceparent
|
||||
# the worker's span starts a brand-new trace, and the POST that caused the work
|
||||
# is in a different trace to the helm call that did it.
|
||||
# Re-parent to the span that enqueued this task. Without the stored traceparent the
|
||||
# worker's span starts a new trace, putting the POST that caused the work in a
|
||||
# different trace from the helm call that did it.
|
||||
ctx = obs.context_from_traceparent(task.traceparent)
|
||||
started = time.monotonic()
|
||||
with obs.tracer().start_as_current_span(
|
||||
@@ -108,10 +104,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
"fail",
|
||||
)
|
||||
else:
|
||||
# Only provisions go in the provision histogram. The buckets run 10s..1800s
|
||||
# because they were sized for helm installs; a sub-second `verify` dropped
|
||||
# into the same series drags the p95 down and quietly stops
|
||||
# SvcforgeProvisionSlow from ever firing.
|
||||
# Only provisions go in the provision histogram. Its buckets run 10s..1800s
|
||||
# for helm installs, so a sub-second `verify` in the same series drags the
|
||||
# p95 down and quietly stops SvcforgeProvisionSlow from ever firing.
|
||||
if task.kind is TaskKind.PROVISION:
|
||||
obs.PROVISION_TIME.observe(time.monotonic() - started)
|
||||
await _report(deps.tasks.complete(task.id, worker_id), task.id, "complete")
|
||||
@@ -122,10 +117,9 @@ async def _run_one(deps: WorkerDeps, task: Task, sem: asyncio.Semaphore) -> None
|
||||
async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
|
||||
"""Claim and run until told to stop, then drain what is in flight.
|
||||
|
||||
Draining is what makes a rolling deploy invisible. Exiting the `async with` block
|
||||
awaits every in-flight handler, so a pod that is being replaced finishes the provision
|
||||
it already started instead of abandoning it half-done for the lease to clean up
|
||||
five minutes later.
|
||||
Draining is what makes a rolling deploy invisible: exiting the `async with` awaits every
|
||||
in-flight handler, so a pod being replaced finishes the provision it started instead of
|
||||
abandoning it for the lease to clean up five minutes later.
|
||||
"""
|
||||
sem = asyncio.Semaphore(deps.settings.worker_concurrency)
|
||||
worker_id = deps.settings.worker_id
|
||||
@@ -158,9 +152,8 @@ async def run_worker(deps: WorkerDeps, stop: asyncio.Event) -> None:
|
||||
async def _amain() -> None:
|
||||
settings: Settings = load_settings()
|
||||
|
||||
# Before anything else: nothing logged above this line is structured, and the metrics
|
||||
# the SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until the
|
||||
# registry is up.
|
||||
# Before anything else: nothing above this line logs structured, and the metrics the
|
||||
# SvcforgeTaskFailed / SvcforgeProvisionSlow alerts query do not exist until it runs.
|
||||
obs.setup("svcforge-worker", settings)
|
||||
settings.check_production()
|
||||
obs.start_metrics_server(settings.metrics_port)
|
||||
|
||||
Reference in New Issue
Block a user