fix: shared runtime helper must live where every image packages it
ci / lint (push) Successful in 28s
ci / types (push) Successful in 37s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 38s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 9s
ci / integration (push) Successful in 49s
ci / image (api) (push) Successful in 2m25s
ci / image (reconciler) (push) Successful in 2m45s
ci / image (worker) (push) Successful in 2m37s
ci / bump (push) Successful in 24s
ci / lint (push) Successful in 28s
ci / types (push) Successful in 37s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 38s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 9s
ci / integration (push) Successful in 49s
ci / image (api) (push) Successful in 2m25s
ci / image (reconciler) (push) Successful in 2m45s
ci / image (worker) (push) Successful in 2m37s
ci / bump (push) Successful in 24s
services/_runtime.py crashed the worker and reconciler on boot with `ModuleNotFoundError: No module named 'services._runtime'`, while every unit and integration gate was green and the API rolled out fine. The cause is packaging, not code. Each service Dockerfile copies only its own `services/<svc>/` subdir — `services/` itself is a namespace package with no __init__.py, so a file added at the `services/` root is never copied into any image. Tests import from the source tree, where the file exists, so nothing below the image boundary could catch it. The API survived only because it does not import the helper. Moved to svcforge_core.runtime, which `COPY libs/ libs/` packages into every image, next to adapters/tempyaml.py for the same reason. Added an import smoke test to the image job: `docker run --entrypoint python <image> -c "import services.<svc>.main"` loads the whole transitive graph inside the built image and fails the build before the digest is pushed. This is the one check the test suite structurally cannot perform — it runs against source, the image is a different filesystem — and it is exactly the gap this bug fell through.
This commit is contained in:
@@ -1,37 +0,0 @@
|
||||
"""Shared asyncio scaffolding for the long-lived services.
|
||||
|
||||
The worker and the reconciler are both a loop that runs until SIGTERM. They wake immediately
|
||||
on shutdown rather than sleeping through it, and they install the same loop-safe signal
|
||||
handlers. Both lived in each service before; keeping one copy means the shutdown behaviour
|
||||
cannot drift between them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import signal
|
||||
|
||||
|
||||
async def sleep_or_stop(stop: asyncio.Event, seconds: float) -> None:
|
||||
"""Sleep for `seconds`, but return the instant `stop` is set.
|
||||
|
||||
`await asyncio.sleep(seconds)` would make every SIGTERM cost up to `seconds` of
|
||||
Kubernetes waiting on terminationGracePeriod for nothing.
|
||||
"""
|
||||
with contextlib.suppress(TimeoutError):
|
||||
await asyncio.wait_for(stop.wait(), timeout=seconds)
|
||||
|
||||
|
||||
def install_stop_signals(stop: asyncio.Event) -> None:
|
||||
"""Set `stop` on SIGTERM and SIGINT, loop-safely.
|
||||
|
||||
add_signal_handler, not signal.signal. signal.signal runs the handler at an arbitrary
|
||||
bytecode boundary on whatever thread the C-level handler lands on, and the loop does not
|
||||
notice until its next timer fires — up to a full sleep interval away. add_signal_handler
|
||||
schedules the callback as an ordinary loop callback, so the sleep_or_stop above returns
|
||||
at once.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
loop.add_signal_handler(sig, stop.set)
|
||||
@@ -34,7 +34,6 @@ from dataclasses import dataclass
|
||||
|
||||
import typer
|
||||
|
||||
from services._runtime import install_stop_signals, sleep_or_stop
|
||||
from svcforge_core.adapters.clock import Clock, SystemClock
|
||||
from svcforge_core.adapters.helm import HelmProvisioner, Provisioner
|
||||
from svcforge_core.adapters.notify import LogNotifier, Notifier
|
||||
@@ -54,6 +53,7 @@ from svcforge_core.repo.db import DictPool, make_pool
|
||||
from svcforge_core.repo.instances import InstanceRepo
|
||||
from svcforge_core.repo.reconcile import ReconcileRepo
|
||||
from svcforge_core.repo.tasks import TaskRepo
|
||||
from svcforge_core.runtime import install_stop_signals, sleep_or_stop
|
||||
from svcforge_core.settings import Settings, load_settings
|
||||
|
||||
log = get_logger("svcforge.reconciler")
|
||||
|
||||
@@ -15,7 +15,6 @@ from collections.abc import Awaitable
|
||||
|
||||
from opentelemetry import trace
|
||||
|
||||
from services._runtime import install_stop_signals, sleep_or_stop
|
||||
from services.worker.deps import WorkerDeps
|
||||
from services.worker.handlers import HANDLERS
|
||||
from svcforge_core import obs
|
||||
@@ -27,6 +26,7 @@ from svcforge_core.domain.models import Task, TaskKind
|
||||
from svcforge_core.repo.db import make_pool
|
||||
from svcforge_core.repo.instances import InstanceRepo
|
||||
from svcforge_core.repo.tasks import TaskRepo
|
||||
from svcforge_core.runtime import install_stop_signals, sleep_or_stop
|
||||
from svcforge_core.settings import Settings, load_settings
|
||||
|
||||
log = obs.get_logger("svcforge.worker")
|
||||
|
||||
Reference in New Issue
Block a user