fix: shared runtime helper must live where every image packages it
ci / lint (push) Successful in 28s
ci / types (push) Successful in 37s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 38s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 9s
ci / integration (push) Successful in 49s
ci / image (api) (push) Successful in 2m25s
ci / image (reconciler) (push) Successful in 2m45s
ci / image (worker) (push) Successful in 2m37s
ci / bump (push) Successful in 24s

services/_runtime.py crashed the worker and reconciler on boot with
`ModuleNotFoundError: No module named 'services._runtime'`, while every unit and
integration gate was green and the API rolled out fine.

The cause is packaging, not code. Each service Dockerfile copies only its own
`services/<svc>/` subdir — `services/` itself is a namespace package with no
__init__.py, so a file added at the `services/` root is never copied into any
image. Tests import from the source tree, where the file exists, so nothing
below the image boundary could catch it. The API survived only because it does
not import the helper.

Moved to svcforge_core.runtime, which `COPY libs/ libs/` packages into every
image, next to adapters/tempyaml.py for the same reason.

Added an import smoke test to the image job: `docker run --entrypoint python
<image> -c "import services.<svc>.main"` loads the whole transitive graph inside
the built image and fails the build before the digest is pushed. This is the
one check the test suite structurally cannot perform — it runs against source,
the image is a different filesystem — and it is exactly the gap this bug fell
through.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 02:40:37 +00:00
parent 9c8d10ce1f
commit e971e04d75
4 changed files with 17 additions and 2 deletions
-37
View File
@@ -1,37 +0,0 @@
"""Shared asyncio scaffolding for the long-lived services.
The worker and the reconciler are both a loop that runs until SIGTERM. They wake immediately
on shutdown rather than sleeping through it, and they install the same loop-safe signal
handlers. Both lived in each service before; keeping one copy means the shutdown behaviour
cannot drift between them.
"""
from __future__ import annotations
import asyncio
import contextlib
import signal
async def sleep_or_stop(stop: asyncio.Event, seconds: float) -> None:
"""Sleep for `seconds`, but return the instant `stop` is set.
`await asyncio.sleep(seconds)` would make every SIGTERM cost up to `seconds` of
Kubernetes waiting on terminationGracePeriod for nothing.
"""
with contextlib.suppress(TimeoutError):
await asyncio.wait_for(stop.wait(), timeout=seconds)
def install_stop_signals(stop: asyncio.Event) -> None:
"""Set `stop` on SIGTERM and SIGINT, loop-safely.
add_signal_handler, not signal.signal. signal.signal runs the handler at an arbitrary
bytecode boundary on whatever thread the C-level handler lands on, and the loop does not
notice until its next timer fires — up to a full sleep interval away. add_signal_handler
schedules the callback as an ordinary loop callback, so the sleep_or_stop above returns
at once.
"""
loop = asyncio.get_running_loop()
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, stop.set)
+1 -1
View File
@@ -34,7 +34,6 @@ from dataclasses import dataclass
import typer
from services._runtime import install_stop_signals, sleep_or_stop
from svcforge_core.adapters.clock import Clock, SystemClock
from svcforge_core.adapters.helm import HelmProvisioner, Provisioner
from svcforge_core.adapters.notify import LogNotifier, Notifier
@@ -54,6 +53,7 @@ from svcforge_core.repo.db import DictPool, make_pool
from svcforge_core.repo.instances import InstanceRepo
from svcforge_core.repo.reconcile import ReconcileRepo
from svcforge_core.repo.tasks import TaskRepo
from svcforge_core.runtime import install_stop_signals, sleep_or_stop
from svcforge_core.settings import Settings, load_settings
log = get_logger("svcforge.reconciler")
+1 -1
View File
@@ -15,7 +15,6 @@ from collections.abc import Awaitable
from opentelemetry import trace
from services._runtime import install_stop_signals, sleep_or_stop
from services.worker.deps import WorkerDeps
from services.worker.handlers import HANDLERS
from svcforge_core import obs
@@ -27,6 +26,7 @@ from svcforge_core.domain.models import Task, TaskKind
from svcforge_core.repo.db import make_pool
from svcforge_core.repo.instances import InstanceRepo
from svcforge_core.repo.tasks import TaskRepo
from svcforge_core.runtime import install_stop_signals, sleep_or_stop
from svcforge_core.settings import Settings, load_settings
log = obs.get_logger("svcforge.worker")