reconciler: scope the drift check to svcforge's own releases
ci / lint (push) Failing after 10s
ci / types (push) Has been skipped
ci / unit (push) Has been skipped
ci / integration (push) Has been skipped
ci / security (push) Has been skipped
ci / dockerfile (push) Has been skipped
ci / chart (push) Has been skipped
ci / image (api) (push) Has been skipped
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / bump (push) Has been skipped

check_drift diffs helm against the database in both directions, and the second
one is `live - known` -> drift.orphan_release at ERROR. `live` was every
release in the cluster, so argocd, longhorn, gitea and cert-manager were all
reported as orphans svcforge is failing to account for, on every sweep. They
are not orphans; they were never svcforge's to know about.

install() now stamps app.kubernetes.io/managed-by=svcforge and list_releases()
selects on it. Releases provisioned before this see one sweep as missing and
get re-provisioned, which is safe by construction — `helm upgrade --install`
against a deterministic release name — and that re-provision applies the label.

This does NOT fix the timeout, and the docstring says so. Measured, not
assumed: unscoped 23 releases in 4392ms, scoped to 0 in 3988ms. `--selector` is
not pushed down as a server-side selector, so helm still fetches and
decompresses every release secret and filters what it already parsed. Around
10%, not the order of magnitude the flag's shape suggests. The 330s timeouts
need CPU for the container or a different read path.

The two sides of the label are asserted against each other rather than a
literal, so a rename that updates only one fails in tests instead of in
production as a silently empty drift check. Control-tested: renaming the read
side alone fails two of the three.
This commit is contained in:
Nguyen Minh Phuc
2026-07-20 12:53:16 +00:00
parent c2a27952d1
commit 2356ac4ef3
2 changed files with 145 additions and 2 deletions
@@ -45,6 +45,16 @@ _RUN_TIMEOUT_MARGIN_S = 30
_STDERR_TAIL_BYTES = 2048
# The label every release svcforge provisions carries, and the only thing that lets the
# reconciler tell its own releases from the rest of the cluster's. Written by install(),
# read by list_releases(). Changing either value without the other silently empties the
# reconciler's view of reality, which reads as "no drift" rather than as an error.
#
# `app.kubernetes.io/managed-by` is the standard key for exactly this, so anyone reading
# the cluster with kubectl gets the same answer the reconciler does.
MANAGED_BY_LABEL = "app.kubernetes.io/managed-by"
MANAGED_BY_VALUE = "svcforge"
class HelmError(SvcforgeError, RuntimeError):
"""Non-zero exit. str(self) is the stderr tail that lands in instances.error.
@@ -206,6 +216,12 @@ class HelmProvisioner:
# and one fewer set of vendored Go CVEs to track. Idempotent: existing
# namespaces are left alone.
"--create-namespace",
# Stamps MANAGED_BY_LABEL onto the release, which is what makes
# list_releases() able to ask for svcforge's releases and nobody else's.
# Without it the reconciler has to list every release in the cluster and
# sort out ownership afterwards, which it cannot actually do.
"--labels",
f"{MANAGED_BY_LABEL}={MANAGED_BY_VALUE}",
"--version",
entry.chart_version,
"--values",
@@ -232,8 +248,42 @@ class HelmProvisioner:
await self._run_helm(argv)
async def list_releases(self) -> list[ReleaseInfo]:
"""Every release helm knows about, in every namespace. The reconciler's view of reality."""
argv = self._base_argv("list", "--all-namespaces", "--output", "json")
"""Every release SVCFORGE provisioned, in every namespace. The reconciler's view of reality.
Scoped by label, and the scope is load-bearing twice over.
Correctness first. The reconciler diffs this against the database in both directions,
and the second direction is `live - known` -> `drift.orphan_release` at ERROR.
Unscoped, `live` is every release in the cluster, so argocd, longhorn, gitea and
cert-manager are all reported as orphans svcforge is failing to account for, every
sweep. They are not orphans. They were never svcforge's to know about.
What this does NOT fix is cost, and that was measured rather than assumed:
unscoped 23 releases in 4392ms, scoped to 0 releases in 3988ms. `--selector` is not
pushed down as a server-side label selector — helm fetches and decompresses every
release secret in the cluster regardless, then filters what it already parsed. The
saving is around 10%, not the order of magnitude the shape of the flag suggests.
That matters because this call was timing out at 330s on every tick, with the CPU
request mutated to 0 by a cluster policy, against ~4s given real CPU. Scoping does
not rescue that. A check that never completes reports no drift, which looks exactly
like no drift existing, so the timeout needs its own fix — CPU for the container, or
reading the release secrets directly with a label selector, which the API server
answers in 141ms because it never decompresses anything.
Releases provisioned before the label existed will not match, so the first sweep
after this ships sees them as missing and re-provisions. That is safe by
construction — provisioning is `helm upgrade --install` against a deterministic
release name — and the re-provision is what applies the label.
"""
argv = self._base_argv(
"list",
"--all-namespaces",
"--selector",
f"{MANAGED_BY_LABEL}={MANAGED_BY_VALUE}",
"--output",
"json",
)
raw = await self._run_helm(argv)
try:
parsed: Any = json.loads(raw or "[]")