58ffb9c2e0
ci / lint (push) Successful in 26s
ci / types (push) Successful in 34s
ci / unit (push) Successful in 26s
ci / security (push) Successful in 38s
ci / dockerfile (push) Successful in 5s
ci / integration (push) Successful in 42s
ci / image (reconciler) (push) Has been skipped
ci / image (worker) (push) Has been skipped
ci / chart (push) Successful in 9s
ci / image (api) (push) Successful in 2m13s
ci / bump (push) Has been skipped
Run #68 re-pulled trivy in all three matrix legs — 178MB each, ~7s each — because the reclaim step deletes it at the end of every leg: Unable to find image 'aquasec/trivy:0.72.0@sha256:cffe...' locally untagged: aquasec/trivy@sha256:cffe... Total reclaimed space: 178.6MB The `--filter until=168h` I added for exactly this reason does not work. That filter reads the image's CREATED timestamp, not the pull time, and aquasec/trivy:0.72.0 was built months ago, so it matches immediately. The comment claiming the age filter fixed this was wrong; corrected in place. Pull by digest, tag it, run the tag. A tagged image is not dangling, which is what actually takes it out of `docker image prune`'s scope. The digest is still the pin — enforced at pull time. Everything else in run #68 is clean: 76 unit (98.59% coverage), 112 integration, 18 CACHED layers per image job, zero uv hardlink warnings, and the trivy DB volume hit on 2 of 3 legs (one download, two silent).
564 lines
29 KiB
YAML
564 lines
29 KiB
YAML
# svcforge CI.
|
|
#
|
|
# The contract, in one line: merge to master -> three images built and scanned -> the
|
|
# chart's image digests bumped -> ArgoCD syncs. CI never touches the cluster. There is no
|
|
# kubeconfig here and there must never be one; the pipeline's last act is a git commit.
|
|
#
|
|
# Rules this file exists to enforce:
|
|
# - Build once, promote the artifact. The digest that trivy scanned is the digest that
|
|
# lands in values.yaml is the digest that runs.
|
|
# - Deploy by digest, never a mutable tag.
|
|
# - Everything pinned: actions by SHA, tool images by digest, deps by uv.lock + --frozen.
|
|
# - Every gate required. None advisory. Fail the PR, not prod.
|
|
#
|
|
# Stage order is deliberate and matches the module: cheapest and most likely to fail first,
|
|
# so a formatting mistake costs 20 seconds instead of three minutes of image builds.
|
|
|
|
name: ci
|
|
|
|
on: [pull_request, push]
|
|
|
|
concurrency:
|
|
# A second push to the same branch makes the first run's answer irrelevant. Cancel it —
|
|
# except on master, where the run ends in a commit and must not be interrupted midway.
|
|
group: ci-${{ github.ref }}
|
|
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
|
|
|
|
env:
|
|
REGISTRY: gitea.oci-oci.duckdns.org
|
|
IMAGE_NS: gitea_admin
|
|
UV_VERSION: "0.11.29"
|
|
# Every uv job logged this, five times a run:
|
|
#
|
|
# warning: Failed to hardlink files; falling back to full copy. This may lead to
|
|
# degraded performance.
|
|
#
|
|
# The uv cache (the runner's cache volume) and the target venv are on different
|
|
# filesystems here, so hardlinking cannot work and uv copies 86 packages anyway. Saying
|
|
# `copy` up front does not make it slower — it is already copying — it just stops the
|
|
# warning from being noise that trains people to skim CI logs. The Dockerfiles set the
|
|
# same variable for the same reason.
|
|
UV_LINK_MODE: copy
|
|
GITLEAKS_VERSION: "8.30.1"
|
|
GITLEAKS_SHA256: "e4a487ee7ccd7d3a7f7ec08657610aa3606637dab924210b3aee62570fb4b080"
|
|
|
|
jobs:
|
|
# --- stage 1: lint -- fast, fails first ---------------------------------------------
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
# The uv store is keyed on the lockfile: same lock, same wheels, cache hit.
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: ruff
|
|
run: uv run ruff check . && uv run ruff format --check .
|
|
|
|
# --- stage 2: types -- your compiler ------------------------------------------------
|
|
types:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: mypy --strict
|
|
run: uv run mypy --strict .
|
|
|
|
# --- stage 3: unit -- domain only, milliseconds, coverage gate -----------------------
|
|
unit:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: pytest unit
|
|
# The gate is on domain/ alone, and only domain/. It is pure, has no I/O, and needs
|
|
# no mocks — there is no excuse for a gap there. Pointing this at the whole repo
|
|
# would let untested SQL be paid for by well-tested pure functions.
|
|
#
|
|
# `--cov=svcforge_core.domain` — the MODULE, not a path. `--cov=libs/svcforge_core/domain`
|
|
# is a path that does not exist (the package nests one level deeper, at
|
|
# libs/svcforge_core/svcforge_core/domain), so coverage measured nothing and reported
|
|
# 0.00%. A path-based --cov that misses silently reports 0 rather than erroring, so
|
|
# without a --cov-fail-under this reads as a passing coverage gate over no code at all.
|
|
run: uv run pytest tests/unit --cov=svcforge_core.domain --cov-fail-under=90
|
|
|
|
# --- stages 4+5: migrate, then integration against that schema -----------------------
|
|
integration:
|
|
runs-on: ubuntu-latest
|
|
needs: [unit]
|
|
services:
|
|
postgres:
|
|
image: postgres:18@sha256:32ca0af8e77bfb8c6610c488e4691f83f972a3e9e64d3b02facf3ab111ad5500
|
|
env:
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: svcforge
|
|
ports:
|
|
- 5432:5432
|
|
options: >-
|
|
--health-cmd "pg_isready -U postgres"
|
|
--health-interval 5s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
env:
|
|
# A scratch Postgres, so a literal password is correct here: it is not a secret, it
|
|
# is a fixture. Real DSNs live in Vault and reach the pods via external-secrets.
|
|
SVCFORGE_PG_DSN: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
SVCFORGE_PG_DSN_SESSION: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
# What tests/integration/conftest.py actually reads. Without it the fixture falls
|
|
# back to testcontainers and starts a SECOND Postgres inside the runner's docker,
|
|
# while the service container above sits unused — slower, and a different database
|
|
# to the one `migrate` just ran against.
|
|
SVCFORGE_TEST_DSN: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: migrate
|
|
# The same entrypoint the chart's pre-upgrade hook runs. If migrations only ever
|
|
# ran under testcontainers, CI would be testing a code path production never takes.
|
|
run: uv run python -m svcforge_core.migrate
|
|
- name: pytest integration
|
|
run: uv run pytest tests/integration
|
|
|
|
# --- stages 6+7+8: SAST, secrets, dependency CVEs ------------------------------------
|
|
# One job, three independent gates. They share a checkout and nothing else; each `run`
|
|
# step fails the job on its own.
|
|
security:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
with:
|
|
# gitleaks scans history, not just the tip. A secret committed and then reverted
|
|
# is still a leaked secret, and a shallow clone cannot see it.
|
|
fetch-depth: 0
|
|
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
|
|
- name: bandit (SAST)
|
|
# `--with`, not a dev dependency: bandit is a CI tool, not something the project
|
|
# imports, and ruff's S ruleset already runs its checks in the lint stage. This is
|
|
# the belt to that suspenders — -ll reports medium severity and above only.
|
|
#
|
|
# `bandit[toml]` + `-c pyproject.toml`: without the toml extra bandit cannot read
|
|
# its own config and silently ignores it, which looks identical to a clean run.
|
|
run: uv run --with 'bandit[toml]' bandit -c pyproject.toml -r libs services -ll
|
|
|
|
- name: gitleaks (secret scan)
|
|
# A downloaded binary, not `docker run -v "$PWD:/repo"`.
|
|
#
|
|
# The container form was scanning NOTHING and passing. `-v "$PWD:/repo"` is
|
|
# interpreted by the dind sidecar's daemon, which cannot see the job container's
|
|
# checkout, so gitleaks got an empty mount and logged:
|
|
#
|
|
# ERR [git] fatal: not a git repository (or any parent up to mount point /)
|
|
# ERR failed to scan Git repository error="stderr is not empty"
|
|
# INF scan completed in 35.2ms
|
|
# INF no leaks found <- exit 0
|
|
#
|
|
# It reported success without looking, which is worse than having no scanner: the
|
|
# gate was green and meaningless. The 35ms runtime was the tell — a real history
|
|
# scan of this repo takes seconds.
|
|
#
|
|
# The `--log-opts` + commit-count assertion below is the guard against that class
|
|
# of failure returning. A scanner that cannot fail is not a gate.
|
|
run: |
|
|
set -euo pipefail
|
|
curl -fsSL -o /tmp/gitleaks.tgz \
|
|
"https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_arm64.tar.gz"
|
|
echo "${GITLEAKS_SHA256} /tmp/gitleaks.tgz" | sha256sum -c -
|
|
tar -xzf /tmp/gitleaks.tgz -C /tmp gitleaks
|
|
install -m 0755 /tmp/gitleaks /usr/local/bin/gitleaks
|
|
|
|
# Prove there is a repository to scan before trusting the verdict.
|
|
commits=$(git rev-list --count HEAD)
|
|
echo "scanning $commits commits"
|
|
test "$commits" -gt 0
|
|
|
|
gitleaks detect --no-banner --source . --redact --exit-code 1
|
|
|
|
- name: pip-audit (dependency CVEs)
|
|
# --strict fails on an audit error rather than shrugging and reporting clean.
|
|
#
|
|
# Audits the LOCKED dependency set, not the installed environment. Auditing the env
|
|
# means auditing `svcforge` and `svcforge-core` too, which are ours, are installed
|
|
# editable, and are not on PyPI — under --strict that is a hard error ("distribution
|
|
# marked as editable"), so the choice was to drop --strict or to stop asking PyPI
|
|
# about packages it has never heard of. This asks about the 56 that actually came
|
|
# from PyPI, and keeps --strict.
|
|
run: |
|
|
uv export --frozen --no-dev \
|
|
--no-emit-project --no-emit-package svcforge-core \
|
|
-o /tmp/requirements-audit.txt
|
|
uv run --with pip-audit pip-audit --strict -r /tmp/requirements-audit.txt
|
|
|
|
# --- stage 9: hadolint ---------------------------------------------------------------
|
|
dockerfile:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- name: hadolint
|
|
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
|
with:
|
|
recursive: true
|
|
dockerfile: "services/*/Dockerfile"
|
|
failure-threshold: warning
|
|
|
|
# --- stage 9b: the chart must render --------------------------------------------------
|
|
# Without this, a chart that does not template reaches ArgoCD and fails in the cluster,
|
|
# where the error surfaces as a sync failure with no PR attached to it. `helm template`
|
|
# is the real gate: it is what ArgoCD does, and _helpers.tpl's image helper calls `fail`
|
|
# on anything that is not a full sha256 digest.
|
|
chart:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
env:
|
|
# The same helm version the worker and reconciler images carry, so CI renders with
|
|
# the version that ships.
|
|
HELM_VERSION: "3.21.3"
|
|
# sha256 of helm-v3.21.3-linux-arm64.tar.gz, from https://get.helm.sh/*.sha256sum.
|
|
# Pinned for the same reason every image here is pinned by digest: a tarball fetched
|
|
# over HTTPS is still a tarball whoever controls the bucket chose to serve.
|
|
HELM_SHA256: "67f58155079ff9ffab98ba5c88daff0ed9b542f3a4732f5dd426dde7dd0f5244"
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
|
|
- name: install helm
|
|
# A downloaded binary rather than `docker run alpine/helm`. The container form
|
|
# cannot see the checkout: `docker run -v "$PWD:/repo"` is interpreted by the dind
|
|
# SIDECAR's daemon, and the chart path resolved to nothing inside it —
|
|
# "stat deploy/chart/Chart.yaml: no such file or directory" while the file plainly
|
|
# exists in the job. A binary on PATH has no such boundary to cross.
|
|
run: |
|
|
set -euo pipefail
|
|
curl -fsSL -o /tmp/helm.tgz \
|
|
"https://get.helm.sh/helm-v${HELM_VERSION}-linux-arm64.tar.gz"
|
|
echo "${HELM_SHA256} /tmp/helm.tgz" | sha256sum -c -
|
|
tar -xzf /tmp/helm.tgz -C /tmp
|
|
install -m 0755 /tmp/linux-arm64/helm /usr/local/bin/helm
|
|
helm version --short
|
|
|
|
- name: helm lint
|
|
run: |
|
|
helm lint deploy/chart
|
|
|
|
- name: helm template (the digest guard rejects bad digests)
|
|
# Feed the guard bad digests explicitly with --set. The earlier version of this
|
|
# step ran a bare `helm template` and asserted it FAILED, on the assumption that
|
|
# values.yaml always holds all-zeros placeholders. That assumption dies the first
|
|
# time the `bump` job runs: bump commits real digests into values.yaml, so the bare
|
|
# render then succeeds and the assertion reports "the guard is not guarding" about
|
|
# a guard that is fine. A test whose expected result flips depending on whether CI
|
|
# has run before is not a test.
|
|
#
|
|
# Each case below is a distinct way to get a digest wrong, and each must be
|
|
# rejected on its own.
|
|
run: |
|
|
set -euo pipefail
|
|
ZEROS="sha256:$(printf '0%.0s' $(seq 64))"
|
|
for bad_desc in \
|
|
"all-zeros placeholder|${ZEROS}" \
|
|
"not a digest at all|latest" \
|
|
"right prefix, wrong length|sha256:abc123" \
|
|
"empty|"; do
|
|
desc="${bad_desc%%|*}"; bad="${bad_desc#*|}"
|
|
if helm template svcforge deploy/chart \
|
|
--set image.api.digest="${bad}" >/dev/null 2>&1; then
|
|
echo "FAIL: the chart rendered with a ${desc} digest (${bad@Q})." >&2
|
|
echo "The digest guard in _helpers.tpl is not guarding." >&2
|
|
exit 1
|
|
fi
|
|
echo "ok: rejected ${desc}"
|
|
done
|
|
|
|
- name: helm template (renders with real digests)
|
|
# Dummy but well-formed digests: this checks the templates themselves render, with
|
|
# the two values-gated monitoring blocks explicitly on so they are covered too.
|
|
run: |
|
|
set -euo pipefail
|
|
A="sha256:$(printf 'a%.0s' $(seq 64))"
|
|
B="sha256:$(printf 'b%.0s' $(seq 64))"
|
|
C="sha256:$(printf 'c%.0s' $(seq 64))"
|
|
helm template svcforge deploy/chart \
|
|
--set image.api.digest="$A" \
|
|
--set image.worker.digest="$B" \
|
|
--set image.reconciler.digest="$C" \
|
|
--set serviceMonitor.enabled=true \
|
|
--set prometheusRule.enabled=true \
|
|
>/dev/null
|
|
echo "ok: chart renders"
|
|
|
|
# --- stage 10: build -> trivy -> push by digest --------------------------------------
|
|
image:
|
|
runs-on: ubuntu-latest
|
|
# Every gate above is required. An image is not built until all of them are green,
|
|
# which is what makes "the digest CI pushed is a digest that passed everything" true.
|
|
needs: [types, unit, integration, security, dockerfile, chart]
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
# Kept for correctness on runners that honour it, but do NOT rely on it here:
|
|
# act_runner IGNORES strategy.max-parallel. Measured in run #14 with this set to 1 —
|
|
# the api leg ran 04:23:03-04:24:04, then the worker leg started 04:25:47 while
|
|
# reconciler was still building. Two concurrent, exactly what it was meant to prevent.
|
|
#
|
|
# The lever that actually binds is the runner's own `capacity`, set in oci-k8s
|
|
# (k8s/roles/addons/tasks/main.yml) and now 1. Contention there is what produced both
|
|
# `DeadlineExceeded: no active session` in the build and `Failed to connect to
|
|
# gitea-http:3000` in checkout.
|
|
max-parallel: 1
|
|
matrix:
|
|
svc: [api, worker, reconciler]
|
|
# Deliberately no `outputs:` here. Matrix legs share one outputs map and clobber each
|
|
# other — the merge is not per-key and not ordered, so two of the three digests would
|
|
# arrive empty or stale, intermittently. The bump job resolves the digests from the
|
|
# registry instead, which is a read, not a rebuild.
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
|
|
|
- name: registry login
|
|
# Not gated to master any more: the build step now reads AND writes the layer cache
|
|
# in the registry, so every run needs credentials. Pushing the release image is
|
|
# still master-only — that gate lives on the `push by digest` step, where it belongs.
|
|
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
|
with:
|
|
registry: ${{ env.REGISTRY }}
|
|
username: ${{ secrets.REGISTRY_USER }}
|
|
password: ${{ secrets.REGISTRY_TOKEN }}
|
|
|
|
- name: build
|
|
# Context is the repo root and the Dockerfile is addressed with -f. It cannot be
|
|
# otherwise: the image needs pyproject.toml, uv.lock and libs/, all of which live
|
|
# above services/<svc>/, and COPY ../.. is illegal.
|
|
#
|
|
# Loaded locally, not pushed. Trivy scans this exact image next; only then does it
|
|
# get pushed. The alternative — push, scan, and hope nobody pulled meanwhile — is
|
|
# how a CRITICAL ends up in the registry with a green checkmark next to it.
|
|
#
|
|
# The layer cache is `type=registry`, NOT `type=gha`. Two reasons, both hard:
|
|
#
|
|
# 1. act_runner's cache server is backed by a 1Gi PVC that also holds `.runner`,
|
|
# the runner's own registration file. `mode=max` stores every intermediate layer
|
|
# of three images — several GB. Filling that volume does not merely lose the
|
|
# cache: the runner cannot write its state and has to be re-registered by hand.
|
|
# Trading "slow CI" for "broken CI" is not a trade.
|
|
# 2. act_runner evicts by AGE, with no size cap in its config. It will fill whatever
|
|
# it is given and then wedge. The registry has no such limit and already holds
|
|
# the images anyway.
|
|
#
|
|
# The uv/pip cache still uses the runner's cache service — that one is a few hundred
|
|
# MB and fits.
|
|
run: |
|
|
docker buildx build \
|
|
-f services/${{ matrix.svc }}/Dockerfile \
|
|
--build-arg BUILD_SHA=${{ github.sha }} \
|
|
--cache-from type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache \
|
|
--cache-to type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache,mode=max \
|
|
--load \
|
|
-t svcforge/${{ matrix.svc }}:ci \
|
|
.
|
|
|
|
- name: import smoke test
|
|
# Import the service's entrypoint module INSIDE the built image, which every unit and
|
|
# integration test that passes cannot do: they import from the source tree, where
|
|
# every file exists. The image is a different filesystem — each Dockerfile copies only
|
|
# its own `services/<svc>/`, so a shared module added at the `services/` root, or any
|
|
# dependency the Dockerfile forgets, is present in the tests and absent in the image.
|
|
#
|
|
# That gap shipped a reconciler that crashed on boot with
|
|
# `ModuleNotFoundError: No module named 'services._runtime'` while every gate was
|
|
# green. Importing main here loads the whole transitive graph and fails the build
|
|
# before the digest is pushed, instead of after ArgoCD has rolled it out.
|
|
run: |
|
|
docker run --rm --entrypoint python svcforge/${{ matrix.svc }}:ci \
|
|
-c "import services.${{ matrix.svc }}.main"
|
|
|
|
- name: trivy
|
|
# Run trivy directly rather than via aquasecurity/trivy-action, for the same reason
|
|
# gitleaks is run directly above: the command is the documented one, pinned by
|
|
# digest, with no nested action resolution.
|
|
#
|
|
# It is also the only thing that works here. trivy-action internally does
|
|
# `uses: aquasecurity/setup-trivy@v0.2.2`, and that tag no longer exists upstream —
|
|
# the earliest published tag today is v0.2.6. The runner clones it and fails with
|
|
# "Unable to resolve v0.2.2: reference not found". A third-party action pinned by
|
|
# SHA still resolves ITS OWN dependencies by mutable tag, so pinning the outer
|
|
# action bought nothing.
|
|
#
|
|
# --ignore-unfixed: a CVE with no fix available is not something this PR can act
|
|
# on, and failing on it only teaches people to add ignore entries. Rebuilding on a
|
|
# new base image picks the fix up the day it exists.
|
|
#
|
|
# The vuln DB cache is a NAMED VOLUME, not `-v "$PWD/.trivycache:..."`.
|
|
#
|
|
# The bind form was the same bug that made gitleaks scan nothing: `$PWD` is a path
|
|
# in the job container, but the -v is resolved by the daemon in the dind sidecar,
|
|
# which has no such directory and silently creates an empty one. Every run then
|
|
# logged `[vulndb] Need to update DB` and spent ~35s re-downloading it, and the
|
|
# cache it wrote went into a throwaway directory inside dind.
|
|
#
|
|
# A named volume lives in the dind daemon's own storage, which is the one thing on
|
|
# this runner both sides agree on. It survives across matrix legs and across runs.
|
|
#
|
|
# Measured, not assumed: cold pass logs the three download lines and leaves 1.1G
|
|
# in the volume; warm pass logs nothing. 1.1G is far more than the 50MB an older
|
|
# comment here claimed, so the retention story matters. Trivy replaces the DB in
|
|
# place rather than accumulating versions, so the volume stays at roughly one DB,
|
|
# and the weekly prune CronJob reclaims it at the cost of one re-download.
|
|
env:
|
|
TRIVY: aquasec/trivy:0.72.0@sha256:cffe3f5161a47a6823fbd23d985795b3ed72a4c806da4c4df16266c02accdd6f
|
|
run: |
|
|
# Pull by digest, then give it a local tag, and run the TAG.
|
|
#
|
|
# An image pulled by digest carries no tag, which makes it dangling the moment its
|
|
# container exits — so the reclaim step below deleted it at the end of every matrix
|
|
# leg and the next leg paid a 178MB re-pull. Three legs, three pulls, every run.
|
|
#
|
|
# The `until=168h` filter there does not save it. That filter reads the image's
|
|
# CREATED timestamp, not when it was pulled, and this image was built months ago,
|
|
# so it matches the age filter immediately. A tag is what actually takes an image
|
|
# out of `docker image prune`'s scope. Verified in run #68: "Unable to find image
|
|
# ... locally" in all three legs, and "untagged: aquasec/trivy@sha256:cffe..." in
|
|
# each prune.
|
|
#
|
|
# The digest is still the pin — it is enforced here, at pull time. `trivy:pinned`
|
|
# is a local alias for an image whose content was already verified.
|
|
docker pull "${TRIVY}"
|
|
docker tag "${TRIVY}" trivy:pinned
|
|
docker run --rm \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v svcforge-trivy-db:/root/.cache/trivy \
|
|
trivy:pinned \
|
|
image \
|
|
--severity HIGH,CRITICAL \
|
|
--ignore-unfixed \
|
|
--exit-code 1 \
|
|
--format table \
|
|
--no-progress \
|
|
svcforge/${{ matrix.svc }}:ci
|
|
|
|
- name: push by digest
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
# Re-running buildx here is a cache hit on every layer, not a second build: the
|
|
# image is byte-identical to the one trivy just cleared. buildx cannot --load and
|
|
# --push in one invocation, which is the only reason this step exists.
|
|
#
|
|
# The commit-SHA tag is a handle for the bump job to resolve, not something anything
|
|
# deploys. What deploys is the digest that tag resolves to.
|
|
run: |
|
|
set -euo pipefail
|
|
IMAGE="${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}"
|
|
docker buildx build \
|
|
-f services/${{ matrix.svc }}/Dockerfile \
|
|
--build-arg BUILD_SHA=${{ github.sha }} \
|
|
--cache-from type=registry,ref=${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}:buildcache \
|
|
--push \
|
|
-t "${IMAGE}:${GITHUB_SHA}" \
|
|
.
|
|
docker buildx imagetools inspect "${IMAGE}:${GITHUB_SHA}" \
|
|
--format '{{.Manifest.Digest}}'
|
|
|
|
- name: reclaim dind disk
|
|
# dind's /var/lib/docker is a hostPath on node2 (see oci-k8s addons role), so
|
|
# nothing reclaims it automatically — kubelet's image GC does not manage a nested
|
|
# daemon's store. Left alone it grows every run until node2 hits disk pressure and
|
|
# starts evicting pods, which looks like a cluster problem rather than a CI one.
|
|
#
|
|
# `always()`: a failed build still leaves layers behind, and that is exactly when
|
|
# the disk is most likely to be the reason it failed.
|
|
#
|
|
# Deliberately narrow. `docker image prune` WITHOUT -a removes dangling images
|
|
# only; with -a it would delete the act runner image, which no container references
|
|
# between jobs, and buy back a 1.6GB re-pull on the very next run. The buildx cache
|
|
# is the part that actually grows without bound, so it is pruned by age and keeps a
|
|
# week — recent enough that `--cache-from` still hits on normal traffic.
|
|
#
|
|
# Named volumes are never pruned here: that is where the trivy vuln DB lives.
|
|
#
|
|
# "Dangling" catches more than it looks: an image pulled by digest has no tag, so it
|
|
# is dangling as soon as its container exits. That is why the trivy scan step tags
|
|
# its image — an age filter does NOT protect it, because `until` reads the image's
|
|
# created time, and a released tool image is always older than any useful window.
|
|
# Run #68 proved that: trivy was untagged and deleted in all three legs despite
|
|
# `until=168h`, and re-pulled 178MB each time.
|
|
#
|
|
# The age filter stays anyway, for the buildx cache, which really does grow by age.
|
|
# The act runner image survives only because this step runs inside an act container,
|
|
# so the image is in use exactly while the prune runs. That is luck, not design; if
|
|
# it ever starts disappearing, tag it the same way.
|
|
if: always()
|
|
run: |
|
|
docker image prune -f --filter until=168h
|
|
docker buildx prune -af --filter until=168h
|
|
echo "--- dind disk after prune ---"
|
|
docker system df
|
|
|
|
# --- stage 11: bump the chart's digests. CI's last act. ------------------------------
|
|
bump:
|
|
runs-on: ubuntu-latest
|
|
needs: [image]
|
|
# The master+push guard lives on the STEPS, not on the job.
|
|
#
|
|
# A job-level `if` here evaluated false and skipped `bump` at 0s — before `image` had
|
|
# even started — on a genuine push to master (`event: push`, `head_branch: master`).
|
|
# The identical expression on the `push by digest` STEP inside the matrix job evaluates
|
|
# true and runs. Gitea appears not to resolve a job-level `if` correctly when `needs`
|
|
# points at a matrix job.
|
|
#
|
|
# `needs: [image]` still does the ordering and still gates on all three legs passing.
|
|
# The steps below carry the branch guard, in the form this runner is known to evaluate
|
|
# correctly. On a PR the job starts and every step no-ops, which costs a few seconds
|
|
# and is the price of a guard that actually fires.
|
|
permissions:
|
|
contents: write
|
|
steps:
|
|
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
with:
|
|
# A bot token with contents:write on this repo and nothing else: no kubeconfig,
|
|
# no cluster credential, no ArgoCD API token. CI's maximum blast radius is a bad
|
|
# commit, which is revertable.
|
|
token: ${{ secrets.CI_BOT_TOKEN }}
|
|
ref: master
|
|
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
with:
|
|
registry: ${{ env.REGISTRY }}
|
|
username: ${{ secrets.REGISTRY_USER }}
|
|
password: ${{ secrets.REGISTRY_TOKEN }}
|
|
- name: bump image digests in the chart
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
env:
|
|
REGISTRY: ${{ env.REGISTRY }}
|
|
IMAGE_NS: ${{ env.IMAGE_NS }}
|
|
IMAGE_TAG: ${{ github.sha }}
|
|
# And then it stops. No kubectl, no helm upgrade, no argocd app sync. ArgoCD is
|
|
# watching master and will have this within a minute.
|
|
run: ./scripts/bump-digests.sh
|