Nguyen Minh Phuc
c9d0176bb3
trivy-action@v0.29.0 internally uses setup-trivy@v0.2.2, a tag removed upstream (earliest published is now v0.2.6), so it cannot resolve on any runner. Run trivy from a digest-pinned image instead, as gitleaks already is. RUNBOOK gains a 'Setting up CI/CD from scratch' section with the traps that actually cost time: GITEA_TOKEN is 401 at the package registry, the runner's cache fails soft, service containers resolve by name not localhost. Not pushed: pushing triggers a run, and the runner is being restarted by the ansible change that enables its cache.
319 lines
15 KiB
YAML
319 lines
15 KiB
YAML
# svcforge CI.
|
|
#
|
|
# The contract, in one line: merge to master -> three images built and scanned -> the
|
|
# chart's image digests bumped -> ArgoCD syncs. CI never touches the cluster. There is no
|
|
# kubeconfig here and there must never be one; the pipeline's last act is a git commit.
|
|
#
|
|
# Rules this file exists to enforce:
|
|
# - Build once, promote the artifact. The digest that trivy scanned is the digest that
|
|
# lands in values.yaml is the digest that runs.
|
|
# - Deploy by digest, never a mutable tag.
|
|
# - Everything pinned: actions by SHA, tool images by digest, deps by uv.lock + --frozen.
|
|
# - Every gate required. None advisory. Fail the PR, not prod.
|
|
#
|
|
# Stage order is deliberate and matches the module: cheapest and most likely to fail first,
|
|
# so a formatting mistake costs 20 seconds instead of three minutes of image builds.
|
|
|
|
name: ci
|
|
|
|
on: [pull_request, push]
|
|
|
|
concurrency:
|
|
# A second push to the same branch makes the first run's answer irrelevant. Cancel it —
|
|
# except on master, where the run ends in a commit and must not be interrupted midway.
|
|
group: ci-${{ github.ref }}
|
|
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
|
|
|
|
env:
|
|
REGISTRY: gitea.oci-oci.duckdns.org
|
|
IMAGE_NS: gitea_admin
|
|
UV_VERSION: "0.5.11"
|
|
|
|
jobs:
|
|
# --- stage 1: lint -- fast, fails first ---------------------------------------------
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
# The uv store is keyed on the lockfile: same lock, same wheels, cache hit.
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: ruff
|
|
run: uv run ruff check . && uv run ruff format --check .
|
|
|
|
# --- stage 2: types -- your compiler ------------------------------------------------
|
|
types:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: mypy --strict
|
|
run: uv run mypy --strict .
|
|
|
|
# --- stage 3: unit -- domain only, milliseconds, coverage gate -----------------------
|
|
unit:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: pytest unit
|
|
# The gate is on domain/ alone, and only domain/. It is pure, has no I/O, and needs
|
|
# no mocks — there is no excuse for a gap there. Pointing this at the whole repo
|
|
# would let untested SQL be paid for by well-tested pure functions.
|
|
#
|
|
# `--cov=svcforge_core.domain` — the MODULE, not a path. `--cov=libs/svcforge_core/domain`
|
|
# is a path that does not exist (the package nests one level deeper, at
|
|
# libs/svcforge_core/svcforge_core/domain), so coverage measured nothing and reported
|
|
# 0.00%. A path-based --cov that misses silently reports 0 rather than erroring, so
|
|
# without a --cov-fail-under this reads as a passing coverage gate over no code at all.
|
|
run: uv run pytest tests/unit --cov=svcforge_core.domain --cov-fail-under=90
|
|
|
|
# --- stages 4+5: migrate, then integration against that schema -----------------------
|
|
integration:
|
|
runs-on: ubuntu-latest
|
|
needs: [unit]
|
|
services:
|
|
postgres:
|
|
image: postgres:16@sha256:33f923b05f64ca54ac4401c01126a6b92afe839a0aa0a52bc5aeb5cc958e5f20
|
|
env:
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: svcforge
|
|
ports:
|
|
- 5432:5432
|
|
options: >-
|
|
--health-cmd "pg_isready -U postgres"
|
|
--health-interval 5s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
env:
|
|
# A scratch Postgres, so a literal password is correct here: it is not a secret, it
|
|
# is a fixture. Real DSNs live in Vault and reach the pods via external-secrets.
|
|
SVCFORGE_PG_DSN: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
SVCFORGE_PG_DSN_SESSION: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
# What tests/integration/conftest.py actually reads. Without it the fixture falls
|
|
# back to testcontainers and starts a SECOND Postgres inside the runner's docker,
|
|
# while the service container above sits unused — slower, and a different database
|
|
# to the one `migrate` just ran against.
|
|
SVCFORGE_TEST_DSN: postgresql://postgres:postgres@postgres:5432/svcforge
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
- name: migrate
|
|
# The same entrypoint the chart's pre-upgrade hook runs. If migrations only ever
|
|
# ran under testcontainers, CI would be testing a code path production never takes.
|
|
run: uv run python -m svcforge_core.migrate
|
|
- name: pytest integration
|
|
run: uv run pytest tests/integration
|
|
|
|
# --- stages 6+7+8: SAST, secrets, dependency CVEs ------------------------------------
|
|
# One job, three independent gates. They share a checkout and nothing else; each `run`
|
|
# step fails the job on its own.
|
|
security:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
with:
|
|
# gitleaks scans history, not just the tip. A secret committed and then reverted
|
|
# is still a leaked secret, and a shallow clone cannot see it.
|
|
fetch-depth: 0
|
|
- uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
|
|
with:
|
|
version: ${{ env.UV_VERSION }}
|
|
enable-cache: true
|
|
cache-dependency-glob: uv.lock
|
|
- run: uv sync --frozen
|
|
|
|
- name: bandit (SAST)
|
|
# `--with`, not a dev dependency: bandit is a CI tool, not something the project
|
|
# imports, and ruff's S ruleset already runs its checks in the lint stage. This is
|
|
# the belt to that suspenders — -ll reports medium severity and above only.
|
|
#
|
|
# `bandit[toml]` + `-c pyproject.toml`: without the toml extra bandit cannot read
|
|
# its own config and silently ignores it, which looks identical to a clean run.
|
|
run: uv run --with 'bandit[toml]' bandit -c pyproject.toml -r libs services -ll
|
|
|
|
- name: gitleaks (secret scan)
|
|
# Pinned by digest and run directly, so the command is the documented one rather
|
|
# than a marketplace action's opinion of it.
|
|
run: |
|
|
docker run --rm -v "$PWD:/repo" -w /repo \
|
|
ghcr.io/gitleaks/gitleaks:v8.21.2@sha256:0e99e8821643ea5b235718642b93bb32486af9c8162c8b8731f7cbdc951a7f46 \
|
|
detect --no-banner --source /repo
|
|
|
|
- name: pip-audit (dependency CVEs)
|
|
# --strict fails on an audit error rather than shrugging and reporting clean.
|
|
#
|
|
# Audits the LOCKED dependency set, not the installed environment. Auditing the env
|
|
# means auditing `svcforge` and `svcforge-core` too, which are ours, are installed
|
|
# editable, and are not on PyPI — under --strict that is a hard error ("distribution
|
|
# marked as editable"), so the choice was to drop --strict or to stop asking PyPI
|
|
# about packages it has never heard of. This asks about the 56 that actually came
|
|
# from PyPI, and keeps --strict.
|
|
run: |
|
|
uv export --frozen --no-dev \
|
|
--no-emit-project --no-emit-package svcforge-core \
|
|
-o /tmp/requirements-audit.txt
|
|
uv run --with pip-audit pip-audit --strict -r /tmp/requirements-audit.txt
|
|
|
|
# --- stage 9: hadolint ---------------------------------------------------------------
|
|
dockerfile:
|
|
runs-on: ubuntu-latest
|
|
needs: [lint]
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- name: hadolint
|
|
uses: hadolint/hadolint-action@54c9adbab1582c2ef04b2016b760714a4bfde3cf # v3.1.0
|
|
with:
|
|
recursive: true
|
|
dockerfile: "services/*/Dockerfile"
|
|
failure-threshold: warning
|
|
|
|
# --- stage 10: build -> trivy -> push by digest --------------------------------------
|
|
image:
|
|
runs-on: ubuntu-latest
|
|
# Every gate above is required. An image is not built until all of them are green,
|
|
# which is what makes "the digest CI pushed is a digest that passed everything" true.
|
|
needs: [types, unit, integration, security, dockerfile]
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
svc: [api, worker, reconciler]
|
|
# Deliberately no `outputs:` here. Matrix legs share one outputs map and clobber each
|
|
# other — the merge is not per-key and not ordered, so two of the three digests would
|
|
# arrive empty or stale, intermittently. The bump job resolves the digests from the
|
|
# registry instead, which is a read, not a rebuild.
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
- uses: docker/setup-buildx-action@c47758b77c9736f4b2ef4073d4d51994fabfe349 # v3.7.1
|
|
|
|
- name: registry login
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
|
|
with:
|
|
registry: ${{ env.REGISTRY }}
|
|
username: ${{ secrets.REGISTRY_USER }}
|
|
password: ${{ secrets.REGISTRY_TOKEN }}
|
|
|
|
- name: build
|
|
# Context is the repo root and the Dockerfile is addressed with -f. It cannot be
|
|
# otherwise: the image needs pyproject.toml, uv.lock and libs/, all of which live
|
|
# above services/<svc>/, and COPY ../.. is illegal.
|
|
#
|
|
# Loaded locally, not pushed. Trivy scans this exact image next; only then does it
|
|
# get pushed. The alternative — push, scan, and hope nobody pulled meanwhile — is
|
|
# how a CRITICAL ends up in the registry with a green checkmark next to it.
|
|
run: |
|
|
docker buildx build \
|
|
-f services/${{ matrix.svc }}/Dockerfile \
|
|
--build-arg BUILD_SHA=${{ github.sha }} \
|
|
--cache-from type=gha,scope=${{ matrix.svc }} \
|
|
--cache-to type=gha,mode=max,scope=${{ matrix.svc }} \
|
|
--load \
|
|
-t svcforge/${{ matrix.svc }}:ci \
|
|
.
|
|
|
|
- name: trivy
|
|
# Run trivy directly rather than via aquasecurity/trivy-action, for the same reason
|
|
# gitleaks is run directly above: the command is the documented one, pinned by
|
|
# digest, with no nested action resolution.
|
|
#
|
|
# It is also the only thing that works here. trivy-action internally does
|
|
# `uses: aquasecurity/setup-trivy@v0.2.2`, and that tag no longer exists upstream —
|
|
# the earliest published tag today is v0.2.6. The runner clones it and fails with
|
|
# "Unable to resolve v0.2.2: reference not found". A third-party action pinned by
|
|
# SHA still resolves ITS OWN dependencies by mutable tag, so pinning the outer
|
|
# action bought nothing.
|
|
#
|
|
# --ignore-unfixed: a CVE with no fix available is not something this PR can act
|
|
# on, and failing on it only teaches people to add ignore entries. Rebuilding on a
|
|
# new base image picks the fix up the day it exists.
|
|
#
|
|
# The mounted cache dir keeps the vuln DB (~50MB) across the three matrix legs on
|
|
# this runner instead of re-downloading it for each.
|
|
run: |
|
|
docker run --rm \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v "$PWD/.trivycache:/root/.cache/trivy" \
|
|
aquasec/trivy:0.58.1@sha256:ab70a02200597efa04748f210f793936eb647cbcdb0ea69cc30b226d6f5a22c7 \
|
|
image \
|
|
--severity HIGH,CRITICAL \
|
|
--ignore-unfixed \
|
|
--exit-code 1 \
|
|
--format table \
|
|
--no-progress \
|
|
svcforge/${{ matrix.svc }}:ci
|
|
|
|
- name: push by digest
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
# Re-running buildx here is a cache hit on every layer, not a second build: the
|
|
# image is byte-identical to the one trivy just cleared. buildx cannot --load and
|
|
# --push in one invocation, which is the only reason this step exists.
|
|
#
|
|
# The commit-SHA tag is a handle for the bump job to resolve, not something anything
|
|
# deploys. What deploys is the digest that tag resolves to.
|
|
run: |
|
|
set -euo pipefail
|
|
IMAGE="${REGISTRY}/${IMAGE_NS}/svcforge-${{ matrix.svc }}"
|
|
docker buildx build \
|
|
-f services/${{ matrix.svc }}/Dockerfile \
|
|
--build-arg BUILD_SHA=${{ github.sha }} \
|
|
--cache-from type=gha,scope=${{ matrix.svc }} \
|
|
--push \
|
|
-t "${IMAGE}:${GITHUB_SHA}" \
|
|
.
|
|
docker buildx imagetools inspect "${IMAGE}:${GITHUB_SHA}" \
|
|
--format '{{.Manifest.Digest}}'
|
|
|
|
# --- stage 11: bump the chart's digests. CI's last act. ------------------------------
|
|
bump:
|
|
runs-on: ubuntu-latest
|
|
needs: [image]
|
|
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
|
|
permissions:
|
|
contents: write
|
|
steps:
|
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
|
with:
|
|
# A bot token with contents:write on this repo and nothing else. Note what is
|
|
# absent: no kubeconfig, no cluster credential, no ArgoCD API token. CI's maximum
|
|
# blast radius is a bad commit, which is revertable.
|
|
token: ${{ secrets.CI_BOT_TOKEN }}
|
|
ref: master
|
|
- uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
|
|
with:
|
|
registry: ${{ env.REGISTRY }}
|
|
username: ${{ secrets.REGISTRY_USER }}
|
|
password: ${{ secrets.REGISTRY_TOKEN }}
|
|
- name: bump image digests in the chart
|
|
env:
|
|
REGISTRY: ${{ env.REGISTRY }}
|
|
IMAGE_NS: ${{ env.IMAGE_NS }}
|
|
IMAGE_TAG: ${{ github.sha }}
|
|
# And then it stops. No kubectl, no helm upgrade, no argocd app sync. ArgoCD is
|
|
# watching master and will have this within a minute.
|
|
run: ./scripts/bump-digests.sh
|