diff --git a/.gitea/workflows/ci.yaml b/.gitea/workflows/ci.yaml index 435e33c..2aec22e 100644 --- a/.gitea/workflows/ci.yaml +++ b/.gitea/workflows/ci.yaml @@ -237,16 +237,35 @@ jobs: . - name: trivy - uses: aquasecurity/trivy-action@18f2510ee396bbf400402947b394f2dd8c87dbb0 # v0.29.0 - with: - image-ref: svcforge/${{ matrix.svc }}:ci - severity: HIGH,CRITICAL - # A CVE with no fix available is not something this PR can act on; failing on it - # only teaches people to add ignore entries. Rebuilding on a new base image picks - # the fix up the day it exists. - ignore-unfixed: true - exit-code: "1" - format: table + # Run trivy directly rather than via aquasecurity/trivy-action, for the same reason + # gitleaks is run directly above: the command is the documented one, pinned by + # digest, with no nested action resolution. + # + # It is also the only thing that works here. trivy-action internally does + # `uses: aquasecurity/setup-trivy@v0.2.2`, and that tag no longer exists upstream — + # the earliest published tag today is v0.2.6. The runner clones it and fails with + # "Unable to resolve v0.2.2: reference not found". A third-party action pinned by + # SHA still resolves ITS OWN dependencies by mutable tag, so pinning the outer + # action bought nothing. + # + # --ignore-unfixed: a CVE with no fix available is not something this PR can act + # on, and failing on it only teaches people to add ignore entries. Rebuilding on a + # new base image picks the fix up the day it exists. + # + # The mounted cache dir keeps the vuln DB (~50MB) across the three matrix legs on + # this runner instead of re-downloading it for each. + run: | + docker run --rm \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v "$PWD/.trivycache:/root/.cache/trivy" \ + aquasec/trivy:0.58.1@sha256:ab70a02200597efa04748f210f793936eb647cbcdb0ea69cc30b226d6f5a22c7 \ + image \ + --severity HIGH,CRITICAL \ + --ignore-unfixed \ + --exit-code 1 \ + --format table \ + --no-progress \ + svcforge/${{ matrix.svc }}:ci - name: push by digest if: github.ref == 'refs/heads/master' && github.event_name == 'push' diff --git a/RUNBOOK.md b/RUNBOOK.md index 4357e1d..4024853 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -12,6 +12,89 @@ alias sfsql='psql "$SVCFORGE_PG_DSN_SESSION"' --- +## Setting up CI/CD from scratch + +What it takes to get this repo building in Gitea Actions, including every trap that cost +real time. Do it in this order; each step fails loudly if the one before it was skipped. + +### 1. The repo and its secrets + +```bash +GITEA=https://gitea.oci-oci.duckdns.org +USER=gitea_admin # the registry namespaces packages by OWNER, so this is IMAGE_NS too + +curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/user/repos" \ + -H 'content-type: application/json' \ + -d '{"name":"svcforge","default_branch":"master","private":false}' + +# A PAT with exactly three scopes. Not admin. +curl -u "$USER:$PASS" -X POST "$GITEA/api/v1/users/$USER/tokens" \ + -H 'content-type: application/json' \ + -d '{"name":"svcforge-ci","scopes":["write:package","write:repository","read:user"]}' +``` + +Then set three repo Actions secrets (`Settings → Actions → Secrets`, or the API): + +| Secret | Value | Why it exists | +|---|---|---| +| `REGISTRY_USER` | `gitea_admin` | | +| `REGISTRY_TOKEN` | the PAT | **The auto-injected `GITEA_TOKEN` is rejected by the package registry with a 401.** This is the single most common reason a first pipeline fails at `docker push`. | +| `CI_BOT_TOKEN` | the PAT | Used only by the `bump` job to push the digest commit. Note what it is *not*: a kubeconfig. CI's maximum blast radius is a bad commit. | + +### 2. The runner needs a cache server, and it fails SOFT without one + +`cache: enabled: false` in the act_runner config is the default, and it does not fail the +build. It prints: + +``` +Warning: Failed to restore: getCacheEntry failed: Cache Service Url not found, unable to restore cache. +``` + +…and carries on, re-downloading every wheel on every run, forever. **A cache that is off +looks exactly like a cache that is always cold.** Enable it in +`oci-k8s/k8s/roles/addons/tasks/main.yml` (Ansible owns this — never `kubectl edit` it): + +```yaml +cache: + enabled: true + dir: /data/cache # the runner's PVC, so it survives a restart + host: "" # auto-detect the pod IP; 127.0.0.1 would resolve to the JOB container + port: 8088 +``` + +```bash +cd oci-k8s/k8s && ansible-playbook 03_install_addons.yml --tags gitea +``` + +Without this, `astral-sh/setup-uv`'s `enable-cache: true` and buildx's `--cache-to +type=gha` are both no-ops. + +### 3. Traps that are specific to Gitea, not GitHub + +| Symptom | Cause | Fix | +|---|---|---| +| `Unable to resolve v0.2.2: reference not found` | A third-party action pinned by SHA still resolves **its own** dependencies by mutable tag. `trivy-action@v0.29.0` does `uses: setup-trivy@v0.2.2`, and that tag was removed upstream. | Run the tool directly from an image pinned by digest. Pinning the outer action bought nothing. | +| `docker push` → 401 | Used `GITEA_TOKEN`. | Use a PAT with `write:package`. | +| The pipeline retriggers itself forever | The `bump` job commits to the repo it is triggered by. | `[skip ci]` in the commit message (Gitea honours it), **and** a `git diff --quiet` guard so an unchanged digest commits nothing. | +| Service container unreachable at `localhost` | Jobs run *inside* a container, so a service is reached by its **service name**, not localhost. | `postgres:5432`, not `localhost:5432`. | + +### 4. Verify the whole loop, not just the green checkmarks + +```bash +# the digest CI pushed +docker buildx imagetools inspect gitea.oci-oci.duckdns.org/gitea_admin/svcforge-api: \ + --format '{{.Manifest.Digest}}' +# the digest the chart deploys — these must be equal +grep -A2 'api:' deploy/chart/values.yaml +# what ArgoCD actually synced +kubectl -n argocd get application svcforge -o jsonpath='{.status.sync.revision}' +``` + +If those three disagree, the deploy is not what CI tested, and every other guarantee in +this document is void. + +--- + ## Measured numbers From `scripts/load.py` + in-process workers on a `FakeProvisioner` (`delay=0.05s`), 200 tasks