diff --git a/RUNBOOK.md b/RUNBOOK.md index 2cb6cc1..6448b8f 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -153,6 +153,53 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc To stop a running job: click Cancel in the UI. +### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough + +`gitea-postgresql-0` CrashLoopBackOff with: + + mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error + +and the Gitea API returning 500, so every CI run dies at checkout with +`Failed to connect to gitea-http:3000`. The "Initializing PostgreSQL database" line above +that error is alarming and is not what it looks like: the data is fine, the mount is broken, +so the container sees an empty directory. + +The documented recovery — scale to 0, wait for `detached`, scale back to 1 — was **not +enough** here. The volume came back `detached/faulted` and simply refused to attach, so the +pod sat in ContainerCreating. `auto-salvage: true` does not help: salvage happens during +attach, and a faulted volume never gets that far, so it cannot rescue itself. + +What the volume was actually saying: + +```bash +V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}') +kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}' # detached/faulted +kubectl -n longhorn-system get replicas.longhorn.io -o json \ + | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv' +``` + +The replica carries a `failedAt` timestamp, and that alone is what keeps the volume faulted. +Clearing it is the salvage: + +```bash +kubectl -n longhorn-system patch replicas.longhorn.io --type merge \ + -p '{"spec":{"failedAt":"","lastFailedAt":""}}' +``` + +The volume went `attached/healthy` and postgres reached 1/1 within 40 seconds, with the repo, +its size and the whole CI run history intact. + +**Check the backups before patching anything**, because this cluster runs Longhorn at one +replica — there is no second copy to fall back on, only the nightly backup: + +```bash +kubectl -n longhorn-system get backups.longhorn.io -o json \ + | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv' +``` + +Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash +recovery on start. If it cannot, restore the most recent Completed backup instead. + ### 7. Verify the whole loop, not just the green checkmarks ```bash