From d4ac3801a3e569b5ece21618fdb89df99d6aa47d Mon Sep 17 00:00:00 2001
From: Nguyen Minh Phuc <son.luong@skylabteam.com>
Date: Mon, 20 Jul 2026 07:10:04 +0000
Subject: [PATCH] runbook: salvaging a faulted Longhorn volume

The documented scale 0/1 recovery did not work this time: the volume returned
detached/faulted and refused to attach, so the pod sat in ContainerCreating.
auto-salvage: true cannot rescue it, because salvage happens during attach and
a faulted volume never gets that far.

The replica's failedAt timestamp is the only thing holding it faulted.
Clearing it brought the volume to attached/healthy and postgres to 1/1 in
under a minute, with repo data and CI history intact.

Adds the backup check first, which matters more now that Longhorn runs at one
replica and there is no second copy to fall back on.
---
 RUNBOOK.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/RUNBOOK.md b/RUNBOOK.md
index 2cb6cc1..6448b8f 100644
--- a/RUNBOOK.md
+++ b/RUNBOOK.md
@@ -153,6 +153,53 @@ it and then 404 on retry. Treat it as a way to remove a finished run, not a canc
 
 To stop a running job: click Cancel in the UI.
 
+### 7. Gitea postgres: `Input/output error`, and when scale 0/1 is not enough
+
+`gitea-postgresql-0` CrashLoopBackOff with:
+
+    mkdir: cannot create directory '/bitnami/postgresql/data': Input/output error
+
+and the Gitea API returning 500, so every CI run dies at checkout with
+`Failed to connect to gitea-http:3000`. The "Initializing PostgreSQL database" line above
+that error is alarming and is not what it looks like: the data is fine, the mount is broken,
+so the container sees an empty directory.
+
+The documented recovery — scale to 0, wait for `detached`, scale back to 1 — was **not
+enough** here. The volume came back `detached/faulted` and simply refused to attach, so the
+pod sat in ContainerCreating. `auto-salvage: true` does not help: salvage happens during
+attach, and a faulted volume never gets that far, so it cannot rescue itself.
+
+What the volume was actually saying:
+
+```bash
+V=$(kubectl -n gitea get pvc -o jsonpath='{.items[?(@.metadata.name=="data-gitea-postgresql-0")].spec.volumeName}')
+kubectl -n longhorn-system get volume $V -o jsonpath='{.status.state}/{.status.robustness}'   # detached/faulted
+kubectl -n longhorn-system get replicas.longhorn.io -o json \
+  | jq -r '.items[]|select(.spec.volumeName=="'$V'")|[.metadata.name,.spec.failedAt]|@tsv'
+```
+
+The replica carries a `failedAt` timestamp, and that alone is what keeps the volume faulted.
+Clearing it is the salvage:
+
+```bash
+kubectl -n longhorn-system patch replicas.longhorn.io <replica-name> --type merge \
+  -p '{"spec":{"failedAt":"","lastFailedAt":""}}'
+```
+
+The volume went `attached/healthy` and postgres reached 1/1 within 40 seconds, with the repo,
+its size and the whole CI run history intact.
+
+**Check the backups before patching anything**, because this cluster runs Longhorn at one
+replica — there is no second copy to fall back on, only the nightly backup:
+
+```bash
+kubectl -n longhorn-system get backups.longhorn.io -o json \
+  | jq -r '.items[]|select(.status.volumeName=="'$V'")|[.status.backupCreatedAt,.status.state]|@tsv'
+```
+
+Salvage reuses the replica exactly as it was when it failed, so Postgres may do crash
+recovery on start. If it cannot, restore the most recent Completed backup instead.
+
 ### 7. Verify the whole loop, not just the green checkmarks
 
 ```bash