diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 9b7b91c..3afdfe7 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -46,7 +46,7 @@ flowchart LR
WORKER -->|"claim
SKIP LOCKED"| PG
WORKER -->|"helm upgrade --install"| K8S
RECON -->|"drift, leases,
TTL, versions"| PG
- RECON -->|"helm list"| K8S
+ RECON -->|"list releases"| K8S
classDef truth fill:#2d4a22,stroke:#5a8f3d,color:#fff
classDef derived fill:#4a3222,stroke:#8f6a3d,color:#fff
@@ -191,12 +191,12 @@ stateDiagram-v2
ready --> deleting: DELETE, or TTL expired
deleting --> deleted: helm uninstall succeeded
- requested --> failed: attempts exhausted
- provisioning --> failed: attempts exhausted
+ requested --> failed: provision attempts exhausted
+ provisioning --> failed: provision attempts exhausted
ready --> failed: drift — the release vanished
failed --> provisioning: retry
failed --> deleting: give up, tear it down
- deleting --> failed: attempts exhausted
+ deleting --> deleting: deprovision retried, never failed
deleted --> [*]: terminal
```
@@ -205,19 +205,31 @@ stateDiagram-v2
chain of `if`s. `deleted` maps to an **empty frozenset** rather than being absent, so
"terminal" is stated rather than implied by a missing key.
-**The state machine is enforced in SQL too.** `TaskRepo.fail` writes `instances.state`
-directly, so it derives its guard from the same `LEGAL` table:
+**Dead-lettering the task does not fail the instance, except for provision.** When a task
+exhausts its retries `TaskRepo.fail` marks the *task* `failed` for every kind. It moves the
+*instance* to `failed` only for `provision`, because that is the only kind where a dead
+letter means the instance is broken. For the others the instance is still healthy and
+something else owns its recovery:
+
+| kind | instance state on dead-letter | why |
+|------|-------------------------------|-----|
+| provision | `failed` | it never came up; a human re-provisions |
+| deprovision | stays `deleting` | so `due_for_deprovision` re-enqueues it; `failed` would strand it and leak the release |
+| upgrade | stays `ready` | `helm --atomic` rolled back; it still serves the old version |
+| verify | stays `ready` | `handle_verify` already halted the rollout |
+
+The provision write is still guarded by the state machine, derived from the same `LEGAL`
+table rather than restated:
```python
_CAN_FAIL = tuple(s.value for s, allowed in LEGAL.items() if InstanceState.FAILED in allowed)
...
-UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
+if kind == 'provision':
+ UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
```
-Without that, a deprovision exhausting its retries against an already-`deleted` instance
-would resurrect it into `failed` — a transition `transition()` explicitly forbids,
-performed by raw SQL that never asked it. A state machine only one layer respects is
-decoration.
+The `SvcforgeTaskDeadLettered` alert fires for every kind, so leaving the instance alone
+loses no operator visibility.
---
@@ -278,7 +290,7 @@ One replica. Four checks. Every 60 seconds.
```mermaid
flowchart LR
- TICK(("tick
every 60s")) --> D["drift
helm list vs DB"]
+ TICK(("tick
every 60s")) --> D["drift
live releases vs DB"]
TICK --> L["lease expiry
running + locked_at old"]
TICK --> T["TTL
ready + expires_at passed"]
TICK --> V["version drift
chart_version ≠ catalog"]
diff --git a/RUNBOOK.md b/RUNBOOK.md
index 569360f..c2d6a9f 100644
--- a/RUNBOOK.md
+++ b/RUNBOOK.md
@@ -118,11 +118,13 @@ kubectl -n gitea exec gitea-actions-runner-0 -c dind -- docker pull \
ghcr.io/catthehacker/ubuntu:act-24.04@sha256:c710431fbad9eb3bcb102d04e5ff74fbd0ce6e383f78afebfb3770a1a817fdf9
```
-The durable fix is to stop the runner restarting. Its `/data` PVC is ReadWriteOnce, so
-every reschedule hits `Multi-Attach error` and the pod sits in Init until Longhorn detaches
-from the old node. It is pinned to node2 in `oci-k8s/.../addons/tasks/main.yml` for exactly
-that reason. A dedicated PVC for the image cache would survive restarts outright, but on
-this cluster that volume faulted and blocked the runner, so it is deliberately not used.
+The durable fix is a persistent image store, which the runner now has: `/var/lib/docker`
+is a hostPath on node0 (see `oci-k8s/.../addons/tasks/main.yml`), so the act image survives
+a restart and is not re-pulled. The runner is pinned to **node0**, not node2 — node2 is a
+single-core control-plane node whose pod network was measured 21x slower under its own
+load, which starved every clone and pull. Its `/data` PVC is NFS ReadWriteMany, so a
+reschedule attaches immediately with no `Multi-Attach` wait. Entries 9 and 10 cover the
+caches and the node move in full.
### 6. Stopping a run, and reading a restarted runner correctly