docs: bring RUNBOOK and ARCHITECTURE up to date
ci / lint (push) Successful in 23s
ci / types (push) Successful in 32s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 37s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 7s
ci / integration (push) Successful in 47s
ci / image (api) (push) Successful in 1m0s
ci / image (reconciler) (push) Successful in 2m14s
ci / image (worker) (push) Successful in 2m16s
ci / bump (push) Successful in 26s

The runner moved to node0 and the drift check reads release secrets via the
Kubernetes API in-cluster; the docs still described node2 and `helm list`.

- RUNBOOK: the durable image-cache fix is now the node0 hostPath store, not a
  node2 pin; /data is NFS RWX, so the Multi-Attach wait is gone. Cross-references
  entries 9 and 10.
- ARCHITECTURE: the reconciler's edge to the cluster is "list releases", not
  "helm list" (helm is the out-of-cluster fallback).
- ARCHITECTURE: the state diagram and its prose described fail() moving every
  dead-lettered instance to `failed`. Corrected to the per-kind behaviour — only
  provision fails the instance; deprovision stays `deleting` for retry, upgrade
  and verify stay `ready` — matching the fix in tasks.py.
This commit is contained in:
Nguyen Minh Phuc
2026-07-21 01:45:09 +00:00
parent 66eb6cb0ee
commit 7079d6340f
2 changed files with 31 additions and 17 deletions
+24 -12
View File
@@ -46,7 +46,7 @@ flowchart LR
WORKER -->|"claim<br/>SKIP LOCKED"| PG
WORKER -->|"helm upgrade --install"| K8S
RECON -->|"drift, leases,<br/>TTL, versions"| PG
RECON -->|"helm list"| K8S
RECON -->|"list releases"| K8S
classDef truth fill:#2d4a22,stroke:#5a8f3d,color:#fff
classDef derived fill:#4a3222,stroke:#8f6a3d,color:#fff
@@ -191,12 +191,12 @@ stateDiagram-v2
ready --> deleting: DELETE, or TTL expired
deleting --> deleted: helm uninstall succeeded
requested --> failed: attempts exhausted
provisioning --> failed: attempts exhausted
requested --> failed: provision attempts exhausted
provisioning --> failed: provision attempts exhausted
ready --> failed: drift — the release vanished
failed --> provisioning: retry
failed --> deleting: give up, tear it down
deleting --> failed: attempts exhausted
deleting --> deleting: deprovision retried, never failed
deleted --> [*]: terminal
```
@@ -205,19 +205,31 @@ stateDiagram-v2
chain of `if`s. `deleted` maps to an **empty frozenset** rather than being absent, so
"terminal" is stated rather than implied by a missing key.
**The state machine is enforced in SQL too.** `TaskRepo.fail` writes `instances.state`
directly, so it derives its guard from the same `LEGAL` table:
**Dead-lettering the task does not fail the instance, except for provision.** When a task
exhausts its retries `TaskRepo.fail` marks the *task* `failed` for every kind. It moves the
*instance* to `failed` only for `provision`, because that is the only kind where a dead
letter means the instance is broken. For the others the instance is still healthy and
something else owns its recovery:
| kind | instance state on dead-letter | why |
|------|-------------------------------|-----|
| provision | `failed` | it never came up; a human re-provisions |
| deprovision | stays `deleting` | so `due_for_deprovision` re-enqueues it; `failed` would strand it and leak the release |
| upgrade | stays `ready` | `helm --atomic` rolled back; it still serves the old version |
| verify | stays `ready` | `handle_verify` already halted the rollout |
The provision write is still guarded by the state machine, derived from the same `LEGAL`
table rather than restated:
```python
_CAN_FAIL = tuple(s.value for s, allowed in LEGAL.items() if InstanceState.FAILED in allowed)
...
UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
if kind == 'provision':
UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
```
Without that, a deprovision exhausting its retries against an already-`deleted` instance
would resurrect it into `failed` — a transition `transition()` explicitly forbids,
performed by raw SQL that never asked it. A state machine only one layer respects is
decoration.
The `SvcforgeTaskDeadLettered` alert fires for every kind, so leaving the instance alone
loses no operator visibility.
---
@@ -278,7 +290,7 @@ One replica. Four checks. Every 60 seconds.
```mermaid
flowchart LR
TICK(("tick<br/>every 60s")) --> D["<b>drift</b><br/>helm list vs DB"]
TICK(("tick<br/>every 60s")) --> D["<b>drift</b><br/>live releases vs DB"]
TICK --> L["<b>lease expiry</b><br/>running + locked_at old"]
TICK --> T["<b>TTL</b><br/>ready + expires_at passed"]
TICK --> V["<b>version drift</b><br/>chart_version ≠ catalog"]