docs: bring RUNBOOK and ARCHITECTURE up to date
ci / lint (push) Successful in 23s
ci / types (push) Successful in 32s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 37s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 7s
ci / integration (push) Successful in 47s
ci / image (api) (push) Successful in 1m0s
ci / image (reconciler) (push) Successful in 2m14s
ci / image (worker) (push) Successful in 2m16s
ci / bump (push) Successful in 26s
ci / lint (push) Successful in 23s
ci / types (push) Successful in 32s
ci / unit (push) Successful in 27s
ci / security (push) Successful in 37s
ci / dockerfile (push) Successful in 6s
ci / chart (push) Successful in 7s
ci / integration (push) Successful in 47s
ci / image (api) (push) Successful in 1m0s
ci / image (reconciler) (push) Successful in 2m14s
ci / image (worker) (push) Successful in 2m16s
ci / bump (push) Successful in 26s
The runner moved to node0 and the drift check reads release secrets via the Kubernetes API in-cluster; the docs still described node2 and `helm list`. - RUNBOOK: the durable image-cache fix is now the node0 hostPath store, not a node2 pin; /data is NFS RWX, so the Multi-Attach wait is gone. Cross-references entries 9 and 10. - ARCHITECTURE: the reconciler's edge to the cluster is "list releases", not "helm list" (helm is the out-of-cluster fallback). - ARCHITECTURE: the state diagram and its prose described fail() moving every dead-lettered instance to `failed`. Corrected to the per-kind behaviour — only provision fails the instance; deprovision stays `deleting` for retry, upgrade and verify stay `ready` — matching the fix in tasks.py.
This commit is contained in:
+24
-12
@@ -46,7 +46,7 @@ flowchart LR
|
||||
WORKER -->|"claim<br/>SKIP LOCKED"| PG
|
||||
WORKER -->|"helm upgrade --install"| K8S
|
||||
RECON -->|"drift, leases,<br/>TTL, versions"| PG
|
||||
RECON -->|"helm list"| K8S
|
||||
RECON -->|"list releases"| K8S
|
||||
|
||||
classDef truth fill:#2d4a22,stroke:#5a8f3d,color:#fff
|
||||
classDef derived fill:#4a3222,stroke:#8f6a3d,color:#fff
|
||||
@@ -191,12 +191,12 @@ stateDiagram-v2
|
||||
ready --> deleting: DELETE, or TTL expired
|
||||
deleting --> deleted: helm uninstall succeeded
|
||||
|
||||
requested --> failed: attempts exhausted
|
||||
provisioning --> failed: attempts exhausted
|
||||
requested --> failed: provision attempts exhausted
|
||||
provisioning --> failed: provision attempts exhausted
|
||||
ready --> failed: drift — the release vanished
|
||||
failed --> provisioning: retry
|
||||
failed --> deleting: give up, tear it down
|
||||
deleting --> failed: attempts exhausted
|
||||
deleting --> deleting: deprovision retried, never failed
|
||||
|
||||
deleted --> [*]: terminal
|
||||
```
|
||||
@@ -205,19 +205,31 @@ stateDiagram-v2
|
||||
chain of `if`s. `deleted` maps to an **empty frozenset** rather than being absent, so
|
||||
"terminal" is stated rather than implied by a missing key.
|
||||
|
||||
**The state machine is enforced in SQL too.** `TaskRepo.fail` writes `instances.state`
|
||||
directly, so it derives its guard from the same `LEGAL` table:
|
||||
**Dead-lettering the task does not fail the instance, except for provision.** When a task
|
||||
exhausts its retries `TaskRepo.fail` marks the *task* `failed` for every kind. It moves the
|
||||
*instance* to `failed` only for `provision`, because that is the only kind where a dead
|
||||
letter means the instance is broken. For the others the instance is still healthy and
|
||||
something else owns its recovery:
|
||||
|
||||
| kind | instance state on dead-letter | why |
|
||||
|------|-------------------------------|-----|
|
||||
| provision | `failed` | it never came up; a human re-provisions |
|
||||
| deprovision | stays `deleting` | so `due_for_deprovision` re-enqueues it; `failed` would strand it and leak the release |
|
||||
| upgrade | stays `ready` | `helm --atomic` rolled back; it still serves the old version |
|
||||
| verify | stays `ready` | `handle_verify` already halted the rollout |
|
||||
|
||||
The provision write is still guarded by the state machine, derived from the same `LEGAL`
|
||||
table rather than restated:
|
||||
|
||||
```python
|
||||
_CAN_FAIL = tuple(s.value for s, allowed in LEGAL.items() if InstanceState.FAILED in allowed)
|
||||
...
|
||||
UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
|
||||
if kind == 'provision':
|
||||
UPDATE instances SET state='failed' WHERE id=%s AND state = ANY(%s)
|
||||
```
|
||||
|
||||
Without that, a deprovision exhausting its retries against an already-`deleted` instance
|
||||
would resurrect it into `failed` — a transition `transition()` explicitly forbids,
|
||||
performed by raw SQL that never asked it. A state machine only one layer respects is
|
||||
decoration.
|
||||
The `SvcforgeTaskDeadLettered` alert fires for every kind, so leaving the instance alone
|
||||
loses no operator visibility.
|
||||
|
||||
---
|
||||
|
||||
@@ -278,7 +290,7 @@ One replica. Four checks. Every 60 seconds.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
TICK(("tick<br/>every 60s")) --> D["<b>drift</b><br/>helm list vs DB"]
|
||||
TICK(("tick<br/>every 60s")) --> D["<b>drift</b><br/>live releases vs DB"]
|
||||
TICK --> L["<b>lease expiry</b><br/>running + locked_at old"]
|
||||
TICK --> T["<b>TTL</b><br/>ready + expires_at passed"]
|
||||
TICK --> V["<b>version drift</b><br/>chart_version ≠ catalog"]
|
||||
|
||||
Reference in New Issue
Block a user