A merged PR may require knowledge-base updates. Review the change and update library/ as needed.
Source PR
- PR: #313 - fix: embed supervisor heals its restart budget, retries after failure, and logs every transition (ISS-023)
- Author: @thenotoriousllama
- Merged into:
main
- Files changed: 13
PR description
ISS-023 — Embed supervisor's lifetime restart budget turns transient slowness into permanent lexical-only
Register entry: library/issues/issue-register.md § ISS-023 (live-diagnosed 2026-07-13; systemic pattern SP-3 — the five kills were invisible in event_log).
On 0.19.1 (CPU inference, cold model load 10-40s, post-deploy job load), real embed_timeout degradations 00:45-00:48 became wedge-kills; five cycles exhausted the LIFETIME maxRestarts=5 budget; from 00:49 the child was dead (port 3851 connection-refused, embed_not_ready on every recall) — embeddings permanently off until a manual daemon restart. The #301 supervisor failed in production the opposite way from the wedge it was built to detect.
Reconstruction: why five kills happened despite /health-first + the warming state
The warming state and the probes-armed-only-after-warm design were not the hole — every respawn path correctly transitioned through warming with probes disarmed, and each fresh child re-warmed. The hole was the on-demand probe chain being load-correlated by construction:
- The trigger is correlated with saturation. Recall's
reportTimeout → checkNow fires at the exact moment a bounded embed burned its deadline (assemble.ts embedGate wiring). On this machine an embed times out because the child's event loop is saturated by 10-27s CPU inferences between FIFO yields — so checkNow launched the kill-capable probe chain precisely when the child could not answer a 2s-bounded /health. The periodic 30s tick fires at random phase and mostly lands in idle moments; checkNow never does.
- The confirm window fit inside the saturation window. The chain's confirming re-probe fired 1s (
livenessRetryMs) after the first miss — guaranteed to land inside the same 10-27s busy stretch. The whole kill decision (2s probe + 1s wait + 2s probe ≈ 5s) was evaluated against a known 10-40s busy envelope: two misses ⇒ "wedge confirmed" ⇒ SIGT
Files touched
.claude-plugin/marketplace.json (+2/-2).claude-plugin/plugin.json (+1/-1)CHANGELOG.md (+4/-0)harnesses/claude-code/.claude-plugin/plugin.json (+1/-1)harnesses/codex/package.json (+1/-1)harnesses/openclaw/openclaw.plugin.json (+1/-1)harnesses/openclaw/package.json (+1/-1)package-lock.json (+2/-2)package.json (+1/-1)src/daemon/runtime/assemble.ts (+34/-10)src/daemon/runtime/health.ts (+47/-0)src/daemon/runtime/services/embed-supervisor.ts (+284/-26)tests/daemon/runtime/services/embed-supervisor-availability.test.ts (+554/-0)
Auto-generated by .github/workflows/kb-issue-on-merge.yaml on merge.
A merged PR may require knowledge-base updates. Review the change and update
library/as needed.Source PR
mainPR description
ISS-023 — Embed supervisor's lifetime restart budget turns transient slowness into permanent lexical-only
Register entry:
library/issues/issue-register.md§ ISS-023 (live-diagnosed 2026-07-13; systemic pattern SP-3 — the five kills were invisible in event_log).On 0.19.1 (CPU inference, cold model load 10-40s, post-deploy job load), real
embed_timeoutdegradations 00:45-00:48 became wedge-kills; five cycles exhausted the LIFETIMEmaxRestarts=5budget; from 00:49 the child was dead (port 3851 connection-refused,embed_not_readyon every recall) — embeddings permanently off until a manual daemon restart. The #301 supervisor failed in production the opposite way from the wedge it was built to detect.Reconstruction: why five kills happened despite /health-first + the warming state
The warming state and the probes-armed-only-after-warm design were not the hole — every respawn path correctly transitioned through
warmingwith probes disarmed, and each fresh child re-warmed. The hole was the on-demand probe chain being load-correlated by construction:reportTimeout→checkNowfires at the exact moment a bounded embed burned its deadline (assemble.tsembedGate wiring). On this machine an embed times out because the child's event loop is saturated by 10-27s CPU inferences between FIFO yields — socheckNowlaunched the kill-capable probe chain precisely when the child could not answer a 2s-bounded/health. The periodic 30s tick fires at random phase and mostly lands in idle moments;checkNownever does.livenessRetryMs) after the first miss — guaranteed to land inside the same 10-27s busy stretch. The whole kill decision (2s probe + 1s wait + 2s probe ≈ 5s) was evaluated against a known 10-40s busy envelope: two misses ⇒ "wedge confirmed" ⇒ SIGTFiles touched
.claude-plugin/marketplace.json(+2/-2).claude-plugin/plugin.json(+1/-1)CHANGELOG.md(+4/-0)harnesses/claude-code/.claude-plugin/plugin.json(+1/-1)harnesses/codex/package.json(+1/-1)harnesses/openclaw/openclaw.plugin.json(+1/-1)harnesses/openclaw/package.json(+1/-1)package-lock.json(+2/-2)package.json(+1/-1)src/daemon/runtime/assemble.ts(+34/-10)src/daemon/runtime/health.ts(+47/-0)src/daemon/runtime/services/embed-supervisor.ts(+284/-26)tests/daemon/runtime/services/embed-supervisor-availability.test.ts(+554/-0)Auto-generated by
.github/workflows/kb-issue-on-merge.yamlon merge.