You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #1633 (closing #1632) raised tests/daemon_test.rs's wait_for exit-poll deadline from 5s to 30s, on the reasoning that a hand-measured graceful shutdown took ~9.6s and a generous ceiling costs nothing on the happy path. That fix is good but the ceiling itself can still be too low: on 2026-09-08, both lifecycle tests failed again, this time past the 30s budget.
Evidence
On the issue-1643-sheets-formatting-validation branch (zero files under src/daemon/ or tests/daemon_test.rs touched, verified via git diff origin/main...HEAD -- tests/daemon_test.rs src/daemon/ — empty):
cargo test (full suite): daemon_run_status_stop_roundtrip failed — daemon did not exit after shutdown, test finished in 61.97s.
Isolated re-run, cargo test --test daemon_test daemon_run_status_stop_roundtrip: failed again — same panic, 65.68s.
A separate full-suite run (excluding that test) instead flaked on the other lifecycle test, daemon_run_shuts_down_gracefully_on_sighup, daemon did not exit after SIGHUP, 66.36s.
Both totals are well past wait_for's 30s ceiling plus the ~5s readiness-ping budget that precedes it, so this isn't the 9.6s-vs-5s gap #1633 fixed — actual drain (or process-exit signalling) is now taking 45s+ under load.
Host state at the time, matching the mechanism the 2026-08-04 investigation in #1632 already identified:
$ ps aux | grep "omni-dev daemon"
jky 32195 0.4 0.1 ... S Mon09AM 589:20.74 /Users/jky/.cargo/bin/omni-dev daemon run --socket ...
$ uptime
15:44 up 16 days, 23:25, 9 users, load averages: 43.23 18.21 9.87
A long-resident real daemon plus a 1-minute load average of 43 on an 18-core machine (>2x core count) is exactly the CPU-starvation scenario documented previously — just severe enough this time to blow through the widened budget too.
Why this is worth a fresh issue rather than reopening #1632
#1633's fix and reasoning are still correct as far as they go — a generous ceiling costs nothing on the happy path, which remains true. The problem is that "generous" turned out to be host-load-dependent in a way a single fixed constant can't fully absorb: 9.6s was one measurement on one occasion, and today's host state pushed the real number past 3x that.
Proposal
Some combination of:
Make the deadline env-tunable (OMNI_DEV_TEST_DAEMON_EXIT_TIMEOUT_SECS or similar), so a known-loaded machine (or CI, if this ever shows up there) can widen it further without a code change.
Consider whether the test can detect "still draining, just slowly" (e.g. poll a liveness signal rather than only final-exit) to distinguish a genuinely hung daemon from one that's merely slow, rather than picking an even larger fixed ceiling and hoping.
Problem
PR #1633 (closing #1632) raised
tests/daemon_test.rs'swait_forexit-poll deadline from 5s to 30s, on the reasoning that a hand-measured graceful shutdown took ~9.6s and a generous ceiling costs nothing on the happy path. That fix is good but the ceiling itself can still be too low: on 2026-09-08, both lifecycle tests failed again, this time past the 30s budget.Evidence
On the
issue-1643-sheets-formatting-validationbranch (zero files undersrc/daemon/ortests/daemon_test.rstouched, verified viagit diff origin/main...HEAD -- tests/daemon_test.rs src/daemon/— empty):cargo test(full suite):daemon_run_status_stop_roundtripfailed —daemon did not exit after shutdown, test finished in 61.97s.cargo test --test daemon_test daemon_run_status_stop_roundtrip: failed again — same panic, 65.68s.daemon_run_shuts_down_gracefully_on_sighup,daemon did not exit after SIGHUP, 66.36s.Both totals are well past
wait_for's 30s ceiling plus the ~5s readiness-ping budget that precedes it, so this isn't the 9.6s-vs-5s gap #1633 fixed — actual drain (or process-exit signalling) is now taking 45s+ under load.Host state at the time, matching the mechanism the 2026-08-04 investigation in #1632 already identified:
A long-resident real daemon plus a 1-minute load average of 43 on an 18-core machine (>2x core count) is exactly the CPU-starvation scenario documented previously — just severe enough this time to blow through the widened budget too.
Why this is worth a fresh issue rather than reopening #1632
#1633's fix and reasoning are still correct as far as they go — a generous ceiling costs nothing on the happy path, which remains true. The problem is that "generous" turned out to be host-load-dependent in a way a single fixed constant can't fully absorb: 9.6s was one measurement on one occasion, and today's host state pushed the real number past 3x that.
Proposal
Some combination of:
OMNI_DEV_TEST_DAEMON_EXIT_TIMEOUT_SECSor similar), so a known-loaded machine (or CI, if this ever shows up there) can widen it further without a code change.Scope
tests/daemon_test.rsonly, as with #1632/#1633. No production code.