Skip to content

test(daemon): 30s exit-poll budget from #1633 still not enough under heavy host load #1653

Description

@newhoggy

Problem

PR #1633 (closing #1632) raised tests/daemon_test.rs's wait_for exit-poll deadline from 5s to 30s, on the reasoning that a hand-measured graceful shutdown took ~9.6s and a generous ceiling costs nothing on the happy path. That fix is good but the ceiling itself can still be too low: on 2026-09-08, both lifecycle tests failed again, this time past the 30s budget.

Evidence

On the issue-1643-sheets-formatting-validation branch (zero files under src/daemon/ or tests/daemon_test.rs touched, verified via git diff origin/main...HEAD -- tests/daemon_test.rs src/daemon/ — empty):

  • cargo test (full suite): daemon_run_status_stop_roundtrip failed — daemon did not exit after shutdown, test finished in 61.97s.
  • Isolated re-run, cargo test --test daemon_test daemon_run_status_stop_roundtrip: failed again — same panic, 65.68s.
  • A separate full-suite run (excluding that test) instead flaked on the other lifecycle test, daemon_run_shuts_down_gracefully_on_sighup, daemon did not exit after SIGHUP, 66.36s.

Both totals are well past wait_for's 30s ceiling plus the ~5s readiness-ping budget that precedes it, so this isn't the 9.6s-vs-5s gap #1633 fixed — actual drain (or process-exit signalling) is now taking 45s+ under load.

Host state at the time, matching the mechanism the 2026-08-04 investigation in #1632 already identified:

$ ps aux | grep "omni-dev daemon"
jky  32195  0.4  0.1 ... S  Mon09AM 589:20.74 /Users/jky/.cargo/bin/omni-dev daemon run --socket ...

$ uptime
15:44  up 16 days, 23:25, 9 users, load averages: 43.23 18.21 9.87

A long-resident real daemon plus a 1-minute load average of 43 on an 18-core machine (>2x core count) is exactly the CPU-starvation scenario documented previously — just severe enough this time to blow through the widened budget too.

Why this is worth a fresh issue rather than reopening #1632

#1633's fix and reasoning are still correct as far as they go — a generous ceiling costs nothing on the happy path, which remains true. The problem is that "generous" turned out to be host-load-dependent in a way a single fixed constant can't fully absorb: 9.6s was one measurement on one occasion, and today's host state pushed the real number past 3x that.

Proposal

Some combination of:

  • Make the deadline env-tunable (OMNI_DEV_TEST_DAEMON_EXIT_TIMEOUT_SECS or similar), so a known-loaded machine (or CI, if this ever shows up there) can widen it further without a code change.
  • Consider whether the test can detect "still draining, just slowly" (e.g. poll a liveness signal rather than only final-exit) to distinguish a genuinely hung daemon from one that's merely slow, rather than picking an even larger fixed ceiling and hoping.
  • At minimum, bump the constant again (e.g. 60s) as a cheap interim mitigation, on the same "costs nothing on the happy path" logic test(daemon): give the lifecycle exit poll 30s instead of 5s #1633 used.

Scope

tests/daemon_test.rs only, as with #1632/#1633. No production code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions