Fix daemon-lifecycle signal handling and unbounded stats growth

[gmr]

Summary

Fixes 6 verified daemon-lifecycle bugs in mcp.py, controller.py, and state.py, centered on a coherent signal/shutdown flow and bounded stats growth.

Problem / Solution

• MCP on_sigchld shuts down the daemon when the last child dies instead of respawning it #76 on_sigchld shut the daemon down when the last child died instead of respawning → only stops on child_abort/max_messages/is_shutting_down; otherwise prunes and lets the next poll respawn. Wired up the previously-dead child_abort flag.
• MCP run() has a lost-wakeup race between the is_running check and signal.pause() #89 Lost-wakeup race between the is_running check and signal.pause() → replaced with a bounded wait loop that re-checks is_running/stop_requested.
• Controller KeyboardInterrupt/exception exit paths never stop child processes #90 KeyboardInterrupt/exception exit paths never stopped children (parent hung at atexit join) → controller.run() stops processes in a finally.
• stop_processes() in a signal handler can interleave with poll(), respawning orphaned children #91 stop_processes() in a signal handler could interleave with poll() and respawn orphaned children → handlers now only set a flag + cancel the itimer; check_process_counts/start_processes bail when not running; shutdown happens on the main thread.
• last_poll_results grows without bound under process churn and inflates logged stats #92 last_poll_results grew without bound and inflated logged stats → dead-process entries are pruned with their final counts folded into a retained per-consumer baseline (keeps Prometheus deltas correct); also fixed input mutation and the process_data aliasing.
• SIGTERM delivered between MCP construction and run() is silently nullified; set_state allows STOPPED → ACTIVE #99 SIGTERM between MCP construction and run() was nullified and set_state allowed STOPPED→ACTIVE → checks _shutdown_requested before run(); set_state rejects the STOPPED→ACTIVE transition.

Note on state.py

set_state now raises ValueError on exactly one transition (STATE_STOPPED → STATE_ACTIVE). Verified safe for process.py/connection.py, which never make that transition. No general transition table was added.

Testing

Full suite green (235 tests) + ruff clean. Tests in tests/test_mcp.py, tests/test_controller.py, tests/test_state.py.

Closes #76
Closes #89
Closes #90
Closes #91
Closes #92
Closes #99

Summary by CodeRabbit

• Bug Fixes
  • Improved shutdown handling so background work stops more reliably, including during interrupts, errors, and partial startup.
  • Prevented certain child processes from being stopped from signal handlers, reducing the chance of unstable shutdown behavior.
  • Fixed a state transition issue so stopped items can’t be reactivated unexpectedly.
  • Made process monitoring and restart behavior more consistent when children exit, fail to start, or are already shutting down.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 930b4d14-333f-41f0-8a65-de7ebefd694a

📥 Commits

Reviewing files that changed from the base of the PR and between 401dd78 and 8ec3de5.

📒 Files selected for processing (6)

• rejected/controller.py
• rejected/mcp.py
• rejected/state.py
• tests/test_controller.py
• tests/test_mcp.py
• tests/test_state.py

---

📝 Walkthrough

Walkthrough

Controller and MasterControlProgram coordinate shutdown via flags (stop_requested, _shutdown_requested) instead of executing stop logic inside signal handlers, closing multiple race conditions. MCP respawn/stop logic, process-count gating, and per-consumer stats aggregation with baseline retirement are reworked. State transitions restrict STOPPED→ACTIVE. Tests expanded accordingly.

Changes

MCP and Controller shutdown/stats overhaul

Layer / File(s)	Summary
State transition guard rejected/state.py, tests/test_state.py	set_state blocks STOPPED→ACTIVE transitions by raising ValueError; tests verify the blocked transition and an allowed STOPPED→SHUTTING_DOWN transition.
Controller run loop shutdown guard and cleanup rejected/controller.py, tests/test_controller.py	run() checks _shutdown_requested after MCP construction and always stops processes in a finally block; _on_sigterm/_request_stop defer child stopping by setting stop_requested instead of calling stop_processes inside handlers; tests cover normal completion, KeyboardInterrupt, exceptions, and construction-time shutdown.
MCP run loop and process gating rejected/mcp.py	Adds stop_requested flag, replaces signal.pause() with a bounded sleep loop, gates check_process_counts/start_processes on is_running, and sets child_abort on OSError during process start; on_sigchld only stops for abort/max_messages/shutdown conditions, otherwise allows respawn.
Consumer baseline stats and process retirement rejected/mcp.py, tests/test_mcp.py	calculate_stats avoids mutating input and folds retired consumer_baselines into per-consumer totals for monotonic counters; new retire_poll_results method prunes last_poll_results and accumulates baselines, invoked from remove_consumer_process; extensive new tests cover these behaviors.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SignalHandler
  participant Controller
  participant MCP
  participant ChildProcess

  SignalHandler-->>Controller: SIGTERM
  Controller->>Controller: _shutdown_requested = True
  Controller->>MCP: stop_requested = True
  MCP->>MCP: sleep loop checks stop_requested
  MCP-->>Controller: run() returns
  Controller->>MCP: stop_processes() (finally)
  MCP->>ChildProcess: terminate/join

Loading

sequenceDiagram
  participant MCP
  participant ChildProcess
  participant Stats

  ChildProcess-->>MCP: SIGCHLD (child exits)
  MCP->>MCP: check child_abort/max_messages/is_shutting_down
  alt should stop
    MCP->>MCP: set_state(STOPPED)
  else respawn
    MCP->>MCP: retire_poll_results(consumer, name)
    MCP->>Stats: fold counts into consumer_baselines
    MCP->>MCP: wait for next poll to respawn
  end

Loading

Possibly related PRs

• gmr/rejected#57: Modifies the same Controller.run() lifecycle and SIGTERM/SIGHUP signal-handling logic in rejected/controller.py.

Poem

A rabbit watched the signals fly,
STOPPED once meant "please don't retry."
Now flags are set, not fired in haste,
No orphaned kits left to waste. 🐇
Baselines held for counters true,
Hop, sleep, check — the run loop's new!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes to daemon signal handling and stats growth.
Linked Issues check	✅ Passed	The changes address #76, #89, #90, #91, #92, and #99 as described in the linked objectives.
Out of Scope Changes check	✅ Passed	The PR stays focused on lifecycle, shutdown, stats, and state-transition fixes with matching test updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[gmr]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.