fix(daemon): contain background forks of shell nodes under a shell guard (#3472)

[harsh839]

Summary

Closes #3472. path: shell nodes run sh -c <args> with no dora code in them, so the in-node DORA_RUN_PARENT_PID orphan guard never arms. They were contained only by the daemon's spawn-time PR_SET_PDEATHSIG, which reaches the direct child alone: a shell that forks a background child (sh -c 'sleep 1000 & wait') leaks that child to ppid 1 when dora run is SIGKILLed.

Unix shell spawns are now routed through a hidden dora __shell-guard CLI subcommand, spawned as the daemon's direct child and process-group leader. On the dora run path it arms on the injected DORA_RUN_PARENT_PID and, once that process is gone, killpgs its whole group — guard, shell, and background forks together — mirroring the in-node orphan_guard (it also clears the daemon's PDEATHSIG once its own 500ms poll is running, exactly like the in-node guard does). When DORA_RUN_PARENT_PID is absent (dora up + dora start) it is a transparent passthrough that waits for the shell and forwards its exit status, preserving #2029. Windows cmd /C is unchanged.

Validation

• New e2e run_killed_by_sigkill_does_not_orphan_shell_nodes: a shell node publishes the shell's pid and a sleep 1000 & background child's pid, dora run is SIGKILLed, and both must be gone within 30s. Verified it fails without the fix (only the shell is PDEATHSIG-contained; the child survives) and passes with it.
• Sibling orphan tests still green: run_killed_by_sigkill_does_not_orphan_nodes, run_killed_before_node_init_does_not_orphan, run_killed_by_sigterm_terminates_nodes_and_exits.
• smoke_shell_node_allowed_with_flag / smoke_shell_node_blocked_without_flag still pass (gate intact).
• cargo clippy -p dora-cli -p dora-daemon --all-targets -- -D warnings, cargo fmt --all -- --check, cargo test -p dora-cli --lib (420) and -p dora-daemon --lib (280) all clean.
• Graceful stop paths are unaffected: the daemon's stop ladder already signals the node's whole process group (shutdown.rs), which now simply includes the guard as leader.

[trunk-io]

Merging to main in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[phil-opp]

🤖 Automated review (Claude) — first review of this PR. Fully automated, no human in the loop; advisory only, not an approval (I can't approve or merge). Reviewed from the diff/code as source of truth, not the description.

No blocking issues found. I verified the load-bearing pieces:

• The killpg is contained. contain() does killpg(getpgrp(), SIGKILL), which is only safe if the guard is its own process-group leader. It is: the daemon wraps every unix spawn with ProcessGroup::leader() (prepared.rs:774), so the guard's pgid == its own pid, and the shell + its background forks inherit that group. It cannot signal the daemon's group.
• No teardown regression. The daemon tears nodes down with group-based killpg(pid, SIGKILL) (shutdown.rs:196, keyed on the recorded node pid). The recorded pid is now the guard = the group leader, so the whole group (guard + shell + forks) still dies. dora stop / the SIGTERM→SIGKILL ladder are unaffected.
• Liveness test is exact on the dora run path. DORA_RUN_PARENT_PID is set to the daemon pid (spawner.rs:662) and the guard is the daemon's direct child, so getppid() == parent ⇒ direct_child = true; the fallback kill(parent, 0)/ESRCH probe (with its pid-recycle caveat) only applies off that path.
• The regression test genuinely fails on main. Without the guard, the daemon's PDEATHSIG SIGKILLs the shell alone; the background sleep 9527 reparents to init and survives, so the child-pid wait times out. With the guard, the group killpg takes both down. #[cfg(unix)], identity-checked teardown — solid.
• Clearing PR_SET_PDEATHSIG on the guard (it fires on spawning-thread death, not daemon death) before entering the poll loop is correct, and the pre-spawn parent check closes the "parent already gone" race.

One non-blocking simplification. The guard is inserted on the coordinator-attached (dora up) path too, where DORA_RUN_PARENT_PID is absent and the guard is a pure passthrough — it adds an extra long-lived dora process (a heavy binary) per shell node and buys no containment there (those nodes are meant to outlive the daemon). The daemon already knows which path it's on (it's the one that sets the marker via bind_nodes_to_parent), so it could skip the guard entirely on the dora up path — mirroring exactly how #3544 gates the Windows KillOnDrop. Failing that, exec-ing the shell in passthrough (instead of spawn-and-wait) would at least avoid the extra resident process. Not required for correctness.

Minor edge note (also non-blocking): containment only holds while the shell stays alive. A shell that forks a background child and then exits immediately (sh -c 'x &', no wait) makes the guard forward the exit and depart, so a later daemon death won't be contained — but that's arguably a finished node with a deliberately-detached child, so the behavior is defensible.

---

Generated by Claude Code

[harsh839]

Addressed the non-blocking suggestion: the shell guard is now scoped to the in-process dora run spawn path only.

path_spawn_command now takes bind_nodes_to_parent and skips the guard entirely on the coordinator-attached path — the daemon spawns sh -c directly there, since without DORA_RUN_PARENT_PID the guard could never arm (pure passthrough + one extra resident dora process per shell node, buying nothing for nodes that are meant to outlive the daemon per #2029). This mirrors the gating the pre-init orphan window in prepared.rs and the Windows KillOnDrop (#3544) use — all keyed on the same parent-bound signal.

The dora run path is unchanged (still routed through dora __shell-guard). Verified:

• run_killed_by_sigkill_does_not_orphan_shell_nodes still passes (guard active),
• a path: shell node under dora start runs plainly with no guard process (manual dora up-shape check),
• cargo fmt --all -- --check and cargo clippy -p dora-daemon -p dora-cli --all-targets -- -D warnings clean.

[phil-opp]

🤖 Automated review (fully automated Claude Code review — no human vetted this; advisory only, not an approval, and I can't approve or merge). Reviewed from the diff.

New commit c321550 since the last review — the bind_nodes_to_parent scoping (guard only on the in-process dora run path, plain sh -c on the coordinator-attached path) looks correct, and the e2e is a genuine on-main failure. But re-reading the whole diff surfaced a blocking issue the earlier pass missed:

This won't compile for Windows targets. In binaries/cli/src/command/mod.rs, mod shell_guard;, the use shell_guard::ShellGuardArgs;, the Command::ShellGuard(...) variant, its execute() arm, and the parse_shell_guard test are all added without a #[cfg(unix)] gate, so binaries/cli/src/command/shell_guard.rs is compiled on every target. Its body relies on Unix-only libc symbols with no Windows fallback:

• libc::getppid (armed, parent_is_gone), libc::kill / libc::ESRCH (parent_is_gone), libc::killpg / libc::getpgrp / libc::SIGKILL / libc::_exit (contain);
• clear_parent_death_signal() is defined only for #[cfg(target_os = "linux")] and #[cfg(all(unix, not(target_os = "linux")))] — there is no Windows definition, so armed()'s unconditional call to it is unresolved on Windows too.

None of these exist in libc's Windows bindings, so cargo build/check -p dora-cli fails for *-pc-windows-*. PR CI is Linux-only, so it stays green here — but nightly's Windows leg, the "CLI Tests (all platforms)" job, and the cross-compile matrix will break. (macOS is unaffected — it's Unix and takes the all(unix, not(linux)) arm.)

Suggest gating the shell_guard module + the ShellGuard variant/arm/parse_shell_guard test behind #[cfg(unix)] (with a Windows stub for ShellGuardArgs::execute, or dropping the subcommand on Windows entirely, since Windows shell spawns already go through cmd /C and are unchanged). The daemon side is already correctly gated (dora_guard_command has a #[cfg(not(unix))] → None arm), so only the CLI subcommand needs the same treatment.

Otherwise the mechanism holds up: contain()'s killpg(getpgrp()) is safe because the daemon makes the guard the process-group leader, the _exit (not exit) choice avoids destructor deadlock, and the regression test genuinely fails on main (the background sleep 9527 reparents to init and survives the shell's PDEATHSIG).

---

Generated by Claude Code

[harsh839]

Fixed the Windows-compile blocker: shell_guard is now gated behind #[cfg(unix)] across the wiring — mod shell_guard, the use, the Command::ShellGuard variant + its execute() arm, and the parse_shell_guard test. This matches the daemon side, where dora_guard_command already has a #[cfg(not(unix))] → None arm; Windows cmd /C shell spawns are unchanged and never reference the guard.

Verified: cargo build -p dora-cli, cargo clippy -p dora-cli --all-targets -- -D warnings, cargo fmt --all -- --check all clean, parse_shell_guard passes. I attempted cargo check -p dora-cli --target x86_64-pc-windows-msvc locally, but the known pre-existing aws-lc-sys build-script blocker (zenoh dependency, pthread on a Windows target) aborts the whole graph before any dora code — same one that already affects the Windows leg, so the Windows CI runs on real MSVC remain the gate for this.

[phil-opp]

🤖 Automated review (fully automated Claude Code review — no human vetted this; advisory only, not an approval, and I can't approve or merge). Reviewed from the diff.

Re-reviewed at 74afbd7. The new commit gates the shell_guard module, the Command::ShellGuard variant / execute() arm, and the parse_shell_guard test behind #[cfg(unix)], which resolves the Windows compile break flagged earlier — clear_parent_death_signal now has full unix coverage (linux + all(unix, not(linux))) and the daemon side was already #[cfg(not(unix))] → None. I re-checked the core safety property independently: every unix spawn is wrapped with ProcessGroup::leader() (prepared.rs), so the guard is its own group leader and contain()'s killpg(getpgrp(), SIGKILL) can only reach the shell + its forks, never the daemon's group. No issues found in this revision.

---

Generated by Claude Code

[harsh839]

@phil-opp Could you proceed with this merge? Claude re-review is clean (no blocking issues) and all CI checks are green. If it's in the queue for trunk, please submit it — or let me know if the author-side /trunk merge is expected to self-serve. Thanks!

[phil-opp]

🤖 Automated review by Claude — fully automated review; no human has vetted this. Advisory only, posted like an outside contributor — I can't approve or merge.

One blocking issue that the earlier passes and PR CI didn't surface (the Test (ubuntu-latest) job runs in the merge queue, not on the PR head, so it shows skipped here):

The new hidden __shell-guard subcommand isn't recorded in cli-surface.txt, so the surface-snapshot test will fail. binaries/cli/tests/cli_surface.rs builds the surface by walking command.get_subcommands() with no filter on hidden or __-prefixed commands — hidden commands are intentionally included (dora daemon / dora coordinator, both hide = true, are already recorded, lines 49/55). Command::ShellGuard is #[cfg(unix)] and registered as __shell-guard, so on Linux the generated surface gains a dora __shell-guard line (plus its positional <command>) that isn't in binaries/cli/cli-surface.txt, and cli_surface_snapshot_is_current panics with "added (fine, but record it)". That blocks the merge-queue Test run.

Worth deciding how to resolve rather than just regenerating the snapshot: cli-surface.txt is the frozen-1.0 surface where "removing any line is a breaking change", so recording __shell-guard would freeze an internal implementation detail into the 1.0 guarantee. Excluding __-prefixed internal subcommands from the walk (and noting that in the snapshot header) seems more appropriate than pinning the guard command.

---

Generated by Claude Code

[harsh839]

Addressed the blocking surface-snapshot issue — new head 605c83c.

cli_surface.rs now skips subcommands whose name starts with __ in the walk, exactly as suggested: the frozen 1.0 surface records dora daemon/dora coordinator (scripted against) but not the unix-only dora __shell-guard spawn wrapper, which would otherwise become a pseudo-breaking change for every non-unix platform. The exclusion is documented in the snapshot header.

Verified: cargo test -p dora-cli --test cli_surface passes (was failing with "added: dora __shell-guard"), snapshot_covers_the_command_tree still green, clippy/fmt clean, parse_shell_guard still passes. From here on I'll run the full cargo test -p dora-cli (not just the --lib subset) before pushing, since the surface test only runs over the whole suite.

[phil-opp]

🤖 Automated review by Claude — fully automated review, no human vetted this; advisory only, not an approval.

Re-reviewed at 605c83c (new commit since my last review). The cli-surface snapshot blocker I flagged is resolved correctly: binaries/cli/tests/cli_surface.rs's walk() now skips __-prefixed subcommands (if sub.get_name().starts_with("__") { continue; }), so dora __shell-guard is neither generated nor recorded, and the HEADER const + cli-surface.txt header were updated in lockstep so cli_surface_snapshot_is_current stays consistent. This is the right fix for the 1.0 surface freeze — recording a unix-only internal wrapper line would have frozen a pseudo-guarantee that doesn't even exist on non-unix, and the __-prefix exclusion auto-covers any future internal subcommand. No new issues.

---

Generated by Claude Code