Abandoned worker processes wedge interpreter exit via the multiprocessing atexit join — Closes #294

[conradbzura]

Summary

WorkerPool teardown abandons a worker that fails to stop within shutdown_timeout — by design. But WorkerProcess extends multiprocessing.Process without the daemon flag, and multiprocessing registers an atexit handler that joins every live non-daemonic child. An abandoned worker never exits on its own: its parent-death watchdog fires only when the parent dies, which the join prevents. The parent blocks in os.waitpid indefinitely, and a one-test flake becomes a whole-suite hang.

Spawn WorkerProcess daemonic by default. The same atexit handler terminates daemonic children before joining them, and the worker's SIGTERM handler already turns that termination into an immediate graceful stop, so the join completes promptly. The parent-death watchdog covers the reverse direction, so neither process outlives the other regardless of which exits first.

One trade-off: daemonic processes cannot spawn multiprocessing children, so a routine that must create subprocesses requires a worker constructed with daemon=False. The class docstring documents the contract and the escape hatch.

Two adjacent hardenings ride along: the load-sensitive Hypothesis dispatch exploration gets its own 90s per-example bound (its timeout cancellation is what strands workers in the first place), and a coverage audit of the pool teardown suite surfaced a silently broken mock, repaired here with the previously untested abandonment branches now covered.

Closes #294

Proposed changes

Spawn workers daemonic

Set daemon=True in WorkerProcess.__init__ via kwargs.setdefault, so an explicit daemon=False still wins. Document the contract in the class docstring: atexit terminates daemonic children before joining them, the SIGTERM handler turns that into a graceful stop, the watchdog guarantees the reverse direction, and daemonic workers cannot spawn multiprocessing children of their own.

Pin the contract at both levels

Add unit tests for the daemonic default and the daemon=False override. Add an integration regression guard: a helper process starts a LocalWorker and reaches interpreter exit without stopping it; the guard requires a clean exit within a bound and the worker terminated. Without the daemon default the guard wedges for its full 30s bound. Add a SIGTERM test asserting a started worker exits with code 0 inside the grace window — the graceful-stop mechanism the atexit terminate-then-join relies on. The helper scripts must remain python -c strings: they carry no __main__ guard, so as files the spawn bootstrap would re-run them in every worker child.

Repair hollow pool teardown coverage

MockWorker.stop did not accept the timeout keyword the pool passes at teardown; the TypeError disappeared into gather(return_exceptions=True), so mock workers were never actually stopped and the teardown test asserted nothing. Accept the keyword, rewrite the test to assert workers are genuinely stopped, and cover the remaining abandonment branches: a stop that raises TimeoutError counts as abandoned, shutdown_timeout is forwarded to each worker's stop, shutdown_timeout=None waits unbounded, and the abandonment warning names the abandoned uid.

Give the Hypothesis exploration timeout headroom

Split _HYPOTHESIS_TIMEOUT = 90 from the shared 30s bound. A heavy example on a loaded machine overran the shared bound, and the resulting asyncio.timeout cancellation is exactly the abandonment path this PR hardens. The deterministic pairwise suite keeps 30s.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestWorkerProcess	No custom parameters	WorkerProcess is instantiated	The process is daemonic	Daemonic default
2	TestWorkerProcess	An explicit daemon=False kwarg	WorkerProcess is instantiated	The process remains non-daemonic	Override escape hatch
3	TestWorkerOrphanPrevention	A parent that started a LocalWorker and never stops it	The parent interpreter exits	The parent exits cleanly within a bound and the worker is terminated	Interpreter-exit regression guard
4	TestWorkerOrphanPrevention	A started real WorkerProcess	SIGTERM is sent to the worker	The worker exits with code 0 inside the grace window	Graceful SIGTERM stop
5	TestWorkerPool	A pool with two spawned mock workers	The async-with block exits	Every worker is stopped with its metadata cleared	Teardown stops spawned workers
6	TestWorkerPool	A worker whose stop never returns	The shutdown bound elapses	The warning names the abandoned count and uid	Abandonment log content
7	TestWorkerPool	A worker whose stop raises TimeoutError under a generous bound	The async-with block exits	The abandonment warning is emitted promptly	Timeout-counted abandonment branch
8	TestWorkerPool	A pool with shutdown_timeout=5.0	The async-with block exits	The worker's stop is awaited once with timeout=5.0	Per-worker timeout forwarding
9	TestWorkerPool	A pool with shutdown_timeout=None	The async-with block exits	Stop is awaited with timeout=None and no warning is emitted	Unbounded teardown path