Prevent LocalDiscovery teardown from crashing on shared-memory double-unlink

[conradbzura]

Description

LocalDiscovery backs pool discovery with a named POSIX shared-memory segment whose name is deterministic per namespace (_short_hash(self._namespace)). Tearing down a pool can crash with FileNotFoundError raised by shm_unlink, and the crash becomes reliable under rapid pool teardown+respawn that reuses a namespace.

Observed traceback (during WorkerPool.__aexit__):

LocalDiscovery.__exit__  ->  self._address_space.unlink()
  ->  multiprocessing/shared_memory.py  unlink
  ->  _posixshmem.shm_unlink
  ->  FileNotFoundError: [Errno 2] No such file or directory: '/<hash>'

A second FileNotFoundError then fires from the atexit-registered cleanup at interpreter shutdown, alongside resource_tracker: There appear to be N leaked shared_memory objects warnings.

Reproduce: create a WorkerPool, tear it down, and immediately create another with the same namespace, repeatedly (e.g. a benchmark harness that calls reset()/respawns a pool per iteration). A pool that starts and stops exactly once usually gets lucky; overlapping same-namespace lifecycles trip it consistently.

Expected Behavior

Pool teardown — including two overlapping pools that share a namespace — unwinds cleanly. Unlinking a shared-memory segment that is already gone is a no-op, not a fatal error, and the atexit fallback never double-fires.

Root Cause

Three interacting issues in src/wool/runtime/discovery/local.py:

1. __exit__ unlinks unguarded (lines ~246-252). If the segment is already gone, shm_unlink raises FileNotFoundError, which propagates out of __exit__ and aborts WorkerPool.__aexit__:

   def __exit__(self, *_):
       if self._owner:
           self._address_space.close()
           self._address_space.unlink()        # can raise FileNotFoundError
           atexit.unregister(self._cleanup)     # skipped if unlink() raised
       else:
           self._address_space.close()
   

2. The atexit fallback double-fires. The owner registers self._cleanup = atexit.register(lambda: self._address_space.unlink()) in __enter__ (line ~241). __exit__ cancels it (line ~250) only after unlink() — so if unlink() raises, the cancel never runs and the lambda fires a second FileNotFoundError at shutdown.

3. The segment name is deterministic per namespace and the CPython multiprocessing resource_tracker also unlinks tracked segments. With multiple attaches (owner + non-owners) and multiple unlink paths (__exit__, the atexit lambda, and the tracker), the segment can only be unlinked once — whoever runs second/third hits "no such file." This is the known CPython shared_memory behavior (bpo-38119), which also produces the "leaked shared_memory objects" warnings.

Suggested fix (small, defensive):

• Wrap the unlink in contextlib.suppress(FileNotFoundError) — unlinking an already-removed segment should be a no-op.
• Call atexit.unregister(self._cleanup) before unlink(), so a failed unlink can't leave the fallback armed to double-fire.
• Prefer unlinking exactly once from the owner and suppressing not-found elsewhere, rather than relying on the resource_tracker auto-unlink for this multiply-attached, deterministically-named segment (bpo-38119).

A sibling atexit-registered shared-memory cleanup pattern at ~lines 548/566 (self._cleanups[name] = atexit.register(...)) should be audited for the same double-unlink hazard.