Shared executor pools: fix the thread-per-node explosion behind the red macOS CI leg

[ptesavol]

Fixes the macOS CI leg that has been red since milestone A landed (#73), and removes a scaling hazard for real applications.

The failure

Layer1ScaleTest.MultipleLayer1Dht and KademliaCorrectnessTest.CanFindCorrectNeighbors abort on the macOS runners with:

libc++abi: terminating due to uncaught exception of type std::__1::system_error:
thread constructor failed: Resource temporarily unavailable

Every DhtNode instance owned ~9 dedicated executor threads (PeerManager pings, DhtNode + PeerDiscovery recovery, Router routing, StoreManager replication, plus a server-side and a 20-thread client-side pool per RPC communicator). The 200–250-node tests peak at ~1900–2100 threads — beyond what the 3-core CI runners tolerate (a 10-core dev machine allows 16384 threads per process, so it never fired locally). The same profile would hurt real applications: trackerless-network creates one layer-1 node per stream partition, so thread count scaled linearly with subscriptions. The TS original runs all of this on one event loop; fixed-size shared pools are the C++ analogue.

New in streamr-utils

• SharedExecutors — two process-wide pools: worker() (general detached work: pings, recovery, replication, RPC dispatch) and background() (low OS priority; keeps the former per-Router nice-10 "routing runs whenever we have time" design from phase A4).
• SharedSerialExecutor — a per-owner serial view of a shared pool, replacing the former {1}-sized pools whose single thread WAS the ordering guarantee (and, in Router's case, the synchronization of its maps). All folly::SerialExecutor template machinery instantiates inside the SharedExecutors TU — instantiating it from an importing module TU trips the known BMI name-lookup pitfall (DistributedMutex internals).
• GuardedAsyncScope — AsyncScope with a close() gate for owners whose event handlers can still fire while stop() drains (folly forbids add() racing joinAsync()).

Conversions (teardown + ordering semantics preserved per class)

Class	Change	Teardown guarantee
PeerManager	pings → serial view of worker()	scope join in stop() outside the mutex — the phase-AA pattern, verbatim
DhtNode	rejoins → serial view	GuardedAsyncScope drained in stop() (strictly earlier than the former destructor join; a KBucketEmpty firing mid-drain is dropped by the gate)
PeerDiscovery, StoreManager, Router/RoutingSession	executor swap only	their detached tasks pin self via sharedFromThis, exactly as before
RpcCommunicatorServerApi	serial view	kept its existing CancellableAsyncScope destructor drain
RpcCommunicatorClientApi	restructured	request()/notify() now run the this-touching work as a scope task; the caller awaits only a promise-contract future that holds no this. The new destructor's scope drain replaces the former 20-thread pool's join — and a timed-out request can no longer leave a detached task referencing a dead communicator (detachOnCancel previously detached a this-capturing task whose only safety net was the pool destructor).

Verification

• Peak thread count for the failing tests: 1870 → 18 (Kademlia, 200 nodes) and 2075 → 21 (MultipleLayer1Dht, ~250 nodes), measured by sampling ps -M over full runs.
• No performance change: isolated pre-refactor runs took 229 s / 254 s; post-refactor 229 s / 256 s. (An apparent "43 s baseline" turned out to be a warm full-suite artifact — isolated single-process runs, as ctest/CI executes them, are the valid comparison.)
• Identical convergence: avg 6.84/8 correct neighbours (6.86 before).
• All unit + integration suites green in streamr-utils, streamr-proto-rpc, streamr-dht; the phase-AA-sensitive SimultaneousConnections/ConnectionLocking/MultipleEntryPointJoining tests pass repeated isolated runs; lint green in all three packages.

Notes

• This complements the ctest --timeout raise already on PR Phase B1: connectivity checking and signed peer descriptors #74 (the heavy tests legitimately run ~4 minutes in isolated processes; the old 300 s limit was sized when the suite took ~35 s).
• After this merges, Phase B1: connectivity checking and signed peer descriptors #74 gets rebased on it, and the WebsocketServerConnector::requestConnectionExecutor added by B1 gets the same conversion there (it lives only on that branch).

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for this team, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.