fix(neo4j): reuse one bounded driver across per-session graph stores

[Diego Colombo (colombod)]

Problem — bolt-driver resource exhaustion

The drain path built a new AsyncGraphDatabase driver for every session_id. Each of those drivers ran its own connection pool at the neo4j-python default of 100, so the pool was bounded but the number of pools was not. Per-session drivers are closed only on clean finalize, cancel, or the 5-day idle reap — none of them reachable while Neo4j is under load — so bolt connections accumulated until the server's bolt thread pool starved and ingest backpressured to clients.

This is a standalone, ship-first extraction of the resource-exhaustion fix, off main; the rest of the hardening stack rebases on top of it.

Fix

• SessionRegistry builds one shared, pool-bounded Neo4j driver (lazily, on first session) and injects it into every Neo4jGraphStore it constructs.
• Neo4jGraphStore accepts an optional pre-built driver and tracks owns_driver; close() only closes a driver it owns — so a per-session finalize can never tear down the driver other live sessions are still using.
• The shared driver is closed exactly once, at lifespan shutdown, and only after the drain workers have been quiesced (see below).
• Driver kwargs live in one helper (build_bounded_neo4j_driver in neo4j_store.py) so the admin driver, the query driver, the doctor CLI, and the registry's shared driver can never diverge.
• New setting: neo4j_max_connection_pool_size (default 50, validated > 0). The neo4j driver's own default is 100; 50 is a deliberate reduction, well under the server's default bolt thread-pool size, now that every session shares one pool instead of holding a private one.

There is deliberately no max_connection_lifetime setting: the driver already recycles pooled connections at 3600 s by default, so a knob whose default equalled the library default would change nothing.

Shutdown: quiesce the drainers before closing the shared driver

Sharing the driver makes its lifetime a cross-session concern, and that turns process shutdown into a correctness problem it never was before.

A drain worker that meets a closed shared driver fails its batch, spends its max_delivery_attempts budget in ~250 ms (5 attempts × the 50 ms _DRAIN_POLL_INTERVAL backoff), and lands in _handle_exhausted_batch — which dead-letters each line and commits the queue offset past it. Those are healthy events that merely happened to be queued when the process stopped, and once dead-lettered they never replay.

SessionRegistry.shutdown_workers() cancels and awaits every drain worker, and the lifespan finally calls it before any driver closes. Cancellation routes each drainer through its existing CancelledError handler → _safe_close(worker) → a final flush on a driver that is still open; anything left uncommitted stays in the durable queue and replays on the next boot.

Measured against a live Neo4j, 400 events queued to one session with the drainer running:

shutdown sequence	events dead-lettered
driver left open (control)	0 of 400
close the shared driver with drainers still live	100–400, in 8 of 8 trials (one trial: all 400)
quiesce, then close (this PR)	0 of 400, in 6 of 6 trials

Driver-kwarg parity

Routing per-session driver construction through the shared helper must not quietly change how those drivers behave. build_bounded_neo4j_driver carries over both kwargs the per-session driver set:

kwarg	old per-session driver	shared driver	lifespan admin/query
max_connection_pool_size	100 (library default)	50	50
connection_acquisition_timeout	30.0 (neo4j_lock_timeout)	30.0	60.0 (library default, unchanged)
max_transaction_retry_time	30.0	30.0	30.0

connection_acquisition_timeout matters more after this change, not less: one bounded pool is now shared by every session, so waiting on a free connection is reachable in a way it was not when each session held a private 100-slot pool. The lifespan admin and query drivers pass no acquisition timeout, preserving exactly what they did before they were routed through the helper.

Evidence (live Neo4j)

The proving test drives 30 concurrent sessions through get_or_create against a real Neo4j container and counts actual Python bolt connections on the shared driver:

[driver-leak evidence] sessions=30 pool_size=8 baseline=0 during_load=8 after_close=0

• Bound to the pool, not one-per-session: during_load (8) < SESSION_COUNT (30) and <= pool_size + 2. One shared driver served all 30 sessions.
• No hanging drivers / full reclaim: after_close == 0 — every bolt connection released once the shared driver is closed.
• Per-session close is safe: test_session_a_close_does_not_disrupt_session_b proves closing one session leaves the shared driver working for others.
• Shutdown discards nothing: test_shutdown_quiesce_then_close_deadletters_nothing dead-letters 0 events; removing the quiesce makes it fail with 200 events dead-lettered and error="Driver closed".

Each regression guard was verified to actually fail without its fix: deleting the driver injection makes the live leak test report 30 open connections against a pool bound of 8, and makes 5 of the CI-run unit tests in tests/test_neo4j_driver_sharing.py fail.

Test results

• tests/neo4j/ (live container) — 92 passed
• Full -m "not neo4j" suite — 1945 passed, 8 skipped
• ruff check clean; ruff format --check clean on every changed file

Notes

• +998 / −56 across 11 files. Merged with main after the durable-queue hardening work landed; re-verified against the reworked drain path.
• Extracted from stack commits 75e2fdd + f4ef6b5; incidental edits to test_boot_safety.py / test_writer_lease.py were dropped because those files do not exist on main.

🤖 Generated with Amplifier