Skip to content

#499: terminate coordinator loop on disposed data source / wrapped cancellation#500

Merged
jeremydmiller merged 1 commit into
mainfrom
fix-499-coordinator-drain-race
Jul 8, 2026
Merged

#499: terminate coordinator loop on disposed data source / wrapped cancellation#500
jeremydmiller merged 1 commit into
mainfrom
fix-499-coordinator-drain-race

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Fixes #499.

Problem

ProjectionCoordinatorBase.executeAsync's per-set error handler treated any exception from TryAttainLockAsync as a transient "will retry later" and looped again after Task.Delay. During host shutdown, a TryAttainLockAsync that faults because the underlying NpgsqlDataSource is being disposed got re-polled instead of terminating — amplifying one racing OpenAsync into the burst of ObjectDisposedException: 'Npgsql.PoolingDataSource' aborts measured in marten#4874. This is priority 2 (case B) of that issue — the root-cause drain correctness fix (the priority-1 symptom guard lives in weasel#349).

Change

In the inner lock-attempt catch of ProjectionCoordinatorBase.executeAsync:

  • ObjectDisposedException is now terminal — the data source is gone for good, so the loop returns instead of re-polling a dead pool.
  • Check stoppingToken.IsCancellationRequested before the retry Task.Delay — a cancellation that surfaces wrapped (not a bare OperationCanceledException) now terminates the loop the same way the outer catch already does, rather than being logged as a lock error and re-polled.

Tests

Two regression tests in ProjectionCoordinatorBaseTests, both confirmed to fail against the pre-fix loop:

  • terminates_the_loop_when_the_data_source_is_disposed_mid_lock_attempt — a disposed-pool fault stops the loop after exactly one attempt (a re-polling loop would keep incrementing the attempt counter).
  • terminates_on_a_wrapped_cancellation_during_a_lock_attempt_without_logging_a_lock_error — an in-flight attempt whose cancellation surfaces wrapped terminates without logging a "will retry later" lock error (asserted via a capturing logger).

All 13 coordinator tests pass with the fix.

🤖 Generated with Claude Code

…urce / wrapped cancellation

ProjectionCoordinatorBase.executeAsync's per-set error handler treated any
exception from TryAttainLockAsync as a transient "will retry later" and looped
again after Task.Delay. During host shutdown that turned one racing OpenAsync
against a disposing NpgsqlDataSource into the burst of ObjectDisposedException
aborts measured in marten#4874 (case B — the root-cause drain correctness fix).

The inner catch now:
- returns out of the loop on ObjectDisposedException (the data source is gone
  for good; re-polling only hammers a dead pool), and
- checks stoppingToken.IsCancellationRequested before the retry Delay, so a
  cancellation that surfaces wrapped (not a bare OperationCanceledException)
  terminates the loop the same way the outer catch already does.

Adds two regression tests that fail against the pre-fix loop: one proves a
disposed-pool fault stops the loop without re-polling, the other proves a
wrapped shutdown cancellation terminates without logging a lock error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremydmiller jeremydmiller merged commit 83a1a05 into main Jul 8, 2026
1 check passed
@jeremydmiller jeremydmiller deleted the fix-499-coordinator-drain-race branch July 8, 2026 15:24
jeremydmiller added a commit that referenced this pull request Jul 8, 2026
…n-race fix, #499/#500)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProjectionCoordinatorBase.executeAsync: terminate on cancellation / disposed data source instead of re-polling (shutdown drain race)

1 participant