Per-database projection batch-write governor + bounded cross-tenant rebuild default (epic #486 WS3)#496
Merged
Conversation
…ebuild default (epic #486 WS3) Measurement on #494 attributed ~29 of ~35 steady-state daemon connections to per-agent projection-batch COMMIT sessions, with the racy cold-start burst (every (projection x tenant) agent committing its first batch near-simultaneously) freezing the Npgsql pool anywhere between ~35 and ~99 connections across identical runs. Both projection execution flavors now acquire a daemon-owned write slot around the batch execute/commit round-trip (applyBatchOperationsToDatabaseAsync), sized by the new DaemonSettings.MaxConcurrentBatchWritesPerDatabase (default 4; <= 0 disables). The governor reaches the executions through default interface members - IDaemonRuntime.BatchWriteThrottle -> SubscriptionAgent -> EventRange.Agent - so no implementation of either interface breaks. Only BatchBehavior.Individual ranges execute batches (composite members ride the parent's batch), so the slot economy cannot deadlock on nested batches. A shard torn down while queued for a slot disposes its batch and exits without marking success or reporting a failure. Also gives CrossTenantRebuild.RebuildEverywhereAsync a real default (maxParallelism 4, was unbounded) - a 100-tenant store no longer fans out 100 concurrent rebuilds against one database by default; 0 opts back into unbounded. EventTests 449/449 green on net9.0 + net10.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 7, 2026
This was referenced Jul 7, 2026
jeremydmiller
added a commit
that referenced
this pull request
Jul 7, 2026
…-out with one shared per-database budget (#498) The #420 leftover (epic #486 WS3): MaxConcurrentRebuildsPerDatabase was only enforced at the projection level in the CLI path, so a single projection on a tenant-partitioned store still fanned out its per-(tenant, shard) rebuild cells without consulting the cap. - JasperFxAsyncDaemon owns ONE shared SemaphoreSlim rebuild budget per daemon (= per database), resolved at construction from DaemonSettings.MaxConcurrentRebuildsPerDatabase ?? IEventStore's store-derived default; every rebuild cell (rebuildAgent) draws a slot for the duration of its replay, so the CLI's projection-level layer and the intra-projection tenant cross-product never multiply the bound - rebuildAgent no longer holds the agent-registry _semaphore across the whole replay (that shape serialized every rebuild cell at an effective concurrency of one, making any cap > 1 unreachable); the lock now guards only the registry mutations - rebuildProjectionAllTenants fans tenants out in parallel at the budget's launch width when a budget is configured; null/non-positive keeps the historical sequential walk (and never reaches ParallelOptions.MaxDegreeOfParallelism, which throws on 0) - IProjectionDaemon grows a MaxConcurrentRebuildsPerDatabase DIM (no-op default) and ProjectionHost threads the SAME resolved --max-concurrent / store-default cap into the daemon before rebuilding - CrossTenantRebuild.RebuildEverywhereAsync's maxParallelism defaults to the daemon's budget when configured, falling back to the #496 default of 4; explicit values (including non-positive = unbounded) still win - VectorizedHighWaterMonitor._current is now lock-guarded: parallel per-tenant rebuild cells drive concurrent polls on top of the pre-existing OnNext/timer poll paths, and an unguarded Dictionary corrupts under concurrent writes - Regression tests drive the REAL daemon with a TCS-gated instrumented loader: peak concurrency over (projections x tenants) cells never exceeds the cap AND actually reaches it, every cell runs exactly once, non-positive caps stay unbounded without deadlock Closes #497 Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The write-side half of the connection-conservation work (#494 landed the read side as #495). Epic #486 WS3.
Why
Measurement on #494 attributed ~29 of ~35 steady-state daemon connections to per-agent projection-batch COMMIT sessions, with the racy cold-start burst (every (projection × tenant) agent committing its first batch near-simultaneously) freezing the Npgsql pool anywhere between ~35 and ~99 connections across identical 200-agent runs.
What
applyBatchOperationsToDatabaseAsync), sized byDaemonSettings.MaxConcurrentBatchWritesPerDatabase(default 4; ≤0 disables).IDaemonRuntime.BatchWriteThrottle→SubscriptionAgent→EventRange.Agent— so no implementation of either interface breaks.BatchBehavior.Individualranges execute batches (composite members ride the parent's batch and never reach the seam), so the slot economy cannot deadlock on nested batches. A shard torn down while queued for a slot disposes its batch and exits without marking success or reporting a spurious failure.CrossTenantRebuild.RebuildEverywhereAsyncgets a realmaxParallelismdefault (4, was unbounded — a 100-tenant store fanned out 100 concurrent rebuilds against one database); passing 0 opts back into unbounded.Measurement (daemonload, 100 tenants × 2 projections = 200 agents, 120s @ ~190 ev/s)
12 ≈ 4 load slots + 4 write slots + 4 appender writers + high-water — the WS2/WS3 O(databases) gate, met. EventTests 449/449 green on net9.0 + net10.0.
🤖 Generated with Claude Code