Cap the intra-projection tenant/shard cross-product rebuild fan-out with one shared per-database budget (#497)

[jeremydmiller]

Closes #497 (the #420 leftover, epic #486 WS3). Follows #463/#496.

Problem

MaxConcurrentRebuildsPerDatabase was only enforced at the projection level in the CLI path (ProjectionHost.RebuildProjectionsWithCapAsync). A single projection on a tenant-partitioned store still fanned out its per-(tenant, shard) rebuild cells inside JasperFxAsyncDaemon without consulting the cap — projections rebuild --max-concurrent 2 on a 100-tenant store bounded concurrent projections at 2 while each tenant cross-product ran unchecked.

Worse, the intra-daemon replay concurrency was accidentally serialized at one: rebuildAgent held the daemon's single-slot agent-registry _semaphore across the entire ReplayAsync, so the existing Parallel.ForEachAsync fan-outs launched wide but replayed one cell at a time — any cap > 1 was unreachable inside a database.

Budget design

One shared SemaphoreSlim budget per daemon (= per database), acquired per rebuild cell.

• A cell = a single (projection, tenant/shard) replay = one rebuildAgent call. Each cell holds exactly one budget slot for the duration of its replay, regardless of which layer launched it (the CLI's projection-level Parallel.ForEachAsync, the intra-projection tenant cross-product, or CrossTenantRebuild.RebuildEverywhereAsync). Outer layers hold no slots, so the two layers share the bound instead of multiplying it (projection slots × tenant cells ≤ cap, never cap²) and nesting cannot deadlock.
• Resolution: DaemonSettings.MaxConcurrentRebuildsPerDatabase ?? IEventStore.MaxConcurrentRebuildsPerDatabase (store-derived, e.g. Marten/Polecat pool-size/8) at daemon construction; the --max-concurrent CLI flag overrides per operation via a new IProjectionDaemon.MaxConcurrentRebuildsPerDatabase DIM (no-op default) that ProjectionHost assigns with the SAME resolved value it uses for the projection-level layer.
• Non-positive = unbounded, and a non-positive value never reaches ParallelOptions.MaxDegreeOfParallelism (0 throws). With no budget, rebuildProjectionAllTenants keeps its historical sequential tenant walk.
• rebuildAgent now takes the agent-registry _semaphore only around the registry mutations (stop-if-running, _agents update), not across the replay — that is what makes a cap > 1 actually reachable.
• CrossTenantRebuild.RebuildEverywhereAsync's maxParallelism becomes int? defaulting to the daemon's budget when configured, falling back to the Per-database projection batch-write governor + bounded cross-tenant rebuild default (epic #486 WS3) #496 default of 4; explicit values (including non-positive = unbounded) still win. Its per-tenant rebuilds draw from the same cell budget either way.
• Hardening: VectorizedHighWaterMonitor's per-tenant readings dictionary is now lock-guarded — parallel tenant rebuild cells poll concurrently on top of the pre-existing OnNext/timer poll paths.

Regression tests

CrossProductRebuildCapTests drives the real JasperFxAsyncDaemon (real SubscriptionAgents, substituted store/database implementing ICrossTenantRebuildSource, tenant-partitioned detector) with a TCS-gated instrumented loader — cells park inside their LoadAsync until released, so over-admission is caught deterministically, no timing:

• 2 projections × 6 tenants rebuilt concurrently under cap 3: Interlocked peak concurrency never exceeds 3 and actually reaches 3; all 12 cells run exactly once
• the CLI-override path (daemon.MaxConcurrentRebuildsPerDatabase = n) replaces the budget
• non-positive/unset caps stay unbounded, complete without deadlock, and run every cell exactly once
• RebuildEverywhereAsync launch-width resolution matrix + a gated test that its default fan-out follows the daemon budget
• updated the Per-database projection batch-write governor + bounded cross-tenant rebuild default (epic #486 WS3) #496 pin in BatchWriteGovernorDefaultsTests to the refined default contract

Test results

• dotnet test src/EventTests/EventTests.csproj --framework net9.0 → 475 passed / 0 failed
• dotnet test src/EventTests/EventTests.csproj --framework net10.0 → 475 passed / 0 failed
• full solution dotnet build clean

Cross-refs: marten#4884 (rebuild load tests validating tuned defaults), JasperFx/CritterWatch#309 (orchestration consumer).

🤖 Generated with Claude Code

https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96