#420: persisted MaxConcurrentRebuildsPerDatabase default + capped rebuild fan-out regression test#463
Merged
Conversation
…ression test Builds on the already-merged --max-concurrent CLI flag with the configured-default half of #420: - IEventStore.MaxConcurrentRebuildsPerDatabase (store-agnostic, default null = unbounded). Concrete stores override to a derived default; Marten/Polecat will derive it from the Npgsql pool size (max(1, poolSize / 8)) as follow-up — that's the only place a pool-size signal exists, since JasperFx.Events is store-neutral. - ProjectionInput.ResolveMaxDegreeOfParallelism(int? configuredDefault): precedence is CLI flag > configured store default > unbounded. Non-positive at any level is coerced to -1 so a misconfigured cap can never wedge a rebuild (MaxDegreeOfParallelism of 0 throws). - ProjectionHost.TryRebuildShardsAsync passes the store default through, and the capped fan-out is extracted to internal RebuildProjectionsWithCapAsync so the cap is independently regression-testable. Also switches the per-cell error collection from List<Exception> to ConcurrentBag — with cap > 1 the lambda runs on multiple threads, so the old List.Add raced (latent bug, only reachable now that the loop is capped). Tests (21 green, net9.0 + net10.0): resolution precedence (flag/default/unbounded + non-positive coercion) and an instrumented peak-concurrency regression — 256 cells (8 projections x 32 tenants) at cap 4 never exceeds 4 and actually reaches 4; cap 1 serializes; non-positive cap runs unbounded without deadlock; every cell runs once. Follow-up (tracked, out of scope here): Marten/Polecat pool-size default derivation; extending the cap to the intra-projection tenant/shard cross-product fan-out in JasperFxAsyncDaemon; docs in rebuilding.md; the EnableExtendedProgressionTracking default-derivation analysis (the cap bounds cell COUNT and is orthogonal to per-cell progression writes, so it composes cleanly — extended tracking informs the derived default magnitude, not the throttle mechanism). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rrent-rebuilds # Conflicts: # src/JasperFx.Events/IEventStore.cs
…ncurrentRebuildsPerDatabase knob + EventStoreUsage descriptor field (delivers #434) - DaemonSettings.MaxConcurrentRebuildsPerDatabase (int?, null = store-derived default) sits beside the WS2/WS3 governors; inherited by Marten as Options.Projections.MaxConcurrentRebuildsPerDatabase via ProjectionGraph - EventStoreUsage.MaxConcurrentRebuildsPerDatabase so monitoring tools (CritterWatch#309) can read the effective cap off the wire — #434 was closed but the field never landed; delivered here - IEventStore.MaxConcurrentRebuildsPerDatabase xml-doc now points stores at the DaemonSettings knob Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tStoreUsage descriptor defaults) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances #420 (cap concurrent projection rebuilds per database) — epic #486 WS3 remainder. The
--max-concurrentCLI flag + projection-level cap already landed onmain; this PR adds the configured-default half, the regression test the acceptance criteria call for, and (post-2.22.0 refresh) reconciles the surface with the WS2/WS3 daemon governors.What's here
DaemonSettings.MaxConcurrentRebuildsPerDatabase(int?, defaultnull) — the configuration knob, sitting beside the shippedMaxConcurrentEventLoadsPerDatabase/MaxConcurrentBatchWritesPerDatabasegovernors (Per-database event-load throttle (#494 Phase A, epic #486 WS2) #495/Per-database projection batch-write governor + bounded cross-tenant rebuild default (epic #486 WS3) #496). Because Marten'sProjectionOptionsderives fromDaemonSettingsviaProjectionGraph, this isStoreOptions.Projections.MaxConcurrentRebuildsPerDatabase— the exact surface Surface MaxConcurrentRebuildsPerDatabase on EventStoreUsage #434 and marten#4710 describe.null= "derive store-side"; zero/negative disables the cap.IEventStore.MaxConcurrentRebuildsPerDatabase— store-agnostic resolved default read by the CLI rebuild path (default-interface-member so out-of-tree stores keep compiling). JasperFx.Events has no notion of a connection pool, so concrete stores override this to consult the knob and fall back to a pool-derived default (Marten/Polecat:max(1, poolSize / 8)).EventStoreUsage.MaxConcurrentRebuildsPerDatabase(int?) — the descriptor field Surface MaxConcurrentRebuildsPerDatabase on EventStoreUsage #434 asked for (that issue was closed but the field never actually landed; delivered here) so CritterWatch#309's dispatcher can read the effective cap off the wire.ProjectionInput.ResolveMaxDegreeOfParallelism(int? configuredDefault)— precedence:--max-concurrentCLI flag (one-off ops override) > store configured default > unbounded. Non-positive at any level is coerced to-1, becauseParallelOptions.MaxDegreeOfParallelism = 0throws and would wedge a rebuild.ProjectionHost.TryRebuildShardsAsyncnow passesstore.MaxConcurrentRebuildsPerDatabasethrough, and the capped fan-out is extracted tointernal static RebuildProjectionsWithCapAsync(...)so the cap is independently testable.List<Exception>toConcurrentBag<Exception>. With a cap > 1 the rebuild lambda runs on multiple threads, so the priorList.Add(e)insideParallel.ForEachAsyncraced — only reachable now that the loop is actually capped/parallel.Tests (463 green, net9.0 + net10.0)
RebuildConcurrencyCapTests): 256 cells (8 projections × 32 tenants) at cap 4 — peak concurrency, tracked viaInterlocked, never exceeds 4 and actually reaches 4 (guards against a false pass where nothing ran in parallel); cap 1 serializes; non-positive cap runs unbounded without deadlock; every cell runs exactly once.RebuildCapDefaultsTests): knob defaults null (= store-derived), descriptor field defaults null and round-trips.Relationship to the shipped 2.22.0 governors
The WS2/WS3 governors (
MaxConcurrentEventLoadsPerDatabase,MaxConcurrentBatchWritesPerDatabase) bound continuous catch-up; this cap bounds the rebuild fan-out — the batchy one-off operation.CrossTenantRebuild.RebuildEverywhereAsyncalready got its boundedmaxParallelism = 4default in #496; this PR does not change it.Follow-up (out of scope here)
MaxConcurrentRebuildsPerDatabaseto consult the knob + derivemax(1, NpgsqlConnectionPoolSize / 8), and populate theEventStoreUsagefield inTryCreateUsage(marten#4710) — next PR in this train.JasperFxAsyncDaemon(today's cap is projection-level).rebuilding.md(two-layer concurrency model, default derivation, CLI override) + cross-refs from the multi-tenancy /UseTenantPartitionedEventsdocs.Companion CritterWatch orchestration story: JasperFx/CritterWatch#309 (depends on this cap being honored).
🤖 Generated with Claude Code