Skip to content

Add demand-driven coarse join scheduling - #967

Draft
ezrosent wants to merge 1 commit into
ezr-better-parallel-packed-triefrom
codex/demand-driven-join-scheduling
Draft

Add demand-driven coarse join scheduling#967
ezrosent wants to merge 1 commit into
ezr-better-parallel-packed-triefrom
codex/demand-driven-join-scheduling

Conversation

@ezrosent

Copy link
Copy Markdown
Contributor

Stacked on #959.

This isolates the demand-driven coarse scheduling change from the packed-executor integration:

  • schedule coarse join work when it becomes available
  • preserve the existing join results and packed representation
  • keep the scheduling policy independently reviewable

Validation:

  • cargo test -p egglog-core-relations
  • cargo fmt --check
  • cargo clippy -p egglog-core-relations --all-targets -- -D warnings

@codecov-commenter

codecov-commenter commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.87302% with 76 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.03%. Comparing base (3cfa98a) to head (d7ea2c3).

Files with missing lines Patch % Lines
core-relations/src/free_join/execute.rs 68.35% 75 Missing ⚠️
concurrency/src/threadpool/mod.rs 98.71% 1 Missing ⚠️
Additional details and impacted files
@@                         Coverage Diff                         @@
##           ezr-better-parallel-packed-trie     #967      +/-   ##
===================================================================
- Coverage                            87.13%   87.03%   -0.11%     
===================================================================
  Files                                   96       96              
  Lines                                33023    33261     +238     
===================================================================
+ Hits                                 28775    28948     +173     
- Misses                                4248     4313      +65     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq

codspeed-hq Bot commented Jul 30, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 5.39%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 36 untouched benchmarks
⏩ 227 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation tests[proof_testing_repro-665-set-union] 17.3 s 16.4 s +5.39%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing codex/demand-driven-join-scheduling (d7ea2c3) with ezr-better-parallel-packed-trie (c7fc1d7)2

Open in CodSpeed

Footnotes

  1. 227 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on ezr-better-parallel-packed-trie (3cfa98a) during the generation of this report, so fff72bb was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Introduce explicit global and worker-local spawning, local-first nested helping, private queues, and timed half-queue donation when workers stall. Cover the scheduler with deterministic tests and expose queue, donation, and stalled-time instrumentation.

For eligible cached top-index scans, preserve the physical shard partitioning and respond to idle workers by publishing bounded coarse batches to the global queue. Each batch executes its complete lower subtree serially for locality, and fixes its grain after the first demand signal to avoid geometric task proliferation.

In five-run paired graph-motif tests, this improves Q1 by 0.8%, 6.8%, and 8.7% at 8, 12, and 16 threads and Q2 by 2.4%, 6.6%, and 3.7%. CPU cost is effectively flat except for a 2.6% increase on Q2 at 12 threads, while scheduler idle time drops substantially.
@ezrosent
ezrosent force-pushed the codex/demand-driven-join-scheduling branch from df8ecce to d7ea2c3 Compare July 30, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants