parallel_scheduler for P2079 Section 4.1 User Facing API (ref #6601) by charan-003 · Pull Request #6655 · TheHPXProject/hpx

charan-003 · 2025-03-27T21:44:20Z

Implement P2079R7 parallel_scheduler in HPX Core

This issue tracks the implementation of hpx::execution::experimental::parallel_scheduler in HPX core, aligning with P2079R7 (Parallel scheduler, Section 4.1) by wrapping thread_pool_policy_scheduler with hpx::launch policies. The goal is to provide a standards-compliant parallel scheduler with global access, cancellation support, task chaining, and bulk operations.

Implementation Checklist

Phase 1: Core Implementation
Implement hpx::execution::experimental::parallel_scheduler class per P2079R7 Section 4.1.
Add schedule() returning parallel_sender with sender_concept and completion_signatures:
Supports set_value_t(), set_stopped_t(), set_error_t(std::exception_ptr).
Wrap thread_pool_policy_scheduler with hpx::launch (async/sync policies).
Implement noexcept move/copy constructors and assignment operators.
Add operator== returning true for scheduler comparison.
Return forward_progress_guarantee::parallel via get_forward_progress_guarantee_t.
Implement then operation for task chaining with parallel_sender.
Code to rely on STDEXEC.
Phase 2: Bulk Support
Extend bulk_t customization to delegate to thread_pool_scheduler_bulk.
Implement bulk_sender for parallel bulk execution per P2079R7.
Support bulk_chunked and bulk_unchunked operations.
Phase 3: Replaceability API
Implement std::execution::system_context_replaceability namespace per P2079R7 Section 4.2.
Add query_parallel_scheduler_backend() returning shared_ptr<parallel_scheduler>.
Support link-time replaceability using weak symbols.
Implement receiver and bulk_item_receiver interfaces for frontend-backend interaction.
Ensure storage handling for scheduling operations.
P3927R2 extends C++26's task_scheduler (a type-erased scheduler wrapper) to support efficient parallel bulk operations. Previously, bulk work executed through task_scheduler would run sequentially even when wrapping a parallel_scheduler, defeating the purpose of parallel execution.
P3804R2 refines the parallel_scheduler specification (from P2079) to clarify execution policy semantics and eliminate implementation ambiguities
- seq policy: Backend receives count=1, executes all work in a single call (truly sequential)
- par policy: Backend receives count=shape, distributes work across multiple calls/threads (truly parallel)

StellarBot · 2025-03-27T21:45:11Z

Can one of the admins verify this patch?

codacy-production · 2025-03-28T01:10:52Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.45%	✅ 94.84%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`6e3a98b`)	193599	164650	85.05%
Head commit (`1267582`)	214837 (+21238)	183682 (+19032)	85.50% (+0.45%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6655)	155	147	94.84%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

charan-003

clang format

charan-003 · 2025-03-31T16:05:40Z

Update:

Moved parallel_scheduler, parallel_sender, and bulk_sender to a new header: hpx/executors/parallel_scheduler.hpp
Removed related code from thread_pool_scheduler.hpp

@hkaiser — ready for re-review when convenient. Thanks for the guidance!

charan-003 · 2025-04-01T09:02:44Z

@hkaiser Can you please verify it now? I'm not sure why those 2 tests are failing.

hkaiser

Wow, I'm impressed! I have a couple of minor comments, though.

hkaiser · 2025-04-02T01:23:17Z

Also, you may want to move all types that are don't have to visible by users into namespace detail to avoid name clashes with other code.

charan-003 · 2025-04-02T03:00:33Z

Also, you may want to move all types that are don't have to visible by users into namespace detail to avoid name clashes with other code.

Sure, I'll do that soon
Thank you so much

charan-003 · 2025-04-03T19:25:28Z

i'm not sure why it's failing for clang_format. i used clang-format -i ` to make sure it's in the format

charan-003 · 2025-04-03T19:30:06Z

@hkaiser
The parallel_bulk_operation_state implements a custom chunking strategy for bulk operations. It divides the workload into chunks based on the number of threads in the underlying thread_pool_scheduler’s pool (get_os_thread_count()), then schedules each chunk as a separate task using exec.execute()

The custom bulk chunking might duplicate functionality in thread_pool_scheduler_bulk.hpp

Offers a parallel execution option that leverages HPX’s thread pool, maintaining high performance for CPU-bound tasks.

While this version lacks P2079R7’s replaceability API (section 4.2) and many more, it could be extended to support it ( query_parallel_scheduler_backend()). This would let HPX users swap in custom schedulers (integrating with GPUs or other runtimes like oneTBB), enhancing HPX’s flexibility and composability

will try to work on this :))

codacy-production · 2025-04-03T23:44:06Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.30%	✅ 88.81%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`15afc8b`)	244963	210174	85.80%
Head commit (`7025dfd`)	219167 (-25796)	187374 (-22800)	85.49% (-0.30%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6655)	268	238	88.81%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

hkaiser · 2025-04-04T14:41:22Z

i'm not sure why it's failing for clang_format. i used clang-format -i ` to make sure it's in the format

The CI uses an older version of clang-format that does not always agree with newer versions on how to format things. Simply surround the offending lines with

// clang-format off
    ....
// clang-format on

to disable it checking those.

charan-003 · 2025-04-14T22:14:51Z

@hkaiser @isidorostsa
This version is about adding P2079R7 parallel_scheduler to HPX core

Implement parallel_scheduler wrapping thread_pool_policy_schedulerhpx::launch, aligning with P2079R7 user-facing API:

Add get_parallel_scheduler() for global access with async/sync policy selection.
Include parallel_sender with set_value_t(), set_stopped_t(), set_error_t(std::exception_ptr).
Support cancellation via stop token in operation_state::start_t.
Ensure parallel forward progress guarantee.
Add then operation for task chaining.
Add tests for progress, single task, cancellation, and exceptions.

replaceability API , bulk operations is the next thing i will keep working on.
Thank you for your support

charan-003 · 2025-04-30T05:57:09Z

@hkaiser @isidorostsa this version is using stdexec

I'm not sure of the failing test case, any hint on the failing cases ?

charan-003 · 2025-05-01T14:40:56Z

@hkaiser @isidorostsa
Does the current implementation, now relying on stdexec, meet our expectations? If so, I’ll proceed with implementing bulk support. Please confirm.

StellarBot · 2026-03-25T02:52:48Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	(=)	(=)

Info

Property	Before	After
HPX Datetime	2026-03-09T14:08:29+00:00	2026-03-24T20:16:57+00:00
HPX Commit	`0eeca86`	`ae805e8`
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T09:15:24.034803-05:00	2026-03-24T21:50:21.236223-05:00

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	++

Info

Property	Before	After
HPX Datetime	2026-03-09T14:08:29+00:00	2026-03-24T20:16:57+00:00
HPX Commit	`0eeca86`	`ae805e8`
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T09:17:15.638328-05:00	2026-03-24T21:52:02.565982-05:00

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	(=)	(=)	--
Stream Benchmark - Scale	(=)	--	--
Stream Benchmark - Triad	(=)	--	--
Stream Benchmark - Copy	(=)	+++	++

Info

Property	Before	After
HPX Datetime	2026-03-09T18:50:37+00:00	2026-03-24T20:16:57+00:00
HPX Commit	`ba89f5d`	`ae805e8`
Envfile
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Datetime	2026-03-09T17:49:10.837937-05:00	2026-03-24T21:52:22.801134-05:00

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

StellarBot · 2026-03-30T15:03:52Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	(=)	(=)

Info

Property	Before	After
HPX Commit	`0eeca86`	`aba2418`
HPX Datetime	2026-03-09T14:08:29+00:00	2026-03-30T14:32:36+00:00
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2026-03-09T09:15:24.034803-05:00	2026-03-30T10:01:23.803300-05:00
Clustername	rostam	rostam
Envfile

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	++

Info

Property	Before	After
HPX Commit	`0eeca86`	`aba2418`
HPX Datetime	2026-03-09T14:08:29+00:00	2026-03-30T14:32:36+00:00
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2026-03-09T09:17:15.638328-05:00	2026-03-30T10:03:04.947346-05:00
Clustername	rostam	rostam
Envfile

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	(=)	(=)	---
Stream Benchmark - Scale	(=)	--	--
Stream Benchmark - Triad	(=)	--	--
Stream Benchmark - Copy	(=)	+++	+

Info

Property	Before	After
HPX Commit	`ba89f5d`	`aba2418`
HPX Datetime	2026-03-09T18:50:37+00:00	2026-03-30T14:32:36+00:00
Compiler	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8	/opt/apps/llvm/18.1.8/bin/clang++ 18.1.8
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2026-03-09T17:49:10.837937-05:00	2026-03-30T10:03:25.284297-05:00
Clustername	rostam	rostam
Envfile

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

codacy-production · 2026-04-12T16:39:19Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 3 duplication

Metric Results

Duplication 3

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

hkaiser · 2026-04-21T14:14:00Z

Just FYI: https://wg21.link/p4031

charan-003 · 2026-04-26T21:43:17Z

@hkaiser @isidorostsa
It's ready for review now. It includes all the updates related to parallel_scheduler

hkaiser · 2026-05-05T20:15:16Z

@charan-003 Could you please resolve the conflicts. I will review the PR once you've done that.

charan-003 · 2026-05-06T00:38:47Z

@hkaiser a signature mismatch for bulk_t: HPX uses (sender, shape, f), while stdexec (P3481R5) expects (sender, policy, shape, f)
should we provide an overload for HPX's bulk that accepts an optional policy

hkaiser · 2026-05-06T17:55:01Z

                hpx::execution::experimental::get_completion_signatures_t,
                transform_mpi_sender const&, Env const&)
-            ->  hpx::execution::experimental::transform_completion_signatures_of<
+            ->  stdexec::__transform_completion_signatures_of_t<


This is not the right way of fixing this as it depends on internals of stdexec. Let's apply a correct fix to our forwarding header instead (see also #7252 for a potential different way of addressing this).

hkaiser · 2026-05-06T17:57:38Z

Inspect is not happy again:

/libs/core/executors/include/hpx/executors/parallel_scheduler.hpp: *I* missing #include (utility) for symbol std::move on line 170, *Line length limit*: 525
/libs/core/executors/include/hpx/executors/parallel_scheduler_backend.hpp: *I* missing #include (cstdint) for symbol std::uint32_t on line 185
    Non-ASCII: line 114
/libs/core/executors/tests/unit/parallel_scheduler.cpp: *I* missing #include (memory) for symbol std::make_shared on line 719

hkaiser · 2026-05-06T17:59:10Z

                [&]() { measure_parallel_foreach(data_representation, exec); });
        }

+#if defined(HPX_HAVE_STDEXEC)


HPX_HAVE_STDEXEC is always defined.

hkaiser · 2026-05-06T18:00:21Z

+
+#if defined(HPX_HAVE_STDEXEC)
+    // Forward declaration for parallel_scheduler_domain
+    class parallel_scheduler;


Please mark all symbols that should be visible outside of HPX with HPX_CXX_CORE_EXPORT to ensure those are accessible through C++ module BMIs.

hkaiser · 2026-05-06T18:02:32Z

+            {
+                return *sched.get_pu_mask();
+            }
+        };


Why do you need a helper to extract the parameters if those are already exposed through the corresponding customization points?

hkaiser · 2026-05-06T18:05:36Z

+                    using params_type =
+                        detail::thread_pool_params<std::decay_t<BaseScheduler>>;
+                    auto* pool = params_type::pool(exec.sched_);
+                    auto first_core = params_type::first_core(exec.sched_);
+                    auto num_cores = params_type::num_cores(exec.sched_);
+                    auto const& policy = params_type::policy(exec.sched_);
+                    auto mask = params_type::pu_mask(exec.sched_);
+
+                    return hpx::parallel::execution::detail::
+                        index_queue_bulk_async_execute(pool, first_core,
+                            num_cores, policy, HPX_FORWARD(F, f), shape, mask,
+                            HPX_FORWARD(Ts, ts)...);


This snippet could be extracted into a new function, which should simplify a lot of the code below.

hkaiser · 2026-05-06T18:07:36Z

@@ -106,18 +140,38 @@ namespace hpx::execution::experimental {
                hpx::execution::experimental::stdexec_internal::__sender_for<


Please move all stdexec specific to the forwarding header to simplify future use of standard library implementations.

charan-003 force-pushed the p2079-section-4.1 branch from cb9d1a7 to 1a77a29 Compare March 28, 2025 07:53

hkaiser added type: enhancement type: compatibility issue category: senders/receivers Implementations of the p0443r14 / p2300 + p1897 proposals labels Mar 28, 2025

charan-003 commented Mar 30, 2025

View reviewed changes

hkaiser reviewed Mar 31, 2025

View reviewed changes

Comment thread libs/core/executors/include/hpx/executors/thread_pool_scheduler.hpp Outdated

charan-003 marked this pull request as ready for review April 1, 2025 19:46

hkaiser reviewed Apr 2, 2025

View reviewed changes

Comment thread libs/core/executors/include/hpx/executors/parallel_scheduler.hpp Outdated

github-advanced-security AI found potential problems Apr 2, 2025

View reviewed changes

hkaiser reviewed Apr 6, 2025

View reviewed changes

Comment thread libs/core/executors/include/hpx/executors/parallel_scheduler.hpp Outdated

charan-003 marked this pull request as draft April 8, 2025 09:20

charan-003 marked this pull request as ready for review April 16, 2025 12:45

isidorostsa reviewed Apr 19, 2025

View reviewed changes

charan-003 requested review from hkaiser and isidorostsa April 30, 2025 11:41

charan-003 force-pushed the p2079-section-4.1 branch from 9803589 to 3b4121f Compare May 1, 2025 12:16

Merge branch 'master' into p2079-section-4.1

55b419e

charan-003 marked this pull request as draft March 25, 2026 18:35

charan-003 added 2 commits March 27, 2026 21:47

parallel scheduler uses cached mask

83b4eb9

add replaceability api

468529d

charan-003 added 5 commits April 24, 2026 06:18

Merge branch 'master' into p2079-section-4.1

4a81d15

fix minor issues

8385dc9

implement P3927

88febe9

implement p3804

c438adc

fix formating

603bb63

charan-003 marked this pull request as ready for review April 24, 2026 19:15

make it truely parallelized

0e65cbc

get back to old one

8623109

charan-003 force-pushed the p2079-section-4.1 branch from ab0fc9b to 8623109 Compare May 5, 2026 23:56

resolve conflicts

dd64808

use HPX bulk

3044549

charan-003 force-pushed the p2079-section-4.1 branch 2 times, most recently from 3044549 to dd64808 Compare May 6, 2026 01:01

charan-003 added 3 commits May 5, 2026 19:09

Merge branch 'master' into p2079-section-4.1

39395d3

use get_completion_scheduler

ce17a58

fix depricated errors

5e389a7

hkaiser reviewed May 6, 2026

View reviewed changes

		@@ -106,18 +140,38 @@ namespace hpx::execution::experimental {
		hpx::execution::experimental::stdexec_internal::__sender_for<

Uh oh!

Conversation

charan-003 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implement P2079R7 parallel_scheduler in HPX Core

Implementation Checklist

Uh oh!

StellarBot commented Mar 27, 2025

Uh oh!

codacy-production Bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

Uh oh!

charan-003 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charan-003 commented Mar 31, 2025

Uh oh!

charan-003 commented Apr 1, 2025

Uh oh!

hkaiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hkaiser commented Apr 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charan-003 commented Apr 2, 2025

Uh oh!

charan-003 commented Apr 3, 2025

Uh oh!

charan-003 commented Apr 3, 2025

Uh oh!

codacy-production Bot commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

Uh oh!

hkaiser commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

charan-003 commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charan-003 commented Apr 30, 2025

Uh oh!

charan-003 commented May 1, 2025

Uh oh!

StellarBot commented Mar 25, 2026

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

charan-003 commented Mar 27, 2025 •

edited

Loading

codacy-production Bot commented Mar 28, 2025 •

edited

Loading

codacy-production Bot commented Apr 3, 2025 •

edited

Loading

hkaiser commented Apr 4, 2025 •

edited

Loading

charan-003 commented Apr 14, 2025 •

edited

Loading

codacy-production Bot commented Apr 12, 2026 •

edited

Loading