Support device buffer for the new libcurl multi poll-based backend (5/n) by kingcrimsontianyu · Pull Request #966 · rapidsai/kvikio

kingcrimsontianyu · 2026-05-24T19:42:36Z

This PR supports device buffer for the multi poll-based remote I/O backend. In addition, this PR makes 5 improvements:

Refactor ConcurrentRequestLimiter with RAII Slot. try_acquire() now returns a move-only Slot that auto-releases on destruction. Acquire/release pairing is now structurally enforced.
Add a new CurlMultiAttachment RAII guard, that removes the easy handle from the multi handle on destruction.
Batch the transfer submission to reduce the number of curl_multi_wakeup calls from the initial "per sub-range" to the current "per reactor, per pread".
Copy _inbox (shared by the submission thread and reactor thread) to a reactor-private _pending deque, to decouple the submission latency from reactor's admission walk.
Use AI to significantly condense the comments and remove incorrect, unclear or redundant comments.

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

kingcrimsontianyu · 2026-07-20T19:08:57Z

Sending this PR to draft. The reactor thread logic is very messy and can potentially be improved by RAII. Refactoring to have a cleaner code now.

kingcrimsontianyu · 2026-07-24T18:20:28Z

Performance results

2026.07.24 cudf-polars TPC-H queries. "v2" is simply a rerun.

t: number of threads
r: number reactors
c: connections
+----------------------+----------+------------+--------------+
| Case                 | Date     | Total (s)  | Δ from base  |
+----------------------+----------+------------+--------------+
| easy_256t (baseline) | 07.25    |  278.6069  |     ——       |
| easy_256t (v2)       | 07.25    |  272.7131  |    -2.12%    | --> best result
| easy_256t            | 07.24    |  284.9486  |    +2.28%    |
| easy_256t (v2)       | 07.24    |  293.2409  |    +5.25%    |
+----------------------+----------+------------+--------------+
| mp_16r/512c          | 07.25    |  294.6391  |    +5.75%    |
| mp_16r/512c (v2)     | 07.25    |  291.7514  |    +4.72%    |
| mp_16r/512c          | 07.24    |  302.0218  |    +8.40%    |
| mp_16r/512c (v2)     | 07.24    |  300.1998  |    +7.75%    |
| mp_16r/256c          | 07.25    |  282.7292  |    +1.48%    |
| mp_16r/256c (v2)     | 07.25    |  280.0991  |    +0.54%    | --> best result
+----------------------+----------+------------+--------------+

The results above show that: (1) For EC2-S3, there exists certain level of network bandwidth variance. (2) Comparing the best results, the new multi-poll backend is ~3% slower than the existing easy thread pool backend, with the benefit of thread usage cut by a factor of 16.

kingcrimsontianyu · 2026-07-26T03:20:24Z

Event pool vs on-the-fly creation/destruction

On a temporary branch, the event pool was modified such that the get() method creates a CUDA event and put() method destroys it. Comparing use of event pool vs on-the-fly event creation/destruction on cudf-polars TPC-H SF-1k, no statistically significant difference in performance was observed.

Event pool runs: total time ranges 280~285 seconds.
On-the-fly creation/destruction: total time ranges 284~287 seconds.

cc @madsbk

bdice · 2026-07-27T18:53:22Z

+  /**
+   * @brief Construct a barrier carrying `cuda_context` as metadata.
+   *
+   * @param cuda_context The CUDA context that pred's H2Ds will land in. Stored only for callers


Suggested change

* @param cuda_context The CUDA context that pred's H2Ds will land in. Stored only for callers

* @param cuda_context The CUDA context that pread's H2Ds will land in. Stored only for callers

bdice · 2026-07-27T18:54:00Z

  explicit RemoteMultiAggregateContext(std::size_t num_subranges);

+  /**
+   * @brief Per-pread event watermark for the device-buffer path.


Is "watermark" a common term of art here? I'm not familiar with what this means, it might be made up by an LLM?

No jargon used here. The per-pread, per-thread event from the IoEventBarrier is simply used repeatedly this way:

H2D (from pread 1) -> record event 1 -> H2D (from pread 1) -> record the same event (from pread 1) -> H2D (from pread 2), record event 2" ...

Anyway I've removed the mention of "watermark" to avoid confusion.

bdice · 2026-07-27T18:56:48Z

+{
+}
+
+ConcurrentRequestLimiter::Slot& ConcurrentRequestLimiter::Slot::operator=(Slot&& o) noexcept


Let's use other instead of o. One character variables are generally not favored in our codebases.

Done. Also replaced other uses of o with other.

bdice · 2026-07-27T18:58:40Z

+  detach_from_multi(_multi, _easy);
+}
+
+CurlMultiAttachment::CurlMultiAttachment(CurlMultiAttachment&& o) noexcept


Let's use other here and throughout instead of o.

bdice · 2026-07-27T18:59:35Z

+{
+}
+
+CurlMultiAttachment& CurlMultiAttachment::operator=(CurlMultiAttachment&& o) noexcept


Do we need to explicitly = delete; the copy constructor and copy assignment? (Trying to remember the rules.)

The move functions cause the copy functions to be deleted, but the rule of five recommends explicit deletion for readability.

bdice · 2026-07-27T19:00:40Z

+        }
+      }
+
+      // Admission walk over the reactor-private _pending. Each entry is either admitted to libcurl


"Admission walk over the reactor-private _pending." What does this mean? See if you can rewrite this.

Simplified a bit.

…ll-device

vuule

looks good, just one potential concern

vuule · 2026-07-31T01:25:11Z

+  return std::async(std::launch::deferred,
+                    [fut = std::move(fut), io_event_barrier]() mutable -> std::size_t {
+                      auto const n = fut.get();
+                      io_event_barrier->sync_all_events();


sync_all_events() is documented as context-agnostic, but it synchronizes events created in the pread’s CUDA context. Make IoEventBarrier::sync_all_events() push _cuda_context before synchronizing its events, and add coverage where the future is consumed on a thread with no current CUDA context.

So this lambda with deferred policy is evaluated at the future.get() time, where the CUDA context is allowed to be different from pread's. I've updated the doc to clarify what context-agnostic means.

I've added a unit test for sync_all_events() which is called when (1) there is no context current on the calling thread, (2) there is a different context.

kingcrimsontianyu · 2026-08-05T19:24:22Z

/merge

kingcrimsontianyu and others added 30 commits February 1, 2026 16:40

Implement event pool

08dd7f9

Fix stream race condition

3948400

Make ctor dtor private. Remove inner NVTX

43a38d9

Initial impl of event pool

bf20744

Set a get() overload to private

7e0c71f

Add Doxygen comments

74e6740

Remove get(ctx, tid) and move its content to get()

812c8a9

Merge branch 'fix-stream-bug' into event-pool

c9d7acd

Log exception msg

d878f79

Update cpp/src/detail/event.cpp

df8a4a8

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

Update name for clarity

9d73727

Update

b71c857

Update

dab118e

Silly bug fixes

3103863

Update

4208e5c

Merge branch 'main' into event-pool

badc794

Merge branch 'main' into event-pool

075dbd8

Add event query

64ee425

Merge branch 'main' into event-pool

87e8fe2

Initial implementation of multi interface poll-based backend

9b85829

Improve impl and clean up

d394d3c

Improve comments

3b3dbdd

Improve impl

efbbbee

Improve comment clarity

3594220

Revert Python changes

7e557cb

Update

268e925

Merge branch 'main' into multi-poll-host

c797e61

Clean up

dbf1d10

Update

9ae8881

Update comment

2add384

kingcrimsontianyu marked this pull request as ready for review July 20, 2026 03:33

kingcrimsontianyu requested review from a team as code owners July 20, 2026 03:33

kingcrimsontianyu marked this pull request as draft July 20, 2026 19:09

Refactor using RAII

1bffb2a

vuule moved this from Burndown to Slip in libcudf Jul 20, 2026

kingcrimsontianyu added 4 commits July 20, 2026 23:30

Clean up and condense comments

b5eadd5

Reorg a bit for clarity

3d5ef58

Update

afc4381

Cleanup

a3692c0

kingcrimsontianyu marked this pull request as ready for review July 24, 2026 19:41

Merge branch 'main' into multi-poll-device

9388151

bdice reviewed Jul 27, 2026

View reviewed changes

kingcrimsontianyu added 2 commits July 27, 2026 17:00

Addres review comments

f0be92c

Merge remote-tracking branch 'origin/multi-poll-device' into multi-po…

8c8d5a8

…ll-device

kingcrimsontianyu requested a review from bdice July 27, 2026 21:24

vuule self-requested a review July 31, 2026 01:40

vuule reviewed Jul 31, 2026

View reviewed changes

Addres review comments

b5f656c

kingcrimsontianyu requested a review from vuule July 31, 2026 16:02

bdice approved these changes Aug 4, 2026

View reviewed changes

vuule approved these changes Aug 5, 2026

View reviewed changes

rapids-bot Bot merged commit 9af385c into rapidsai:main Aug 5, 2026
64 checks passed

josephine-wolf-oberholtzer added this to libcudf Aug 12, 2026

josephine-wolf-oberholtzer moved this to Slip in libcudf Aug 12, 2026

	* @param cuda_context The CUDA context that pred's H2Ds will land in. Stored only for callers
	* @param cuda_context The CUDA context that pread's H2Ds will land in. Stored only for callers

Uh oh!

Conversation

kingcrimsontianyu commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kingcrimsontianyu commented Jul 20, 2026

Uh oh!

kingcrimsontianyu commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance results

Uh oh!

kingcrimsontianyu commented Jul 26, 2026

Event pool vs on-the-fly creation/destruction

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vuule left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kingcrimsontianyu commented May 24, 2026 •

edited

Loading

kingcrimsontianyu commented Jul 24, 2026 •

edited

Loading

kingcrimsontianyu Jul 31, 2026 •

edited

Loading