Skip to content

test: First basic e2e exchange test: - #16

Draft
dan13bauer wants to merge 42 commits into
upstream_patchesfrom
comms_unit_test
Draft

test: First basic e2e exchange test:#16
dan13bauer wants to merge 42 commits into
upstream_patchesfrom
comms_unit_test

Conversation

@dan13bauer

Copy link
Copy Markdown
Owner
  • Mimicks a source task with a mock partitioned output operator
  • Mimicks a sink task with a mock driver that executes the hybrid exchange operator
  • The basic test checks whether the correct number of rows arrive at the exchange.

The idea is to add more tests:

  • Create multiple source tasks (CudfPartitionedOutputMocks) to test parallelism upstream
  • Create multiple exchange tasks to test parallelism at the receiving side
  • Increase the number of partitions
  • Add remoteSplits while the exchange task is running instead of adding the splits before

- Mimicks a source task with a mock partitioned output operator
- Mimicks a sink task with a mock driver that executes the hybrid
  exchange operator
- The basic test checks whether the correct number of rows arrive
at the exchange.
@dan13bauer dan13bauer self-assigned this Oct 10, 2025
@SeanRooooney

Copy link
Copy Markdown
Collaborator

Runs fine, fixed a small error and am extending the types.
Idea to test data consistency by, generating random string AND
then setting the other columns to be derived, i.e. col1=string col2=length of string, col3=hash of string

@SeanRooooney

Copy link
Copy Markdown
Collaborator

Added some attempt to verify the integrity of data, still needs work nut you can run it with:
./_build/release/velox/experimental/cudf-exchange/tests/cudf_exchange_test -v=3 -logtostdout --gtest_filter=CudfExchangeTest.dataTest

@SeanRooooney

Copy link
Copy Markdown
Collaborator

Note one cannot run, multiple tests within the same process as something in ucxx is not released properly, on 2nd test:

terminate called after throwing an instance of 'ucxx::BusyError'
what(): Device is busy

@SeanRooooney SeanRooooney changed the title First basic e2e exchange test: test: First basic e2e exchange test: Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants