Add logical type inspection for raw Parquet VARIANT values

[abigalekim]

Description

Implements feature mentioned in #23183. Adds an experimental libcudf operation, cudf::io::parquet::experimental::get_variant_type_id, that takes a Variant value column and returns an uint8 column of logical type identifiers. This PR also introduces a new variant_logical_type enum covering all Variant categories.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added support for identifying the logical type of each VARIANT value, including objects, arrays, strings, numbers, dates, timestamps, binary values, UUIDs, and nulls.
  • Results are returned as integer type identifiers while preserving input nulls and explicitly representing encoded null values.
  • Added handling for empty, sliced, mixed, malformed, and unsupported VARIANT data.

• Tests

  • Added comprehensive coverage across supported logical types, null handling, malformed inputs, and large datasets.

Walkthrough

Adds the public variant_logical_type enum and get_variant_type_id API. CUDA code decodes VARIANT headers into nullable INT32 type IDs. Tests cover mappings, malformed values, nulls, empty inputs, slicing, mixed types, and large columns.

Changes

VARIANT type-ID extraction

Layer / File(s)	Summary
Logical type contract cpp/include/cudf/io/experimental/variant_spec.hpp, cpp/include/cudf/io/experimental/variant.hpp	Defines variant_logical_type and declares get_variant_type_id with null and invalid-input behavior.
Type-ID classification and output cpp/src/io/parquet/experimental/variant_extract.cu	Classifies VARIANT headers on the device and returns nullable INT32 output.
Classification and edge-case coverage cpp/tests/io/experimental/variant_extract_test.cpp	Tests fixture mappings, malformed and unknown values, encoded nulls, empty inputs, slicing, mixed types, and large columns.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

• rapidsai/cudf#23183 — Covers the variant_logical_type and get_variant_type_id feature.
• rapidsai/cudf#23182 — Covers the corresponding libcudf logical-type inspection operation.
• rapidsai/cudf#23184 — Covers the API proposed for Java bindings.
• rapidsai/cudf#22312 — Covers related VARIANT logical-type decoding APIs.

Possibly related PRs

• rapidsai/cudf#23036 — Shares VARIANT definitions, extraction code, and related tests.
• rapidsai/cudf#23069 — Adds Java bindings for related Parquet VARIANT extraction APIs.
• rapidsai/cudf#23400 — Adds related Parquet VARIANT encoding infrastructure.

Suggested labels: tests

Suggested reviewers: vuule, mattgara, davidwendt

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding logical type inspection for raw Parquet VARIANT values.
Description check	✅ Passed	The description directly explains the new API, enum, input and output types, tests, and documentation updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

cpp/include/cudf/io/experimental/variant_spec.hpp (1)

56-71: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assign explicit values to variant_logical_type.

The enumerators of variant_logical_type rely on implicit sequential values. get_variant_type_id casts these values to int32_t and returns them as output data. If a new logical type is inserted in the middle of this list later, every subsequent numeric ID shifts silently. This can break downstream consumers that persist or compare these IDs.

Assign each enumerator an explicit value now, while the API is new, to fix the wire contract.

♻️ Proposed fix to pin enumerator values

 enum class variant_logical_type : uint8_t {
-  object,
-  array,
-  null_value,
-  boolean,
-  long_value,
-  string,
-  double_value,
-  decimal,
-  date,
-  timestamp,
-  timestamp_ntz,
-  float_value,
-  binary,
-  uuid
+  object        = 0,
+  array         = 1,
+  null_value    = 2,
+  boolean       = 3,
+  long_value    = 4,
+  string        = 5,
+  double_value  = 6,
+  decimal       = 7,
+  date          = 8,
+  timestamp     = 9,
+  timestamp_ntz = 10,
+  float_value   = 11,
+  binary        = 12,
+  uuid          = 13
 };

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/include/cudf/io/experimental/variant_spec.hpp` around lines 56 - 71,
Update the variant_logical_type enum so every enumerator has an explicit numeric
value, preserving its current sequential IDs and establishing a stable wire
contract for get_variant_type_id. Use the existing declaration order and assign
values starting at zero through uuid; do not change the enum members or their
ordering.

cpp/src/io/parquet/experimental/variant_extract.cu (1)

790-817: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the repeated value-span construction into a helper.

Lines 803-806 repeat the same offset/span construction already used in cast_variant_primitive_kernel (lines 632-636) and cast_variant_string_fn::operator() (lines 670-674). This is now the third copy of the same three-line idiom.

Extract a small __device__ helper that returns the value device_span<uint8_t const> for a row, and call it from all three sites. This reduces duplication and keeps future changes to the span-lookup logic in one place.

♻️ Proposed helper extraction

__device__ inline device_span<uint8_t const> value_span_at(
  cudf::lists_column_device_view const& values, size_type row)
{
  auto const val_begin = values.offset_at(row);
  auto const val_end   = values.offset_at(row + 1);
  return {values.child().data<uint8_t>() + val_begin,
          static_cast<std::size_t>(val_end - val_begin)};
}

     auto const val_begin = values.offset_at(row);
     auto const val_end   = values.offset_at(row + 1);
-    device_span<uint8_t const> const val{values.child().data<uint8_t>() + val_begin,
-                                         static_cast<std::size_t>(val_end - val_begin)};
+    auto const val = value_span_at(values, row);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/src/io/parquet/experimental/variant_extract.cu` around lines 790 - 817,
Extract the repeated value offset/span construction into a shared __device__
helper, such as value_span_at, accepting the lists_column_device_view and row
and returning device_span<uint8_t const>. Replace the duplicated three-line
logic in get_variant_type_id_kernel, cast_variant_primitive_kernel, and
cast_variant_string_fn::operator() with calls to this helper, preserving the
existing span contents and behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tests/io/experimental/variant_extract_test.cpp`:
- Around line 1621-1657: Increase num_rows in
GetVariantTypeIdTest.LargeMultiRowColumn from 128 to a value above the kernel
block size, such as 600, so get_variant_type_id exercises the multi-block
grid-stride path while preserving the existing cycling type coverage and
expected_ids generation.

---

Nitpick comments:
In `@cpp/include/cudf/io/experimental/variant_spec.hpp`:
- Around line 56-71: Update the variant_logical_type enum so every enumerator
has an explicit numeric value, preserving its current sequential IDs and
establishing a stable wire contract for get_variant_type_id. Use the existing
declaration order and assign values starting at zero through uuid; do not change
the enum members or their ordering.

In `@cpp/src/io/parquet/experimental/variant_extract.cu`:
- Around line 790-817: Extract the repeated value offset/span construction into
a shared __device__ helper, such as value_span_at, accepting the
lists_column_device_view and row and returning device_span<uint8_t const>.
Replace the duplicated three-line logic in get_variant_type_id_kernel,
cast_variant_primitive_kernel, and cast_variant_string_fn::operator() with calls
to this helper, preserving the existing span contents and behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f1423210-95f8-4ad2-ab54-2d2cafd21e53

📥 Commits

Reviewing files that changed from the base of the PR and between 7f58752 and 7bbdfb7.

📒 Files selected for processing (4)

• cpp/include/cudf/io/experimental/variant.hpp
• cpp/include/cudf/io/experimental/variant_spec.hpp
• cpp/src/io/parquet/experimental/variant_extract.cu
• cpp/tests/io/experimental/variant_extract_test.cpp

[mhaseeb123]

Few nits

[abigalekim]

/ok to test b0dfd69

[vuule]

Few API design questions, mostly for the feature requester :)

[abigalekim]

/ok to test e5246f8

[abigalekim]

/ok to test b98aebd

[abigalekim]

/ok to test dbc580a

[vuule]

few suggestions related to tests

[abigalekim]

/ok to test 7e6c2c2

[abigalekim]

/ok to test fc3c5c3

[abigalekim]

/ok to test fbd2ee8

[abigalekim]

/merge