Skip to content

Extract shared list-array construction from the split_* functions in split.rs #6983

Open

Description

@andygrove

What is the problem the feature request solves?

In native/spark-expr/src/string_funcs/split.rs, the four generic split functions end with the same block of code: finish the values builder, pick Utf8/LargeUtf8 from O::IS_LARGE, build the item field, and construct a GenericListArray with the offsets.

They differ only in the per-row call and in how the null buffer is produced (cloned input nulls vs NullBufferBuilder). A fix to the list construction, such as the item field's nullability, has to be made in four places.

Describe the potential solution

Extract fn finish_split_list<O: OffsetSizeTrait>(offsets: Vec<O>, values: GenericStringBuilder<O>, nulls: Option<NullBuffer>) -> ColumnarValue and call it from all four functions. Optionally, also factor the per-row loop into a generic driver so each variant only supplies its null predicate and push call.

Additional context

This is small and a good first issue. Line references are pinned to commit 7d8d9b1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions