Skip to content

[C++][Parquet] Fast path for writing fixed-size lists #60

Open

Description

@rok

Describe the enhancement requested

When writing a FixedSizeListArray (for example embeddings, coordinates or RGB tuples), Parquet C++ uses the same generic path as variable-length lists. MultipathLevelBuilder (cpp/src/parquet/arrow/path_internal.cc) walks the FixedSizeListNode / FixedSizedRangeSelector path node by node. It materializes one int16_t definition level and one repetition level per leaf value. Then LevelEncoder::Encode (column_writer.cc) RLE/bit-pack encodes those buffers again one value at a time.

For a list of size n with no nulls, both level streams are fully predictable:

  • def levels: a constant max_def_level, which is a single RLE run
  • rep levels: the periodic pattern 0, 1, …, 1 (n - 1 ones), repeated for every record

So we spend time and memory building and encoding 2 × num_values levels that could be written directly. This is the write-side counterpart of the read-side fast path described in Fast path for fixed-length lists in Parquet.

Proposal

Add a writer fast path that applies when:

  • the leaf column sits under a FixedSizeListArray (optionally also a ListArray / LargeListArray whose offsets show every list has the same length), and
  • the list has no null slots, the child values have no nulls, and there are no nullable or repeated ancestors (at least for the first version).

In that case:

  1. Skip the generic path traversal and produce the def/rep levels directly. Ideally they would go straight into the RLE encoder as runs (one def-level run per page; a precomputed periodic rep-level pattern per page) without materializing int16_t buffers.
  2. Pass the child values slice straight to WriteArrowDense (it is already contiguous: values()->Slice(value_offset(0), length * n)).
  3. Compute the row count, page and record boundaries, and SizeStatistics level histograms arithmetically (num_rows = num_values / n) instead of scanning rep levels.

Output must be byte-identical to the existing path, so files stay fully readable by every existing reader. No format or spec change is involved.

Acceptance criteria

  • Round-trip tests for fixed_size_list<T, n> covering several n (1, 3, 8, 9, 16, 768), several physical types, V1 and V2 data pages, dictionary on/off, and a range of page and row-group sizes.
  • A test that the fast path and the generic path produce identical column chunks, including levels, page boundaries and statistics.
  • The generic path still handles null lists, null elements and nested cases.
  • A benchmark in reader_writer_benchmark.cc comparing writing fixed_size_list<float, n> against writing the same values as a flat column, before and after the change.

Context

Component(s)

C++, Parquet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions