Repository navigation
[C++][Parquet] Fast path for writing fixed-size lists #60
Description
Describe the enhancement requested
When writing a FixedSizeListArray (for example embeddings, coordinates or RGB tuples), Parquet C++ uses the same generic path as variable-length lists. MultipathLevelBuilder (cpp/src/parquet/arrow/path_internal.cc) walks the FixedSizeListNode / FixedSizedRangeSelector path node by node. It materializes one int16_t definition level and one repetition level per leaf value. Then LevelEncoder::Encode (column_writer.cc) RLE/bit-pack encodes those buffers again one value at a time.
For a list of size n with no nulls, both level streams are fully predictable:
- def levels: a constant
max_def_level, which is a single RLE run - rep levels: the periodic pattern
0, 1, …, 1(n - 1ones), repeated for every record
So we spend time and memory building and encoding 2 × num_values levels that could be written directly. This is the write-side counterpart of the read-side fast path described in Fast path for fixed-length lists in Parquet.
Proposal
Add a writer fast path that applies when:
- the leaf column sits under a
FixedSizeListArray(optionally also aListArray/LargeListArraywhose offsets show every list has the same length), and - the list has no null slots, the child values have no nulls, and there are no nullable or repeated ancestors (at least for the first version).
In that case:
- Skip the generic path traversal and produce the def/rep levels directly. Ideally they would go straight into the RLE encoder as runs (one def-level run per page; a precomputed periodic rep-level pattern per page) without materializing
int16_tbuffers. - Pass the child values slice straight to
WriteArrowDense(it is already contiguous:values()->Slice(value_offset(0), length * n)). - Compute the row count, page and record boundaries, and
SizeStatisticslevel histograms arithmetically (num_rows = num_values / n) instead of scanning rep levels.
Output must be byte-identical to the existing path, so files stay fully readable by every existing reader. No format or spec change is involved.
Acceptance criteria
- Round-trip tests for
fixed_size_list<T, n>covering severaln(1, 3, 8, 9, 16, 768), several physical types, V1 and V2 data pages, dictionary on/off, and a range of page and row-group sizes. - A test that the fast path and the generic path produce identical column chunks, including levels, page boundaries and statistics.
- The generic path still handles null lists, null elements and nested cases.
- A benchmark in
reader_writer_benchmark.cccomparing writingfixed_size_list<float, n>against writing the same values as a flat column, before and after the change.
Context
- Related read-side issue: [C++][Parquet] Fast path for reading fixed-size lists by detecting constant list length from encoded levels #61
- Existing report of slow fixed-size-list IO:
apache/arrow#34510 - Parquet dev list discussion on a
FIXED_SIZE_LISTlogical type: https://lists.apache.org/thread/qhq33wg1loxhymsyqjnsfsrd82qnv43m. This issue works independently of that discussion, because it needs no spec change. If a spec-level fixed-size marker lands later, the writer can emit it from this same path.
Component(s)
C++, Parquet
Activity