Add cross-segment Select with result_offset for wide table scans#54
Open
krleonid wants to merge 2 commits into
Open
Add cross-segment Select with result_offset for wide table scans#54krleonid wants to merge 2 commits into
krleonid wants to merge 2 commits into
Conversation
Two optimizations for wide tables with selective filters: 1. Cross-segment SelectVector: Previously, when a vector (2048 rows) spanned multiple storage segments (common with 16KB blocks where VARCHAR segments hold ~1279 rows), StandardColumnData::Select fell back to scanning all 2048 rows then slicing. Now SelectVector splits the selection across segments, calling segment.Select() on each with a result_offset, reading only the selected rows from each segment. 2. StructColumnData::Select: Propagates the selection vector to child columns instead of scanning all rows. Short-circuits immediately for constant-NULL structs. The compression_select_t API gains a result_offset parameter, allowing Select to write at arbitrary positions in the result vector. All compression implementations updated: RLE, uncompressed string, FSST, dict_fsst, validity, numeric constant, and empty validity. Benchmarks (559-col table, 135K rows, 16KB blocks, filter selects ~34/2048): - Scan: 0.115-0.128s → 0.103-0.123s (15% faster) - 982-col table: latency 0.326s → 0.259s (21% faster) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
c1220d1 to
b2fbe7f
Compare
Several compression Select implementations ignored the result_offset parameter, writing results at index 0 instead of the correct offset when spanning multiple segments. Also fix DictFSST dictionary mode to not mutate the result vector type (scan+slice made it non-flat, crashing subsequent segments), and guard StandardColumnData::Select against non-flat vectors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
result_offsetparameter tocompression_select_tAPI, enabling Select to write at arbitrary positions in the result vectorColumnData::SelectVectornow handles cross-segment selection by splitting the selection vector across segment boundariesStandardColumnData::Selectno longer falls back to full Scan+Slice when vectors span multiple segmentsStructColumnData::Selectpropagates selection to child columns and short-circuits for constant-NULL structsProblem
With 16KB block size, VARCHAR segments hold ~1279 rows. A 2048-row vector spans ~2 segments. Previously,
StandardColumnData::Selectcheckedscan_entire_vectorwhich was alwaysfalsefor cross-segment vectors, forcing a fallback toScan(all 2048 rows) + Slice(keep 34)— wasting 98% of decode work.Solution
SelectVectornow iterates over segments, callingsegment.Select()with the appropriateresult_offsetfor each segment's portion of selected rows. All compression Select implementations (RLE, uncompressed string, FSST, dict_fsst, validity, numeric constant, empty validity) accept the new offset parameter.Benchmarks
DB1: 559 columns, 135K rows, 16KB blocks, dynamic filter selects ~34 rows per 2048-row chunk:
DB2: 982 columns, 21K rows, 16KB blocks:
Test plan
test/sql/types/struct/*— all 44 tests pass🤖 Generated with Claude Code