Repository navigation
Allow dictionary pages to opt out of compression, as DataPageV2 already can #627
Description
Problem
DataPageHeaderV2 can declare a page uncompressed independently of the column chunk codec:
struct DataPageHeaderV2 {
...
/** whether the values are compressed.
Which means the section of the data page between the levels and the values is compressed **/
7: optional bool is_compressed = 1;
}DictionaryPageHeader has no equivalent:
struct DictionaryPageHeader {
1: required i32 num_values;
2: required Encoding encoding;
3: optional bool is_sorted;
}Compression is otherwise a column-chunk-level property (ColumnMetaData.codec), so a writer that wants zstd for its data pages has no way to leave the dictionary page uncompressed. The asymmetry looks incidental rather than intentional: V2 gained a per-page escape hatch, dictionary pages did not.
Why it matters
A dictionary page is a deduplicated set of distinct values. It is exactly the part of a column chunk where compression pays least — duplicates are already gone, so the codec is working on close-to-unique data — while it sits on the critical path of every reader that decodes any value from the chunk.
The cost is also paid at a different granularity than data pages: one dictionary page per column per row group. On files with many columns and small row groups that is a large count of small, high-entropy decompressions. In one file we looked at (14 columns, 12 row groups, every column dictionary-encoded) that is 168 dictionary pages, all zstd.
Two concrete readings:
- Skip-heavy scans. A reader that skips a column chunk entirely still has to decompress its dictionary today, unless the implementation defers it. We measured dictionary decompression at 13.6–17.9% of CPU on two production query-server classes, all of it reached from the skip path. Parquet:
skip_recordseagerly decompresses dictionary pages for column chunks it never decodes arrow-rs#11154 tracks deferring that decode, which removes the cost for chunks that decode nothing — but only for those. - Read-heavy scans. Any chunk that decodes a value still pays it, and deferral cannot help there. An uncompressed dictionary page would, and it is the same knob for both cases.
Proposal
Add an optional is_compressed to DictionaryPageHeader, with the same default and semantics as the DataPageHeaderV2 field:
struct DictionaryPageHeader {
1: required i32 num_values;
2: required Encoding encoding;
3: optional bool is_sorted;
/** whether the dictionary values are compressed. When absent the page is
compressed with the column chunk codec, as today. **/
4: optional bool is_compressed = 1;
}Defaulting to true keeps every existing file and writer valid, and a reader that honours the field needs one branch in the same place DataPageHeaderV2.is_compressed is already handled.
Compatibility, stated plainly
Adding the field is backward compatible — a reader that does not know it skips it as an unknown Thrift field, and files that never set it behave exactly as today.
The other direction is not free, and it is the main thing to decide. A file that does set is_compressed = false is unreadable by existing readers: they ignore the field, apply the column chunk codec to bytes that were never compressed, and fail in the decompressor. So this is a forward-compatibility break for any file that uses it, not an inert addition.
DataPageHeaderV2.is_compressed avoids that question because V2 pages are already gated behind a writer version that old readers do not claim to support. A dictionary page has no such gate today, which is probably the substantive reason the field is missing rather than mere oversight.
Options, roughly in order of how conservative they are:
- Allow it only in files that already use DataPageV2. Those files are already off-limits to readers that do not implement V2, so nothing new breaks. This costs nothing in compatibility and covers writers who have already opted into V2.
- Gate it behind a new writer feature/version, so the break is explicit and negotiated the same way V2 was.
- Allow it unconditionally and document the break, leaving the decision to writers who control both ends — a reasonable position for closed pipelines, a poor default for interchange.
I lean towards (1) as the first step: it is the smallest change that is strictly safe, it reuses an existing gate rather than inventing one, and it can be widened later if the wider case turns out to matter. But this is the part I would most like input on, since it decides whether the field is broadly useful or only useful to V2 writers.
Writers would then be free to trade a little file size for decode time on the page where that trade is most favourable, and to decide it per column — leaving it compressed for a large low-cardinality dictionary that compresses well, uncompressed for a high-cardinality one that does not.
Open questions
- Is the omission from
DictionaryPageHeaderdeliberate? I could not find a discussion of it, but I may have missed one. - Should the same escape hatch be considered for V1
DataPageHeader, or is V2 the intended migration path for that case? - Is per-page opt-out preferable to a separate column-chunk-level "dictionary codec" field? The per-page form matches the existing V2 precedent and needs no new column metadata, which is why it is proposed here.
Happy to work on the spec change and the arrow-rs side if this direction seems reasonable.
Activity
zhuqi-lucas commentedon Sep 29, 2026
cc @etseidl @alamb — this came out of the dictionary-deferral work in apache/arrow-rs#11168, which you both looked at.
That PR removes the dictionary decompression for column chunks that decode nothing. The limitation is that it can only ever help those chunks: anything that decodes a value still pays, and deferral cannot change that. An uncompressed dictionary page would be the same knob for both cases, and a dictionary is the part of a chunk where compression pays least anyway, since duplicates are already gone before the codec sees it.
The question I would most value your read on is whether the omission from
DictionaryPageHeaderis deliberate. My guess in the issue is that it is substantive rather than an oversight —DataPageHeaderV2.is_compressedis safe only because V2 is already gated behind a writer version, and a dictionary page has no such gate — but you would know the history better than I can reconstruct it.etseidl commentedon Sep 29, 2026
@zhuqi-lucas those decisions predate my involvement, but having run down this same rabbit hole a few years back I can offer some guesses. What you note about the dictionary page already having high entropy applies even more so to the data pages, which are RLE batches of bit-packed ints. I would suspect the
is_compressedflag was added to avoid cases where data pages grew in size due to compression that still helped (albeit maybe not so much) the dictionary page. If compression buys you nothing in the dictionary, it's probably just best to skip compression altogether for that column. And the same goes for read performance; compression is always a space/compute tradeoff. If compute is expensive, don't compress. If your main use case is small reads of targeted rows, don't use dictionary.That said, I once saw the need for an
is_compressedon the dictionary page, and there may still be cases where one is desirable. But as you point out, it would be a forward incompatible change and would need some strong motivation. And it would need to be behind a hard version gate as is being discussed now in the community (too many threads, but see this one for example).If you want more visibility for this discussion, I'd post to the parquet-dev list and link this issue.
zhuqi-lucas commentedon Sep 29, 2026
Thanks @etseidl — that is a much better read of the history than I had, and checking the encoders shows you are right in a way I had backwards. The dictionary page is
PlainEncoderover the deduplicated values; the data pages areRleEncoderover bit-packed indices. The high-entropy argument applies to the data pages, not the dictionary — a dictionary of strings has plenty of shared structure for a codec to work with.So I measured, since my premise deserved checking. On ClickBench
hits.parquet(105 columns, 226 row groups, SNAPPY), dictionary pages are 14.7% of all compressed bytes, and for high-cardinality columns they dominate their own chunk:UserID54.5%,FUniqID53.6%,RefererHash45.7%,WatchID27.5%. Leaving those uncompressed is not a cheap trade.The one angle that survives the measurement is that a dictionary is decompressed once per column chunk regardless of how many rows are read, while data page cost scales with what is actually touched. Its share of decompressed bytes therefore climbs steeply as a scan gets selective:
That is the shape of the workload behind the 13.6–17.9% profile figure.
But your two suggestions cover it without a format change, and they cover it on the first scan rather than only on repeats. Where the dictionary dominates, those are the high-cardinality columns where dictionary encoding is barely earning its place to begin with, so turning it off per column is a better answer than making it uncompressed. For the rest, per-column
UNCOMPRESSEDis one line of writer config. We control our writers, so that is the direction we will take.That leaves the format change wanted only where someone needs compressed data pages and an uncompressed dictionary in the same column. Narrower than I framed it. And thank you for the versioning pointer — reading that vote, the gating question I raised in the issue is already being answered: this would be an in-preview Version 3 feature behind a writer feature flag, with the two-implementations-and-cross-testing bar to clear. That bar is the right one, and I do not think this case earns it on what I can show today.
Happy to leave this open for the record in case others want the middle case — the selectivity numbers may save someone the measurement — but I will not push it further.
alamb commentedon Sep 29, 2026
Hi @zhuqi-lucas -- in general I think it would help me to understand this proposal if you could provide more details on your usecase
Is the idea that you want to support faster single row reads by allowing uncompressed dictionary pages ?
Deferring decode of the dictionary until we know it is needed sounds like the right approach to me there
I am not sure what you mean by a read-heavy scan (aren't scans by definition read heavy 🤔 )
zhuqi-lucas commentedon Sep 30, 2026
Thanks @alamb — "read-heavy" was the wrong word. The contrast I meant is not heavy versus light but a chunk that decodes nothing versus a chunk that decodes at least one value. Deferral removes the dictionary cost entirely for the first and cannot help the second, since by then the dictionary is genuinely needed.
And yes, that is the case — point lookups and small key-range reads over large files, which is just the extreme end of the same curve. The dictionary is decompressed once per column chunk no matter how many rows come out of it, while data page cost scales with what is touched, so the fixed part dominates as fewer rows are decoded. On ClickBench
hits.parquetthe dictionary is 14.7% of compressed bytes, but 63% of decompressed bytes when 10% of rows are decoded, 94.5% at 1%, and approaching all of it for a single row.That said, my comment above already walks this back: @etseidl pointed out the high-entropy argument applies to the data pages (bit-packed indices), not the dictionary (PLAIN raw values), and the encoders confirm it. His two existing suggestions — per-column
UNCOMPRESSED, or turning dictionary encoding off where the dictionary dominates — cover our case today with no format change. I do not think this clears the two-implementations bar on what I can show, so it is not worth more of your time. Left open only so the selectivity numbers are on the record.