Repository navigation
Parquet: choose dictionary encoding per column the way parquet-java does #11384
Description
Is your feature request related to a problem or challenge?
parquet-rs writes a dictionary page for high-cardinality columns where parquet-java writes none. That matters to anyone replacing a parquet-java writer with parquet-rs. DataFusion Comet's native Iceberg writer ran into it (apache/datafusion-comet#6114), and #9699 came from the same need.
parquet-java decides per column chunk, on the chunk's first data page (FallbackValuesWriter): unless the dictionary-encoded page plus the dictionary comes out smaller than the plain page, the whole chunk is written plain and has no dictionary page. parquet-rs never makes that comparison. It keeps a column dictionary-encoded until the dictionary reaches dictionary_page_size_limit, then falls back, and still writes the dictionary page because the earlier pages reference it. So columns such as ids, UUIDs and timestamps each carry a dictionary page of up to the limit, which every selective read of the chunk has to fetch. Comet measured 1M rows of a sorted unique BIGINT and a unique 16-character string, with Iceberg's 2 MiB dictionary limit. parquet-rs wrote a 2.1 MB dictionary page for each column and parquet-java wrote none, and WHERE id BETWEEN 1000 AND 1100 read 4.7 MB of the parquet-rs file against 0.6 MB of the parquet-java one.
Comet now makes parquet-java's decision itself (apache/datafusion-comet#6241), and @adriangb asked there whether the work could be upstreamed. I'd like to see that happen, but I'm not sure when I'll have time for it, so I'm filing this in case someone wants to pick it up.
Describe the solution you'd like
A function in the parquet crate that takes the Arrow schema, the WriterProperties and the first rows to be written, replays parquet-java's first-page accounting for each leaf column, and returns the properties with set_column_dictionary_enabled(path, false) for every column parquet-java would write plain. The writer itself doesn't change, and nothing changes for callers who don't use it, so this is closer to the sampling helper alamb suggested in #10775 than to a new fallback policy. The goal is to write what parquet-java writes, not to find the best encoding, so etseidl's point that one page is a small sample still applies. It's for callers that need to match the writer they are replacing.
Comet's version is DictionaryChooser in iceberg_dictionary.rs. It is about 570 lines plus tests and uses only arrow and parquet, apart from a HashSet and an error type from DataFusion. It ports the size accounting of FallbackValuesWriter and DictionaryValuesWriter, including the 4-byte length of binary values, the fallback when the dictionary outgrows its limit within the first page, and RunLengthBitPackingHybridEncoder's run splitting for the ids. It handles nested leaves and pages that hold only nulls, and reads one column at a time. Its tests compare it with answers recorded from parquet-mr, which were the same in 1.13.1, 1.15.0, 1.16.0 and 1.17.1. Over about 3,000 column and setting pairs it disagreed with parquet-mr on five, all close to the cut-off with 32–64 KiB pages. Those come from where a page ends: it stops at the first row past parquet-mr's size threshold, while parquet-mr waits for its next periodic size check. At the default page settings there were no mismatches. Choosing over 20,000 rows of 75 columns took 27 ms.
A general version would need a few things Comet does without. Comet skips INT96 leaves, which Iceberg doesn't have, and FIXED_LEN_BYTE_ARRAY leaves, which parquet-rs only dictionary-encodes under format v2. It hardcodes the row of parquet-mr's first page size check (100, the default). And it makes the choice once, before the file is opened, where parquet-java decides again for every column chunk.
Describe alternatives you've considered
The writer option #9699 asks for, which #9700 started as DictionaryFallback::OnUnfavorableAfter. Deciding inside the column writer would match parquet-java for every row group without callers holding rows back, but the writer would have to drop the dictionary and re-encode the first page, which runs into #9739. With #10917's per-row-group properties, a caller could also run the helper again for each row group. Otherwise each engine keeps its own copy, which is where Comet is today.
Additional context
To make the choice, Comet holds back the first page of rows for each partition it writes. In the worst case, where every partition has fewer rows than a page, every row is sampled and that adds about 40% to Parquet encoding. With bigger partitions the choice pays for itself, because high-cardinality columns are cheaper to write plain: on the same 20,000 rows, a zstd write took 48 ms with the chosen properties and 54 ms with dictionary encoding on for every column.
I drafted this issue with AI assistance (Claude Code), and the Comet code linked above was written with it too.
Activity
alamb commentedon Oct 6, 2026
Hi @andygrove -- thank you -- it sounds reasonable
Could you explicitly list on this ticket what you are trying to accomplish other than "do what the Java writer does" ? I suspect you are trying to make smaller files (or faster to read files, etc) but that is sort of lost with all the discussion of the details of under what circumstances what kinds of pages are written, etc
I think the key design consideration for any change like this is "what are the tradeoffs" -- if there is some class of usecase that will get worse we should have some setting to get the old behavior back as a workaround for downstream users, and then we can figure out what to default the setting to
mzabaluev commentedon Oct 6, 2026
Should the proposed function be dedicated to tuning dictionary-related writer properties, or be introduced as a general entry point for any such tuning behavior based on a sample of row data, where the dictionary fallback is the first heuristic to be implemented?
Should there be any governing value, either defined by the user or requested by the API as a black box, of how big a row sample this evaluation requires before making the decisions on column parameters? In the course of work on #9700, @etseidl made a case for this to be adjustable by the user.
etseidl commentedon Oct 6, 2026
My issue from the start has been that the java behavior is a holdover from a time when the default page size was 1MB (so 256k int32 values, 128k int64, etc). Post page index introduction, pages were limited to 20_000 rows, a much smaller sample size. So a column with a moderate cardinality in the 30k range will fallback to plain encoding, even though the dictionary would actually help a great deal. I think users are in a much better position to know the cardinality of their data than low-level parquet encoding engines with tiny samples. As long as the engine is configurable, I don't see an issue with placing the burden of tuning on the user (or downstream database layer). Solutions proposed for this to date, IMO, have suffered from adding complexity for very little real gain. Certainly no gain over simply setting dictionary encoding to
falsefor high cardinality columns.This I would support. It moves the guessing game from the encoder to a separate API that behaves as a "linter" (like https://github.com/XiangpengHao/parquet-linter). The downstream user then decides the sample size and lives with the consequences...want a better estimate, send a larger sample. I agree with @mzabaluev that it should have the ability to decide more than just dictionary true/false. But we could start with that and add more capability later.
alamb commentedon Oct 6, 2026
I do think the idea of a parquet optimizer (that will take your file and spend time to figure out actual data distributions and pick appropriate encodings, etc would be super helpful (as an aside, and as another project I don't actually have time to do)
Abhisheklearn12 commentedon Oct 9, 2026
hey @andygrove, i'd like to pick this up if you haven't started on it. plan would be to port
DictionaryChooserfrom comet (with credit to you ofc), but happy to back off if you'd rather do it yourself.@alamb re the goal: from what i can tell it's mostly faster selective reads, and smaller files as a side effect. with page indexes the reader always fetches the dictionary page of a column chunk, so unique columns (ids, uuids, timestamps) drag up to
dictionary_page_size_limitalong on every read even when only one page is selected.i ran a quick check on main: 1M rows in one row group, unique id/uuid/ts columns + a low cardinality one, default props vs dictionary turned off for just the unique columns (which is what this helper would return):
id BETWEEN 1000 AND 1100, zstd, all columns: 1.07 MB / 3.1 ms -> 0.29 MB / 0.56 mssynthetic data read from memory, so take it as rough. btw i used claude code to help put the bench together.
tradeoffs as i see them: the writer itself doesn't change, so nobody is affected unless they call the helper. the real downside is what @etseidl pointed out, a small sample can pick plain for a medium cardinality column where the dictionary would have won. if the caller passes the sample, they can just send a bigger one when that matters.
@mzabaluev on the api: i'd lean towards the caller passing the rows to sample (so sample size is theirs), keeping it dictionary only for now but naming it so other sample based tuning could go there later. i'd also make parquet-mr's first size check row a parameter instead of hardcoding 100, and handle the int96 / fixed len byte array leaves comet skips.
does that sound like the right direction before i start?
alamb commentedon Oct 9, 2026
I wonder if this PR from @zhuqi-lucas makes that different:
zhuqi-lucas commentedon Oct 10, 2026
Thanks @alamb — it does not change that, and I think the two are complementary rather than overlapping.
The page-index fetch planning is
ColumnFetchinparquet/src/arrow/in_memory_row_group.rs. The selection prunes data pages, butdictionary_rangenever looks at it — the dictionary range is always chained intoranges(), even when no data page is selected. So @Abhisheklearn12's statement is literally what the code does.#11168 does not touch that file. It changes
column/reader.rsandserialized_reader.rs— what happens to the bytes after they are fetched. On the async path the chunk ranges are pulled up front, so it saves the dictionary's decompression, not the read. (On the sync path, whereget_bytesis per page, a chunk skipped end to end does avoid the read as well.)So for
WHERE id BETWEEN 1000 AND 1100the 2.1 MB dictionary page is still fetched. This issue removes it at write time, which nothing on the read side can do.Abhisheklearn12 commentedon Oct 10, 2026
thanks @zhuqi-lucas for checking, that matches what i saw. also very good to know #11168 helps on the read side too, so the two fit together.
since nobody else has picked this up i'm going to assign myself and start on a draft PR this week. @alamb if you think the direction is off or someone else should take it, feel free to unassign me, no problem.
plan is to keep the api small to begin with: caller passes the sample rows, dictionary only for now, and it returns the adjusted WriterProperties. happy to change direction if anyone prefers something else :)
Abhisheklearn12 commentedon Oct 10, 2026
take