Repository navigation
[WIP][POC] Pfor encoding - #50088
Draft
prtkgaur wants to merge 6 commits into
Draft
[WIP][POC] Pfor encoding#50088prtkgaur wants to merge 6 commits into
prtkgaur wants to merge 6 commits into
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
prtkgaur
force-pushed
the
pfor-encoding
branch
4 times, most recently
from
October 3, 2026 01:25
d179a6e to
1298e2b
Compare
A frame-of-reference decoder adds the frame to every value right after unpacking it. Doing that in a second pass reads and writes the whole output again, so the scalar and SIMD unpackers now take an optional bias and add it before each store. With no bias the existing paths are unchanged, including the plain memcpy used when the bit width equals the output width. The tests run the biased form across widths, offsets, epilogues and every dispatched instruction set.
PFOR packs each vector of 1024 values at a width chosen for most of them and stores the rest as patched exceptions. This adds frame and width selection, the exception section, an endian-independent page layout, and decoding that checks every length against the page before reading it. int32 and int64 are supported. The codec builds under both CMake and Meson, with unit tests and a microbenchmark.
Registers PFOR for INT32 and INT64 and wires it into the encoder and decoder factories. Reads work for dense and nullable columns,, and the decoder can also return one vector at a time.
The tests write and read whole files, including nulls, batched reads, malformed pages and the factory paths. The benchmark compares PFOR with the existing integer encodings on synthetic columns modeled on common analytic data. It skips any compression codec the build does not include rather than aborting.
The PFOR templates are instantiated in libarrow and used from libparquet and its tests. Without export markers this only links on ELF platforms, where symbols are visible by default.
Lists the supported integer types and the Preview status. Writers never pick PFOR on their own; a column has to ask for it.
prtkgaur
force-pushed
the
pfor-encoding
branch
from
October 3, 2026 22:09
1298e2b to
687bdc8
Compare
prtkgaur
force-pushed
the
pfor-encoding
branch
2 times, most recently
from
October 7, 2026 00:01
7c52367 to
fe3d6ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doc : https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
Please remove this line and the above text before creating your pull request.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)