GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers by 1fanwang · Pull Request #51128 · apache/arrow

1fanwang · 2026-09-01T22:32:40Z

Rationale for this change

A corrupt Parquet page can trigger a large scratch allocation from its header, even when it contains only one value.

What changes are included in this PR?

Before allocating, the decoder checks that the remaining input has at least one bit-width byte per required miniblock. Block decoding stays separate from this header check. Single-value pages skip the unused buffer.

Are these changes tested?

Testing Done

On macOS arm64, the probe below exercises the real C++ decoder with a single-value page and a truncated two-value page. Both allocate 1,048,576 scratch bytes in the original decoder. After the fix, both allocate zero, and the truncated page fails the header bound. File-backed reader tests were not run.

$ cmake -S cpp -B cpp/build -DARROW_PARQUET=ON -DARROW_BUILD_TESTS=ON -DARROW_BUILD_STATIC=ON -DARROW_BUILD_SHARED=OFF -DCMAKE_BUILD_TYPE=Release
-- Configuring done
-- Generating done

$ cmake --build cpp/build --target parquet-encoding-test -j2
[100%] Built target parquet-encoding-test

Before the fix, using the original decoder from 3ad410b:

$ git show 3ad410b7b1a2d3712e2d6a55d905a7035d026cae:cpp/src/parquet/decoder.cc > cpp/build/decoder-before.cc
$ c++ -std=c++20 -fno-aligned-new -DPARQUET_EXPORTING -Icpp/src -Icpp/build/src -Icpp/build/_deps/xsimd-src/include -I/opt/homebrew/include -c cpp/build/decoder-before.cc -o cpp/build/decoder-before.o
$ c++ -std=c++17 -Icpp/src -Icpp/build/src cpp/build/delta-allocation-repro.cc cpp/build/decoder-before.o -Lcpp/build/release -lparquet -larrow -Lcpp/build/mimalloc_ep/src/mimalloc_ep/lib -lmimalloc -L/opt/homebrew/lib -lsimdjson -lz -o cpp/build/delta-allocation-before
$ ./cpp/build/delta-allocation-before
decoded_count=1 decoded_value=0 scratch_bytes=1048576
invalid_header=Unexpected end of stream: Decode bit-width EOF scratch_bytes=1048576

The pre-fix probe exits 1 because it observes scratch allocation.

After the fix:

$ c++ -std=c++17 -Icpp/src -Icpp/build/src cpp/build/delta-allocation-repro.cc -Lcpp/build/release -lparquet -larrow -Lcpp/build/mimalloc_ep/src/mimalloc_ep/lib -lmimalloc -L/opt/homebrew/lib -lsimdjson -lz -o cpp/build/delta-allocation-repro
$ ./cpp/build/delta-allocation-repro
decoded_count=1 decoded_value=0 scratch_bytes=0
invalid_header=the required number of miniblock bit widths (1048576) exceeds the number of bytes remaining in the data (1) scratch_bytes=0

The fixed probe exits 0.

Reproducer source: cpp/build/delta-allocation-repro.cc

#include <cstdint>
#include <iostream>
#include <vector>

#include "arrow/memory_pool.h"
#include "parquet/encoding.h"
#include "parquet/exception.h"

int main() {
  const std::vector<uint8_t> encoded = {0x80, 0x80, 0x80, 0x10, 0x80,
                                        0x80, 0x40, 0x01, 0x00};
  arrow::ProxyMemoryPool pool(arrow::default_memory_pool());
  auto decoder = parquet::MakeTypedDecoder<parquet::Int32Type>(
      parquet::Encoding::DELTA_BINARY_PACKED, nullptr, &pool);
  int32_t decoded = 1;
  decoder->SetData(1, encoded.data(), static_cast<int>(encoded.size()));
  const int count = decoder->Decode(&decoded, 1);
  std::cout << "decoded_count=" << count << " decoded_value=" << decoded
            << " scratch_bytes=" << pool.total_bytes_allocated() << '\n';
  const std::vector<uint8_t> invalid = {0x80, 0x80, 0x80, 0x10, 0x80,
                                        0x80, 0x40, 0x02, 0x00, 0x00};
  arrow::ProxyMemoryPool invalid_pool(arrow::default_memory_pool());
  auto invalid_decoder = parquet::MakeTypedDecoder<parquet::Int32Type>(
      parquet::Encoding::DELTA_BINARY_PACKED, nullptr, &invalid_pool);
  int32_t values[2];
  try {
    invalid_decoder->SetData(2, invalid.data(), static_cast<int>(invalid.size()));
    invalid_decoder->Decode(values, 2);
    std::cout << "invalid_header=accepted\n";
    return 1;
  } catch (const parquet::ParquetException& error) {
    std::cout << "invalid_header=" << error.what()
              << " scratch_bytes=" << invalid_pool.total_bytes_allocated() << '\n';
  }
  return count == 1 && decoded == 0 && pool.total_bytes_allocated() == 0 &&
                 invalid_pool.total_bytes_allocated() == 0
             ? 0
             : 1;
}

Are there any user-facing changes?

Valid pages decode as before. Headers exceeding the input bound fail before allocation; truncated block data still produces the decoder's EOF error.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Parquet] Reject outlandish values in DELTA_BINARY_PACKED decoder #50314

github-actions · 2026-09-01T22:33:06Z

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot

🟡 Changes recommended

The decoder change still allocates miniblock scratch space unconditionally (including for single-value pages) and the new guard should account for required min_delta_ bytes, leaving a remaining allocation-DoS gap that should be closed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the C++ Parquet DELTA_BINARY_PACKED decoder against corrupt page headers that can otherwise drive disproportionate memory allocations and produce misleading EOF errors, aligning behavior with the security/robustness goals described in GH-50314.

Changes:

Add header validation in DeltaBitPackDecoder::InitHeader to reject pages whose miniblock count is incompatible with the remaining input bytes and emit a more actionable ParquetException.
Add new encoding tests to cover the single-value page path and the corrupt-header rejection case (including allocation behavior via ProxyMemoryPool).

File summaries

File	Description
cpp/src/parquet/decoder.cc	Adds early validation/error reporting for invalid DELTA_BINARY_PACKED miniblock headers (and aims to prevent oversized allocations).
cpp/src/parquet/encoding_test.cc	Adds regression tests for single-value decoding and for rejecting invalid miniblock-width headers without allocating.

Review details

Files reviewed: 2/2 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-03T19:39:14Z

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot

🟢 Approval recommended

The changes directly address the allocation-before-validation issue with clear guards and are covered by focused regression tests.

Review details

Files reviewed: 2/2 changed files
Comments generated: 0 new
Review effort level: Lite

github-actions · 2026-09-22T02:22:16Z

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot

Copilot review overview

🔵 Needs a closer look

Header validation can still allow allocation when min_delta_ uses multiple bytes and width data is truncated.

Review effort: Lite
Findings: None

Resolved since last review (1)

InitHeader still allocates delta_bit_widths_ unconditionally, even when total_value_count_ == 1…

Copilot

Copilot review overview

🟢 Approval recommended

Remaining feedback is limited to non-blocking test-name nits.

Review effort: Lite
Findings: None

…ders InitHeader() sizes the bit-width buffer from the header's miniblock count without tying it to the page size, so a 10-byte page claiming 2^20 miniblocks allocates 1 MiB before failing. InitBlock() reads one bit-width byte per miniblock, so such a page can never decode. Signed-off-by: 1fanwang <1fannnw@gmail.com>

Single-value pages never initialize a block, so leave the bit-width buffer unallocated. Account for the required min-delta byte when validating block metadata, and use cumulative allocation counts in the regression tests. Signed-off-by: 1fanwang <1fannnw@gmail.com>

InitHeader subtracted one byte for min delta, so a two-byte min delta with no bit widths still allocated the aligned scratch buffer and then failed with Decode bit-width EOF. Consume min delta first, then reject when remaining bytes cannot hold the declared miniblock widths. Signed-off-by: 1fanwang <1fannnw@gmail.com>

github-actions · 2026-09-25T20:53:39Z

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot

Copilot review overview

🟢 Approval recommended

The changes are covered by regression tests, with only a minor diagnostic wording nit remaining.

Review effort: Lite
Findings: None

pitrou · 2026-10-06T08:53:54Z

Sorry for the delay here. I don't like the proposed approach because it starts decoding a block in InitHeader. This makes the code more complicated and brittle.

I think we can do simpler. We know that each miniblock requires at least one byte for its bitwidth.
So the check can look like:

    const int64_t required_blocks = CeilDiv(total_value_count_ - 1, values_per_block_);
    if (required_blocks * mini_blocks_per_block > decoder_->bytes_left()) { /* throw exception */ }

Signed-off-by: 1fanwang <1fannnw@gmail.com>

github-actions · 2026-10-06T18:30:39Z

⚠️ GitHub issue #50314 has been automatically assigned in GitHub to PR creator.

Copilot

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified.

Review effort: Lite
Findings: None

1fanwang requested review from pitrou and wgtmac as code owners September 1, 2026 22:32

Copilot AI lite review requested due to automatic review settings September 1, 2026 22:32

github-actions Bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Sep 1, 2026

Copilot started reviewing on behalf of 1fanwang September 1, 2026 22:33 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/parquet/decoder.cc Outdated

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Sep 2, 2026

pitrou reviewed Sep 2, 2026

View reviewed changes

Comment thread cpp/src/parquet/encoding_test.cc Outdated

Copilot AI review requested due to automatic review settings September 3, 2026 19:40

Copilot started reviewing on behalf of 1fanwang September 3, 2026 19:41 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 22, 2026 02:24

1fanwang requested a review from HuaHuaY as a code owner September 22, 2026 02:24

Copilot started reviewing on behalf of 1fanwang September 22, 2026 02:24 View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 24, 2026 06:31

1fanwang force-pushed the 1fannnw/reject-invalid-delta-headers branch from bb561d0 to 18ae360 Compare September 24, 2026 06:31

Copilot started reviewing on behalf of 1fanwang September 24, 2026 06:32 View session

Copilot AI reviewed Sep 24, 2026

View reviewed changes

1fanwang added 3 commits September 25, 2026 13:52

Copilot AI review requested due to automatic review settings September 25, 2026 20:52

1fanwang force-pushed the 1fannnw/reject-invalid-delta-headers branch from 18ae360 to fca97c6 Compare September 25, 2026 20:52

Copilot started reviewing on behalf of 1fanwang September 25, 2026 20:52 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

apacheGH-50314: Validate DELTA_BINARY_PACKED bit-width bounds

025e748

Signed-off-by: 1fanwang <1fannnw@gmail.com>

Copilot AI lite review requested due to automatic review settings October 6, 2026 18:31

Copilot started reviewing on behalf of 1fanwang October 6, 2026 18:32 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Navigation Menu

Repository navigation

GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers - #51128

GH-50314: [C++][Parquet] Reject invalid DELTA_BINARY_PACKED headers#51128
1fanwang wants to merge 4 commits into
apache:mainfrom
1fanwang:1fannnw/reject-invalid-delta-headers

1fanwang commented Sep 1, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

pitrou commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

1fanwang commented Sep 1, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Testing Done

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

pitrou commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

1fanwang commented Sep 1, 2026 •

edited by github-actions Bot

Loading