Skip to content

[CI][C++][Parquet] Failure on parquet-reader-test with simdjson 5.x #51638

Closed
Closed
[CI][C++][Parquet] Failure on parquet-reader-test with simdjson 5.x#51638
Assignees
tadeja
Labels
Component: C++Component: Continuous IntegrationPriority: BlockerMarks a blocker for the releaseType: bug
Milestone
26.0.0

Description

@tadeja
tadeja
opened on Sep 29, 2026

Describe the bug, including details regarding any error messages, version, and platform.

Failing jobs: AMD64 Windows MinGW MINGW64 C++, AMD64 Windows MinGW CLANG64 C++, AMD64 Conda C++ AVX2 and
C++ Extra / AMD64 Ubuntu Meson

With simdjson 5.x failing on parquet-reader-test

[  FAILED  ] 4 tests, listed below:
[  FAILED  ] TestJSONWithLocalFile.JSONOutputWithStatistics
[  FAILED  ] TestJSONWithLocalFile.JSONOutput
[  FAILED  ] TestJSONWithLocalFile.JSONOutputFLBA
[  FAILED  ] TestJSONWithLocalFile.JSONOutputSortColumns
...
The following tests FAILED:
	 92 - parquet-reader-test (Failed)                      parquet-tests unittest

First MinGW job failure on main: https://github.com/apache/arrow/actions/runs/36565311423/job/109395674762#step:13:1885

Because of MSYS2 simdjson update to 5.0.1
mingw-w64-x86_64-simdjson-5.0.1-1-any downloading...
( -- ARROW_SIMDJSON_BUILD_VERSION: v4.6.4 later in the log is probably misleading )

Last successful MinGW job log on main still had
mingw-w64-x86_64-simdjson-4.6.11-1-any downloading...

Component(s)

Continuous Integration, C++

Activity

  1. tadeja
    added
    Type: bug
    on Sep 29, 2026
  2. github-actions
    added
    Component: C++
    Component: Continuous Integration
    on Sep 29, 2026

  3. aryan9948 commented on Sep 29, 2026

    @aryan9948
    aryan9948
    on Sep 29, 2026

    Hi @tadeja, I'd like to investigate this issue. I can look into the MinGW parquet-reader-test failures and whether the MSYS2 simdjson 5.0.1 update is causing the regression. Would you be okay with assigning this issue to me?

  4. github-actions
    assigned
    tadeja
    on Sep 29, 2026

  5. kou commented on Sep 29, 2026

    @kou
    kou
    on Sep 29, 2026
    Member

    You can just do it and share what you found here. We can collaborate with it. See also the findings in #51640.

  6. tadeja commented on Sep 29, 2026

    @tadeja
    tadeja
    on Sep 29, 2026
    MemberAuthor

    PR #50877 introduced simdjson::fractured_json_string and testing exact outputs in reader_test.cc

    Now simdjson 5.0 changes FracturedJson (simdjson/simdjson#2870, simdjson/simdjson#2878 and simdjson/simdjson#2879 change whitespace, line breaks, key order in tables),
    which already causes parquet-reader-test failures for MinGW jobs and soon other jobs (probably ARM64 macOS 14 C++ and AMD64 Conda C++ AVX2 when Docker layer cache gets invalidated.)

    I have a workaround in #51640 to get MinGW two jobs from 5.0.1 to bundled 4.6.4 simdjson and passing CI for the time being.

    Likely this needs a more permanent fix follow-up in parquet/reader_test.cc to accommodate for both test outputs, 4.6 and 5.0
    (I have a working demo by 🤖 on my fork tadeja#2)

    Fyi, @kou @Reranko05

  7. kou commented on Sep 30, 2026

    @kou
    kou
    on Sep 30, 2026
    Member

    How about testing TestJSONWithLocalFile.JSONOutput* only with simdjson 5 and updating the expected JSONs only for simdjson 5?

  8. raulcd commented on Sep 30, 2026

    @raulcd
    raulcd
    on Sep 30, 2026
    Member

    I think the approach @kou is proposing is reasonable. Either only test and validate the output (as I understand is slightly different in format) on simdjson 5 or update the check so it matches the expected output based on the simdjson version.
    @tadeja I am marking this as 26.0.0 even though it does not feel like a blocker is good to have a green CI on the release.

  9. raulcd
    added this to the 26.0.0 milestone on Sep 30, 2026

  10. tadeja commented on Sep 30, 2026

    @tadeja
    tadeja
    on Sep 30, 2026
    MemberAuthor

    @kou, @raulcd so what would you say about both in #51661 - new expected output for >= 5 (else existing 4.x test output)

  11. raulcd
    added
    Priority: BlockerMarks a blocker for the release
    on Oct 1, 2026

  12. tadeja commented on Oct 1, 2026

    @tadeja
    tadeja
    on Oct 1, 2026
    MemberAuthor

    Two more jobs fail on main:
    C++ / AMD64 Conda C++ AVX2
    C++ Extra / AMD64 Ubuntu Meson

  13. tadeja
    changed the title [-][CI][C++][Parquet] MinGW jobs fail on `parquet-reader-test` with MSYS2 simdjson 5.0.1[/-] [+][CI][C++][Parquet] Failure on `parquet-reader-test` with simdjson 5.x[/+] on Oct 1, 2026
  14. kou
    added a commit that references this issue on Oct 1, 2026

    GH-51638: [CI][C++][Parquet] Update expected JSON output in parquet-r…

    ba6c5fe

  15. kou commented on Oct 1, 2026

    @kou
    kou
    on Oct 1, 2026
    Member

    Issue resolved by pull request 51661
    #51661

  16. kou
    closed this as completedon Oct 1, 2026
  17. raulcd
    added a commit that references this issue on Oct 2, 2026

    GH-51638: [CI][C++][Parquet] Update expected JSON output in parquet-r…

    cca44b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Component: Continuous IntegrationPriority: BlockerMarks a blocker for the releaseType: bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions