GH-53033: [Python][Dataset] Support mixed FileInfo and path sources without crashing

[Mohit-25-tech]

What changes are included in this PR?

Fixes #53033

This PR fixes a bug in pyarrow.dataset.dataset() where mixing FileInfo objects and path-like sources produces inconsistent behavior depending on their order.

Previously:

• [path, path] — works.
• [FileInfo, FileInfo] — works.
• [path, FileInfo] — raises TypeError.
• [FileInfo, path] — causes a native segmentation fault.

The issue occurs because the dataset construction logic determines how to process the entire source list based on its first element, even when subsequent elements have different types.

Root cause

The public dataset() API accepts lists containing path-like objects or FileInfo objects.

However, _filesystem_dataset() previously dispatched based on the first element alone.

When the first element was FileInfo, the mixed list could reach FileSystemDatasetFactory, where the Cython unwrap_finfos() implementation performed an unchecked cast of subsequent elements to FileInfo.

This could terminate the Python interpreter instead of raising a normal exception.

Changes

1. Support mixed sources in the Python wrapper

• Normalize path-like entries in mixed lists into FileInfo objects using the selected filesystem.
• Preserve existing FileInfo objects and their metadata.
• Preserve source ordering and duplicate file entries.
• Keep homogeneous path and FileInfo lists on their existing processing paths.

2. Add defensive Cython validation

• Validate element types before casting them to FileInfo.
• Raise a descriptive TypeError for unsupported mixed inputs passed directly to the native factory.
• Prevent an unchecked cast from causing a process crash.

3. Add regression tests

Add coverage for:

• All four path/FileInfo combinations.
• List and tuple inputs.
• Duplicate sources and source ordering.
• Subtree filesystem handling.
• Preservation of supplied FileInfo objects.
• Missing files and directory paths.
• Direct FileSystemDatasetFactory invalid-input handling.

Behavior after the fix

For valid files, all four input combinations are supported:

ds.dataset([path, path], format="parquet")
ds.dataset([info, info], format="parquet")
ds.dataset([path, info], format="parquet")
ds.dataset([info, path], format="parquet")

Each should construct a dataset successfully without losing source ordering or duplicate entries.

Invalid or missing paths continue to raise appropriate Python exceptions rather than being silently ignored.

Testing

Independent reproduction

• Reproduced the original bug on Kaggle/Linux using PyArrow 23.0.1 and 26.0.0.
• Confirmed that [FileInfo, path] terminates the original process with SIGSEGV (exit code -11).
• Previously reproduced the same issue locally on Windows using PyArrow 16.1.0 and 24.0.0.

Patched implementation verification

• Loaded the actual modified dataset.py wrapper with the installed PyArrow 26.0.0 native runtime in Kaggle.
• All four input combinations passed and returned the expected data.
• 25 focused local checks passed against the modified Python wrapper using the installed native PyArrow runtime.
• Python syntax checks passed.
• Flake8 passed.
• autopep8 diff checks passed.
• git diff --check passed.

Testing limitations

• The modified Cython extension has not been compiled or executed locally.
• Full checkout pytest and source-build verification remain pending.
• Remote filesystem behavior has not been independently verified against an actual remote service.

The combined Python and Cython changes require repository CI validation.

Compatibility and performance considerations

The change extends the public dataset API to support mixed path and FileInfo lists.

Existing homogeneous-source behavior is preserved.

For mixed lists, resolving path entries to FileInfo objects can introduce an additional filesystem metadata lookup, including on remote filesystems.

AI assistance

OpenAI Codex assisted with investigating the issue, implementing the changes, creating regression tests, and reviewing the patch.

I independently reproduced the original crash, tested the modified Python implementation using Kaggle, and reviewed the proposed behavior and verification results.

Related issue

Fixes #53033

• GitHub Issue: [Parquet][Python][Dataset] Mixed FileInfo and path sources can crash Python in dataset() #53033

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

⚠️ GitHub issue #53033 has been automatically assigned in GitHub to PR creator.