PyArrow: Cast dictionary-encoded arrays to target schema during scan projection (#3260)

[hedger9487]

Description

Fixes #3260.

When an Iceberg table contains mixed data files where some files contain dictionary-encoded string columns (e.g. written by clients using dictionary encoding) and other files contain plain strings (e.g. after Athena/Trino OPTIMIZE data compaction), ArrowScan.to_table() fails during batch concatenation with:

pyarrow.lib.ArrowTypeError: Unable to merge: Field col has incompatible types: string vs dictionary<values=string, indices=int32, ordered=0>

This occurs because ArrowProjectionVisitor._cast_if_needed() did not cast primitive dictionary-encoded arrays to the target schema type when projecting batches unless explicitly requested in dictionary_columns.

This PR updates _cast_if_needed() in pyiceberg/io/pyarrow.py to cast DictionaryType arrays to the target Iceberg primitive schema type when the column is not in dictionary_columns.

Testing

• Added regression test test_arrow_scan_mixed_dict_encoded_and_plain_strings in tests/io/test_pyarrow.py.
• All dictionary and scan projection unit tests pass locally.

[Copilot]

Pull request overview

This PR fixes ArrowScan.to_table() failures when scanning tables that contain a mix of dictionary-encoded and plain string columns across different data files, by ensuring projected batches are cast to the target Iceberg/Arrow schema during scan projection.

Changes:

• Pass dictionary_columns through the scan projection path so the projector can make correct type decisions.
• Update ArrowProjectionVisitor._cast_if_needed() to cast primitive DictionaryType arrays to the target primitive Arrow type unless the column is explicitly requested as dictionary-encoded output.
• Add a regression test covering mixed dictionary-encoded and plain string files for the same column.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
pyiceberg/io/pyarrow.py	Ensure scan projection casts dictionary arrays to the requested primitive Arrow type unless opted into dictionary output via dictionary_columns.
tests/io/test_pyarrow.py	Add regression test reproducing the mixed dict/plain string scan scenario and asserting successful concatenation with correct output type/values.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.