binary_as_string over a Parquet Binary column with invalid UTF-8 causes a segfault #25509

Describe the bug

datafusion.execution.parquet.binary_as_string = true changes a Parquet Binary column to a string type in the schema that DataFusion gives to the Parquet reader. If the column holds bytes that are not valid UTF-8, the reader returns an invalid string array and reports no error. The first operation that decodes those bytes then stops the process with SIGSEGV. The user gets no message and no error code from the query.

This is not limited to unusual schemas. A top level column is sufficient.

To Reproduce

datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'" -c "set datafusion.execution.parquet.binary_as_string = true" -c "SELECT b FROM 'bin.parquet'"

DataFusion CLI v54.0.0
+-------+
| count |
+-------+
| 1     |
+-------+
1 row(s) fetched.
Elapsed 0.010 seconds.

0 row(s) fetched.
Elapsed 0.001 seconds.

The exit code is 139. The macOS crash report shows SIGSEGV with the fault in comfy_table::Row::max_content_widths, which measures the width of the invalid string. With SELECT upper(b) the fault moves to alloc::str::to_uppercase.

Measured on the released datafusion-cli 54.0.0 and on a debug build of main at 4e90755 (arrow and parquet 59.3.0):

count(*), octet_length(b) and the byte-wise comparison b = 'ok' all succeed, because they do not decode UTF-8. The invalid array moves through the plan without a sign of a problem.

An explicit cast is safe, which shows the difference:

datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'" -c "SELECT arrow_cast(b, 'Utf8') FROM 'bin.parquet'"

Error: Arrow error: Invalid argument error: Encountered non UTF-8 data: invalid utf-8 sequence of 1 bytes from index 0

Valid multibyte UTF-8 in a Binary column reads correctly with the option on.

Expected behavior

The query must fail with an error, in the same way as the explicit cast. DataFusion must not give an invalid string array to the rest of the plan.

Additional context

The cause is in parquet-rs. ByteArrayColumnValueDecoder::new selects UTF-8 validation from the physical Parquet annotation only:

// parquet/src/arrow/array_reader/byte_array.rs:192
let validate_utf8 = desc.converted_type() == ConvertedType::UTF8;

A plain BYTE_ARRAY column has no UTF8 annotation, so the target Arrow type that DataFusion supplies through ArrowReaderOptions::with_schema has no effect. OffsetBuffer::into_array then uses ArrayDataBuilder::build_unchecked in release builds. Upstream report: apache/arrow-rs#11140

Found while reviewing #25342. That PR does not introduce the problem. It applies the same override to map children, so it makes more reads reach this path.