parquet: dictionary page decode copies and validates the whole dictionary even when few values are referenced #11303

Describe the bug

Decoding a byte-array dictionary page copies every value into a fresh buffer and UTF-8 validates all of them, even when the read only resolves a handful of keys. On a selective read the cost is proportional to the dictionary, not to the rows produced: a 100k-entry dictionary costs the same whether the query returns 2 rows or 200k.

set_dict materializes the whole dictionary:

// parquet/src/arrow/array_reader/byte_array.rs
fn set_dict(&mut self, buf: Bytes, num_values: u32, encoding: Encoding, _is_sorted: bool) -> Result<()> {
    ...
    let mut buffer = OffsetBuffer::with_capacity(0);
    let mut decoder = ByteArrayDecoderPlain::new(buf, num_values as usize, Some(num_values as usize), self.validate_utf8);
    decoder.read(&mut buffer, usize::MAX)?;   // every value copied, all of them validated
    self.dict = Some(buffer);
    Ok(())
}

OffsetBuffer owns its bytes, so this is a real allocation plus a full copy of the page:

pub struct OffsetBuffer<I: OffsetSizeTrait> {
    pub offsets: Vec<I>,
    pub values: Vec<u8>,
}

But the only consumer takes slices and copies just the referenced entries:

// ByteArrayDecoderDictionary::read
output.extend_from_dictionary(keys, dict.offsets.as_slice(), dict.values.as_slice())

pub fn extend_from_dictionary<K, V>(&mut self, keys: &[K], dict_offsets: &[V], dict_values: &[u8]) -> Result<()> {
    for key in keys {
        ...
        // Dictionary values are verified when decoding dictionary page
        self.values.extend_from_slice(&dict_values[start_offset..end_offset]);
        ...
    }
}

So the dictionary is used purely as a lookup table of (offsets, bytes). The owned copy built in set_dict is never needed as such.

To Reproduce

Read a small RowSelection (a few rows) from a row group whose column chunk has a large dictionary. The dictionary copy and validation dominate, and they do not shrink as the selection shrinks.

Expected behavior

Building the dictionary should cost a scan of the length prefixes, not a copy of the page.

Concretely: keep the decompressed page as the already-refcounted Bytes and build only the offsets, then let extend_from_dictionary copy the referenced entries out of that borrowed slice as it already does. That removes the allocation and the full copy entirely.

UTF-8 validation would move with it. Today set_dict validates the whole dictionary and extend_from_dictionary skips validation on that basis (see its comment). Validating the copied-out entries instead makes the cost proportional to rows produced. Note this is a small behavior change: a dictionary entry containing invalid UTF-8 that is never referenced would no longer fail the read. That seems like an improvement, but it is a change, so worth deciding deliberately.

What this does not address is decompression: a compressed dictionary page still has to be decompressed as a unit before the offsets can be walked.

Additional context

This is the case that the lazy dictionary work does not reach. Skipping the dictionary when a column chunk is skipped entirely (#11154, #11168) helps only when no rows are needed from the chunk; as soon as one row is needed, the full dictionary cost returns. Selective point lookups and small LIMIT queries over high-cardinality string columns hit this constantly.

Happy to work on this.