feat: keep column groups through compaction and add rewrite_columns

[dshepelev15]

What

• CompactionOptions.column_groups (Python compact_files(column_groups=[["emb"], ["cap", "tags"]]), dataset config key lance.compaction.column_groups = "emb;cap,tags"): each listed group of top-level columns is written to its own data file per compacted fragment; every column not listed goes to one shared file. A single scan feeds one writer per group with the same planned row counts, so the files of a fragment stay row-aligned. Binary copy is disabled while groups are set (force_binary_copy reports NotSupported) and max_bytes_per_file is ignored so all groups split at the same rows. Unknown columns fail at planning, before tasks are distributed. The Java JNI builds its options through from_dataset_config, so it needs no change and picks the groups up from the table config.
• Dataset::rewrite_columns(columns, data_storage_version) / FileFragment::rewrite_columns (Python LanceDataset.rewrite_columns, LanceFragment.rewrite_columns): rewrite only the named top-level columns of every fragment into one new data file in the requested V2 version and tombstone them in the files they came from; a file left holding only tombstones is dropped. The other columns' files are neither read nor written, and rows, fragment ids, row addresses and indices are unchanged. Commits one Update in RewriteColumns mode with empty fields_modified, since the values did not change. Fragments whose columns already sit alone in a file of the requested version are skipped, so an interrupted rewrite can be rerun; the fragment-level call returns metadata for a single distributed commit.

Why

Compaction rewrites whole fragments, so moving a table to a newer data file version through compaction re-encodes every column, including the multi-KB embedding columns that dominate its size. It also folds any per-column file layout (from merge_insert in rewrite_columns mode or add_columns) back into one wide file, which makes the next single-column update rewrite every column again. Together the two options let a table keep its wide columns in their own files across compactions and migrate its narrow columns without touching them.

Notes

• rewrite_columns rejects blob columns for the same reason the merge-insert RewriteColumns path does: the stored descriptor form differs from the logical form the writer expects. Nested paths are not accepted in column_groups.
• versions::validate_write_schema factors out the blob checks write_fragments already ran, and create_seed_writers becomes pub(super) so the grouped writer can seed indices too.
• Tests: Rust dataset::optimize (column groups across compaction modes and target versions, config parsing, validation) and dataset::rewrite_columns (migration and layout, index coverage kept, bad input, compaction preserving the layout); Python test_optimize.py and test_schema_evolution.py, including the per-fragment commit path.

[Xuanwo]

I did similar thing before: #8614

This change been postponed because I think it's better to refactor our compaction logic so that make lance to produce stats instead of a black box.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The earlier physical-layout PR and the statistics-first concern do not change the recommendation. Producing layout statistics would improve policy selection, but it does not replace the executor needed to rewrite chosen groups. This PR supplies that row-aligned execution through existing compaction jobs and table configuration, while rewrite_columns avoids rewriting wide columns; its row, index, overlay, and stable-row-ID contracts remain preserved.

Please mark this PR with the breaking-change label.

[wjones127]

Thanks for making a PR on this. Like Xuanwo said, I think we should build out a more pluggable compaction planner and executor. Then this can be integrated as part of this. I think compacting horizontally like this is a good idea and overdue.

[wjones127]

suggestion(blocking): instead of adding a new API, I think we should find a way to fold this into the existing compaction operation. That way we aren't adding yet another maintenance operation a user has to call. Rewrite_columns can just be a particular runtime configuration you pass, if you only want to do those jobs. #8614 (comment)