bug: a legacy Delete reinstates a data file a concurrent Project pruned

[wjones127]

A Delete that rebases over a concurrently committed Project reinstates a data file the projection had pruned. The resulting manifest carries a data file none of whose fields are in the schema, and the field-id watermark goes back up over an id the projection retired.

Reproduction

On main (verified at 8cdffd30e), through the public write paths only:

1. Create a dataset with columns a, b; then add_columns a third column c, so fragment 0 carries two data files — [a, b] and [c]. Call this version v.
2. Stage a delete against v with DeleteBuilder::execute_uncommitted.
3. Commit a Project down to {a, b}. It lands at v+1, correctly pruning the [c] data file: fragment 0 now has one file.
4. Commit the staged delete against v. It rebases over the projection and lands at v+2.

Observed across the three versions:

version	schema field ids	fragment 0 data files	Manifest::max_field_id
v	[0, 1, 2]	[[0, 1], [2]]	2
v+1 (project)	[0, 1]	[[0, 1]]	1
v+2 (delete)	[0, 1]	[[0, 1], [2]]	2

The [2] file is back, and the watermark is non-monotonic — so a later write can mint field id 2 while a data file already claims it.

Cause

Same mechanism as #9216: Operation::Delete's apply replaces the fragment entry wholesale from a post-image built at the read version, which still lists the file the projection dropped. check_delete_txn permits a concurrent Project for the same reason it permits a concurrent DataOverlay.

The scan still succeeds — the orphaned file is simply unreachable — so this does not surface as a read error today. It is a correctness hazard for field-id allocation and leaves a file that cleanup will not collect.

Coverage

A failing test asserting the correct behaviour is committed #[ignore]d as io::commit::conflict_matrix::cases::delete_must_not_reinstate_a_pruned_data_file, pointing at this issue. The (Delete, Project) cell is excluded from that module's matrix until this is fixed.

cargo test -p lance --lib conflict_matrix -- --ignored

[wjones127]

The characterization suite found this is not specific to Delete. Three apply arms reinstate a data file a concurrent Project pruned, and each needs its own fix:

ours	theirs	ignored test
Delete	Project	delete_must_not_reinstate_a_pruned_data_file
Update { update_mode: RewriteRows }	Project	update_must_not_reinstate_a_pruned_data_file
Update { update_mode: RewriteColumns }	Project	covered by the matrix exclusion; same assertion

RewriteColumns is the sharpest case: the update writes a new data file for the column, the projection then drops that column, and the update reinstates its own now-unreachable file.

All four (ours, Project) cells are excluded from the matrix in rust/lance/src/io/commit/conflict_matrix/cases.rs until this is fixed. The invariant that catches it is no_orphaned_data_file — no data file may have all of its field ids absent from the manifest schema. Note the weaker form is deliberate: a file may legitimately carry a dropped field, because a projection only prunes a file when every field in it goes away.

[wjones127]

A fourth cell: (delete, project_alter_nullability) — a delete rebasing over a Project that does not assert preserves_nullability reinstates the pruned data file too, through the same apply arm as the plain Project case.

Only Delete reaches it. Both Update modes are held off by the may_alter_nullability pre-check, which makes them retryable against a nullability-altering projection before the apply arm runs.

Found by the characterization suite in #9220, which excludes the cell and links here.