[Data] Upgrade pyiceberg to 0.12.x and adopt its commit retry and concurrency validation

[moomindani]

Description

Ray pins pyiceberg==0.11.0 (released 2026-02-10) in python/requirements/ml/data-test-requirements.txt
and the compiled/deplock files. 0.12.0 was released on 2026-09-01, and the change that matters for Ray
is apache/iceberg-python#3320 (merged 2026-08-12), which adds to Transaction.commit_transaction():

• automatic commit retry with exponential backoff, refreshing table metadata and regenerating manifests
• concurrency validation following Java's BaseOverwriteFiles.validate()
• isolation levels via write.delete.isolation-level / write.update.isolation-level (default serializable)
• table properties commit.retry.num-retries, commit.retry.min-wait-ms, commit.retry.max-wait-ms

Ray currently has no conflict validation on Iceberg writes at all. On 0.11.0 any concurrent commit
raises CommitFailedException whether or not the writes actually conflict, and
IcebergDatasink._with_retry only matches transport-level patterns ("429", "503",
"Connection reset", "UNAVAILABLE", ... in python/ray/data/context.py), so a concurrent writer
surfaces as a plain failure with no way to tell a benign concurrent append from a real conflict. After
the upgrade, compatible concurrent writes are retried transparently and genuine conflicts raise
ValidationException.

Adopting this is more than a version bump. Points to settle:

1. The datasink's snapshot producers do not populate the conflict detection filter.
_SnapshotProducer._validate_concurrency in 0.12.0 computes

conflict_detection_filter = self._predicate if self._predicate != AlwaysFalse() else None

The UPSERT commit (iceberg_datasink.py L547-L558) and the OVERWRITE path only call
delete_data_file(), never delete_by_predicate(), so _predicate stays AlwaysFalse. Under
serializable isolation _validate_added_data_files(..., None, ...) is then unscoped, so every data
file added by a concurrent APPEND or OVERWRITE snapshot in the validation window counts as a
conflict (VALIDATE_ADDED_DATA_FILES_OPERATIONS = {APPEND, OVERWRITE}), while the predicate-scoped
checks (_validate_no_new_delete_files, _validate_deleted_data_files) are skipped entirely. Passing
the delete predicate to the producer scopes this the way PyIceberg's own Transaction.delete() does.
delete_by_predicate already exists in 0.11.0 (_SnapshotProducer, snapshot.py L418), so this part
can land before or after the upgrade.

2. Two retry layers. IcebergDatasink._with_retry wraps txn.commit_transaction, which after the
upgrade retries internally (4 attempts by default). The layering should be decided deliberately, along
with whether Ray surfaces the commit.retry.* properties or leaves them to table configuration.

3. Private API surface. The datasink depends on _dataframe_to_data_files
(iceberg_datasink.py L147, L792) and txn._append_snapshot_producer (L580). These should be
confirmed against 0.12.0 as part of the upgrade.

4. Lock regeneration. pyiceberg appears in python/requirements/ml/data-test-requirements.txt
and in the data depsets under python/deplocks/ci/ for py3.10/3.11/3.12, including the
pyarrow-latest / pyarrow-v17 / pyarrow-nightly variants.

Dependency feasibility, checked against 0.12.0's pyproject.toml: requires-python = ">=3.10.0,<4.0.0"
(Ray is python_requires=">=3.10"), pydantic>=2.0,<3.0 excluding 2.4.0/2.4.1/2.12.0/2.12.1 (Ray pins
2.12.3), pyarrow>=18.0.0 (Ray pins 23.0.1). No blocker found.

For clarity on what this is not: 0.12.0 still cannot write format-version 3
(SUPPORTED_TABLE_FORMAT_VERSION = 2 in pyiceberg/table/metadata.py, and
TableMetadataV3.model_dump_json raises NotImplementedError), so this is not V3 write enablement.

Related: #66133 covers a separate defect in the OVERWRITE path, and its fix has to choose the conflict
detection filter described in point 1.

Use case

Concurrent writers against one Iceberg table: several Ray Data jobs appending to a shared table, or a
Ray Data write running while another engine compacts or rewrites the same table. Today any commit that
loses the race fails outright, even for writes that do not conflict, and a retry cannot distinguish the
two cases. With 0.12.0 the compatible cases succeed transparently and genuine conflicts are reported as
conflicts.

Would maintainers prefer the upgrade to land before #66133's fix, so that fix can set the conflict
detection filter deliberately, or the other way round?