feat(transaction): add idempotency key to fast append

[NoahKusaba]

Which issue does this PR close?

• Related to Transaction commits against a base it never validated, so an external writer cannot make a commit conditional #3134 (covers its idempotent-write case without making TransactionAction public).

What changes are included in this PR?

An engine that reruns a task can commit the same append twice. Checking for an earlier commit before Transaction::commit costs an extra table load and still races, because do_commit reloads the table and rebuilds the actions.

FastAppendAction::with_idempotency_key(key) records key in the snapshot summary and checks the refreshed table for it on every commit attempt, skipping the append if a snapshot on main already has it. A racing loser fails its ref requirement, retries, finds the key and skips. Flink's committer uses the same pattern with flink.job-id.

Not obvious from the diff:

• The check runs before file validation, so a rerun appending the files the first attempt committed is skipped, not rejected as a duplicate.
• An append with a key but no data files or snapshot properties commits nothing instead of failing, so it does not depend on the properties-only workaround in Clean up snapshot property logic when deletion supported #1548.
• do_commit no longer calls update_table when no action produced updates or requirements, as in Java. This affects every transaction: for example, expire_snapshots with nothing to expire no longer writes a metadata file.
• Keys on snapshots expired or rolled back off main are not found.

Are these changes tested?

Yes, unit tests: reruns from a stale table and with the same files, the race (replayed through a mock catalog in sequence), appends without files with and without snapshot properties, a skipped append next to an action that still commits, and a commit without updates.

AI Disclosure

Drafted with Claude Code; I reviewed it and ran the tests, clippy and fmt.

[NoahKusaba]

cc @mbutrovich @comphead @gabotechs

This is a major blocker for correct inserts into iceberg for distributed engines.

[comphead]

Thanks for the PR. Suggestions below, none blocking.

Smaller points:

• test_rerun_finds_its_key_below_later_snapshots and test_snapshot_with_idempotency_key_finds_each_key walk the same two-snapshot history, and already_committed only forwards to the helper. The first one can go. test_append_without_files_records_its_key asserts the whole summary map, including the total-* zeros, so an unrelated summary change breaks it. Asserting the IDEMPOTENCY_KEY_SUMMARY_PROPERTY entry is enough.
• No test covers a skipped append next to a sibling action that still has updates, for example update_table_properties in the same transaction. That is the case where the new early return in do_commit must not fire. A short test beside append in idempotency_key_tests would pin it.
• A skip is silent, and the caller gets Ok with no signal. A tracing::info! with the key and the matching snapshot id would make a dropped append diagnosable. Spark logs the same event in SparkWrite ("Skipping epoch ... as it was already committed").
• snapshot_properties() shares its name with the field but returns the field plus the key. A name like summary_properties() avoids the confusion.

[comphead]

A zero-file append only works today because SnapshotProducer::produce_manifests accepts a properties-only append (snapshot.rs:345). That branch is marked as a workaround to clean up (#1548). If the cleanup removes it, an append with a key and no files would likely start failing with PreconditionFailed, which contradicts this paragraph.

Is the empty snapshot needed, for example so a caller can use the key as a completion marker? A retried zero-file job adds no data, so there is nothing to protect from duplication. If the marker is wanted, please note the dependency in the TODO in snapshot.rs. If not, skipping a zero-file append would avoid an empty snapshot per job and let this paragraph and test_append_without_files_records_its_key go.

[NoahKusaba]

I originally kept it in as AI pointed out consistency with Flink's write design, but I'm overall indifferent and you're right the empty snapshot isn't needed. A keyed append with no data files and no snapshot properties now commits nothing and returns Ok, so nothing new depends on the #1548 workaround. With caller-set snapshot properties it still commits as before, so the key never drops them. The paragraph and test_append_without_files_records_its_key are replaced by test_append_without_files_commits_nothing and test_append_without_files_but_with_properties_commits.

[NoahKusaba]

Sorry for responding late (having a busy week) @comphead and thanks for the review, all addressed in db9d1a5:

• Dropped test_rerun_finds_its_key_below_later_snapshots. test_append_without_files_records_its_key is replaced (see the zero-file thread), so the full summary-map assertion is gone too.
• Added test_skipped_append_keeps_the_other_updates: a property update in the same transaction still commits when the append is skipped.
• A skip now logs tracing::info! with the key and the snapshot ID that holds it.
• Renamed snapshot_properties() to summary_properties().