perf(table): avoid full snapshot materialization when creating metadata builders

[technicolorbeat]

Proposed Change

#1956 adds deferred materialization of unreferenced snapshot history when a REST commit response is parsed with snapshot-loading-mode=refs. The benefit is lost when the returned table is used to start another transaction.

Table.NewTransaction* calls MetadataBuilderFromBase, which currently requests the complete snapshot collection. This immediately materializes every deferred historical snapshot, even though the normal REST commit path primarily needs the accumulated requirements and updates rather than a fully rebuilt metadata document.

This path pays both the deferred indexing cost and the later full-decoding cost. It is therefore more expensive than eager parsing when another transaction is created immediately.

Measured on an Apple M1 Max using medians of five runs:

Metadata	Eager parse + builder	Deferred parse + builder	Time	Allocations	Allocated bytes
1,999,830 B	80.611 ms	98.314 ms	+22.0%	+20.7%	+38.9%
5,856,882 B	232.241 ms	287.704 ms	+23.9%	+20.7%	+38.5%

This is a performance limitation, not a known correctness problem. It is intentionally separate from #1946, whose scope is commit-response parsing.

Potential approaches:

This issue does not select a design. The following alternatives should be evaluated independently:

• Short term: Delay complete snapshot materialization during MetadataBuilderFromBase construction. Materialize only when a builder operation actually requires the complete snapshot collection.
• Mid term: Represent builder snapshot changes as mutations over an immutable deferred base, allowing additions, removals, ref validation, and individual ID lookup without decoding unrelated historical snapshots.
• Long term: Introduce a shared snapshot-collection abstraction used by metadata and metadata builders, with consistent support for indexed lookup, lazy iteration or materialization, mutation, cloning, validation, and serialization.

Correctness requirements:

• Preserve eager and deferred metadata behavior and parse-time validation semantics.
• Preserve complete snapshot history across metadata building, serialization, equality, and v1/v2/v3 conversion.
• Builder additions and removals must not lose, duplicate, or resurrect deferred snapshots.
• Ref and current-snapshot validation must account for both materialized and deferred snapshots.
• Preserve defensive-copy and concurrency guarantees.
• Default behavior remains unchanged for metadata without deferred snapshot state.

Performance validation:

Add benchmarks at multiple metadata sizes that measure, at minimum:

• eager parse followed by MetadataBuilderFromBase;
• deferred parse followed by MetadataBuilderFromBase;
• representative snapshot addition and ref-update transactions;
• snapshot removal or expiration operations;
• full metadata build and serialization.

The intended result is to avoid full history decoding for REST transactions that do not require it, without making operations that genuinely require the complete collection materially worse.

Backward compatibility:

• No breaking public API changes should be required.
• Full snapshot history must remain available and must never be silently truncated.
• Existing metadata validation and error behavior must be preserved.
• Eagerly parsed metadata should continue to use the existing builder behavior unless a common abstraction is adopted.

Related:

• perf(rest): lazy-decode snapshots in commitTableResponse when snapshot-loading-mode=refs #1946 — defer unreferenced snapshot materialization while parsing REST commit responses
• perf(rest): lazy-decode snapshots in commitTableResponse when snapshot-loading-mode=refs #1956 — implementation of the deferred commit-response parsing path

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

Proposal document

Not applicable — client-side performance optimization.

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other