Umbrella: API and dependency cleanup before the v1.0.0 freeze

[laskoviymishka]

Background

We are heading toward a v1.0.0 release of iceberg-go. Under Go semver, once we tag 1.0.0 any breaking change to the public API means a new /v2 module path, so the pre-1.0 window is the one cheap chance to fix API and layering warts. This is an umbrella issue to collect those breaking cleanups in one place so we can agree on scope and sequence them, rather than discovering them right after the freeze.

Two efforts already in flight roll up under this:

• Proposal: Decouple metadata & REST client from Arrow execution dependencies, and add an in-process REST testkit #2044 covers decoupling the metadata and REST client from the Arrow execution stack, plus an in-process REST testkit.
• Enormous dependency tree — consider "drivers"? #696 covers shrinking the dependency tree for consumers that do not need every backend (the IO / cloud-SDK side, via the driver pattern in refactor(cli): migrate from docopt-go to go-arg (#965) #980 and the per-cloud split in io/gocloud registers S3, GCS, and Azure backends as one inseparable package and no way to opt into just one cloud #1874 / feat(io): split io/gocloud into per-cloud backend packages #1961).

The intent is to keep things additive or behind facades where we can, and be explicit about the few genuinely breaking changes that have to land before the tag. Measurements and the detailed Arrow analysis live in #2044. Thanks @nssalian and @zeroshade for the early review; I have folded your notes into the sections below.

Status

• Drop pterm from the library path: refactor(table): drop pterm from library path #2061 (merged).
• AWS SigV4 optional backend: feat(rest): make SigV4 backend optional #2060 (open).
• Remove the //go:linkname namespace-property hack: refactor(catalog): move namespace-property helper into catalog/internal, drop go:linkname #2066 (open, thanks @nssalian).
• io.IO context threading: prototype ready, PR to follow.

1. Dependency footprint and layering

Goal: a consumer that only reads/writes metadata or talks to a REST catalog should not link a cloud SDK it never uses, or the Substrait execution engine.

• Drop the pterm terminal-UI dependency from table so table and catalog/rest build for GOOS=js GOARCH=wasm (refactor(table): drop pterm from library path #2061, merged).
• Make AWS SigV4 signing an optional backend so catalog/rest does not link the AWS SDK unless signing is used. This removes rest.WithAwsConfig(aws.Config) from the public option set (breaking). (feat(rest): make SigV4 backend optional #2060)
• Isolate the Substrait / compute execution code in table behind engine-facing sub-packages, keeping the public Transaction / Scan types in place as facades (Proposal: Decouple metadata & REST client from Arrow execution dependencies, and add an in-process REST testkit #2044). Per the decision below this is about keeping the Substrait engine and a cleaner dependency graph out of metadata/REST consumers, not about removing Arrow types; the js/wasm win already landed with the pterm removal, so this is lower priority than the AWS work.
• Continue the backend split from Enormous dependency tree — consider "drivers"? #696: separate the heavy catalog backends (glue, sql, hive) and the io backends into their own modules so the minimal consumer's module graph shrinks. glue.WithAwsConfig(aws.Config) has the same AWS-type leak as the REST option above.
• Decide whether the root Literal API stays Arrow-backed Decided (see Proposal: Decouple metadata & REST client from Arrow execution dependencies, and add an in-process REST testkit #2044): a zero-Arrow metadata core is not a v1 goal. We keep the Arrow-originated types for decimal / variant / geometry / geography rather than duplicating that work; the unused parts of arrow-go are dead-stripped when the core package uses only the types, and AWS is the real binary-size lever. VariantLiteral / DecimalLiteral stay as they are, and there is no non-Arrow scan path. Box closed.

2. Public API consistency (breaking, so pre-1.0 or /v2 only)

• io.IO: Open and Remove take no context.Context, while BulkRemovableIO.DeleteFiles does. Thread ctx through the base interface for a consistent, cancellable IO surface. (prototype ready)
• Catalog capabilities: view operations and RegisterTable are not part of the core Catalog interface and are supported unevenly across the REST / SQL / Hive / Glue / Hadoop implementations. Decide whether these become optional capability interfaces (e.g. a ViewableCatalog) so support is discoverable and uniform.
• table write API: the string-path and DataFile method pairs overlap (AddFiles / AddDataFiles, ReplaceDataFiles / ReplaceDataFilesWithDataFiles, ReplaceFiles / ReplaceFilesWithDeleteFiles). Settle on canonical names before we freeze all of them.
• PartitionField.SourceID() reads ambiguously next to the SourceIDs field now that a field can have multiple source ids. Rename or clarify.
• Schema.FindFieldByIDRef / FieldsRef: as @nssalian noted, the internal.SchemaRef{} argument already gates these to in-module callers (external modules cannot construct a type under internal/), so it is a deliberate sentinel rather than a leak. The action is to pin the intended shape (keep the sentinel, or move to an unexported entry point) before someone opens a PR, not to "hide" it.

3. Deprecations to remove at the freeze

Thanks @nssalian for tracing these. They are not equal cost, so splitting:

Clean deletes (batchable in one PR at the tag):

• DataFileBuilder.DistinctValueCounts (distinct_counts / field 111 is deprecated in the spec; no non-test callers in-repo).
• io/gocloud ParseAWSConfig / ParseGCSConfig redirect shims (only referenced by io/gocloud/compat_test.go; superseded by the per-cloud s3.ParseAWSConfig / gcs.ParseGCSConfig).
• WitMaxConcurrency (the typo'd alias of WithMaxConcurrency, which is the one we keep) and RewriteFiles.Apply (superseded by RewriteFiles.ApplyResult).

Migration (separate PR):

• Transform.ToHumanStr: ToHumanStrType (used by PartitionToPath) delegates to ToHumanStr, and every transform implements it, so removing it means inverting that delegation across every transform. That is a migration, not a delete, and deserves its own PR.

Out of scope for v1 release

To keep the freeze achievable, we would not block v1 on these:

• The full table package decomposition in feat(table): reorganize the table package into domain sub-packages (post-v3) #1149. It is maintainer ergonomics rather than API semantics, and moving exported symbols across packages is breaking, so it fits a later /v2 better than the 1.0 window.
• The Arrow dependency on the scan path (Scan.ToArrowRecords / ReadTasks return arrow.RecordBatch). Confirmed out of scope in Proposal: Decouple metadata & REST client from Arrow execution dependencies, and add an in-process REST testkit #2044: the Arrow scan path stays; we document the pin rather than remove it.

cc @zeroshade @nssalian @slachiewicz

[nssalian]

Thanks for putting this together. Excited to see go heading to v1.0.0.
No objection to the scope. A few notes from tracing the code, to sharpen the Section 2/3 items before PRs start:

Section 3: I wouldn't say the four deprecations are equal cost. Two are clean deletes, two aren't:

• DataFileBuilder.DistinctValueCounts - no non-test callers in-repo. Clean delete.
• io/gocloud.ParseAWSConfig / ParseGCSConfig (the deprecated shims in io/gocloud/gocloud.go) - only referenced by io/gocloud/compat_test.go. Clean delete.
• ToHumanStr - it's a Transform interface method implemented by every transform, and ToHumanStrType (used by PartitionToPath) delegates to it. Removing it means inverting that delegation across every transform, so it's a migration, not a delete. Worth its own PR.
• Naming nit on the last item: the deprecated symbols are WitMaxConcurrency (the typo'd alias; WithMaxConcurrency is the one we keep) and RewriteFiles.Apply (ApplyResult is its replacement), not the reverse. Both fine to batch.

Suggest splitting Section 3 into "clean deletes (one PR at the tag)" and "ToHumanStr migration (separate)".

Section 2: Schema.FindFieldByIDRef / FieldsRef: the internal.SchemaRef{} arg already gates these to in-module callers (external modules can't construct a type under internal/), so it reads more like a deliberate sentinel than a leak. Worth pinning the target shape before someone opens a PR.

Separately, the //go:linkname getUpdatedPropsAndUpdateSummary hack in Glue/Hive/SQL can go now: catalog/internal already imports catalog, so the helper moves there and returns catalog.PropertiesUpdateSummary directly, no cycle. I went ahead and opened #2066.
Happy to help on the others but those should be doable.

[zeroshade]

io.IO: Open and Remove take no context.Context, while BulkRemovableIO.DeleteFiles does. Thread ctx through the base interface for a consistent, cancellable IO surface. (prototype ready)

They take no context because they were intended to mirror the standard go io/fs package. Are we okay with no longer being compatible with it? Should we provide a wrapper for it?

[laskoviymishka]

Good question, and the io/fs mirroring made sense, but our main backends are object stores and Open there is a network call. Today you can't cancel or put a deadline on a pending Open. DeleteFiles already takes a context, so I'd fix the base interface before v1.

io.IO isn't used as an fs.FS today anyway: Open returns io.File, not fs.File, and the interface also has Remove. FS(fs.FS) IO still works. A reverse AsFS adapter can be added later if needed.

The open question is reads: io.Reader.Read has no context. Do we tie the returned file to the Open context, or keep reads on a longer-lived context like blobfs does today?

I lean toward tying it to Open.