Proposal: Decouple metadata & REST client from Arrow execution dependencies, and add an in-process REST testkit

[slachiewicz]

Summary

github.com/apache/iceberg-go is a single Go module, so a consumer that only needs the metadata layer (schema, types, partition specs, table metadata, manifests) or the REST catalog client still links the Arrow execution stack. Measured on main (833e10c): a package main whose only import is _ "github.com/apache/iceberg-go/catalog/rest" builds to a 66 MB unstripped darwin/arm64 binary, against 4.9 MB for the same program importing only net/http, and its go.mod carries 65 indirect requirements.

go list -deps ./catalog/rest shows the chain: catalog/rest → table (plus table/dv, table/internal, table/substrait) → arrow-go/v18/{arrow,compute,parquet,ipc,flight}, substrait-go/v8, geoarrow-go. The root iceberg package now imports Arrow as well (manifest.go, exprs.go, literals.go, transforms.go, types.go, variant_cast.go, variant_extract.go → arrow/decimal128, parquet/variant, geoarrow-go), so the separation described in #399, where all Arrow use was confined to table, no longer holds.

Relationship to earlier issues

• Enormous dependency tree — consider "drivers"? #696 raised the same dependency-tree concern for the IO layer. The driver pattern from refactor(cli): migrate from docopt-go to go-arg (#965) #980 and the per-cloud split in io/gocloud registers S3, GCS, and Azure backends as one inseparable package and no way to opt into just one cloud #1874 addressed the cloud SDK side; this issue is about the Arrow and Substrait coupling on the metadata and REST-client side, which those changes did not touch.
• Dependency on github.com/apache/arrow-go/v18/arrow/compute greatly increases binary size #399 asked for the Arrow integration to move behind a sub-package, and the response suggested pushing scanning into a sub-package so that most table operations do not need Arrow. This proposal is a concrete plan for that.
• The guidance in Enormous dependency tree — consider "drivers"? #696 was to isolate packages first and treat separate go.mod files as a possible second pass. This proposal follows that ordering.

Motivation

1. Metadata-only consumers. Catalog proxies, REST gateways, metadata translation tools (Apache XTable-style bridges), and serverless functions read and write table metadata but never scan data. They pay for Arrow compute, Substrait, and Parquet decoding they cannot use, and inherit arrow-go and aws-sdk-go-v2 version constraints through MVS.
2. GOOS=js GOARCH=wasm. The root package and io already build for js/wasm. table and catalog/rest do not, and the single failing package is atomicgo.dev/keyboard, reached through pterm, which table/arrow_utils.go imports for progress output. That is a small fix on its own, but it shows how a terminal UI dependency in table reaches a REST client that has no terminal.
3. Reusable in-process REST catalog for tests. The unit tests in catalog/rest already run without containers against httptest handlers, and the container-backed tests are behind //go:build integration (internal/recipe/docker-compose.yml, apache/iceberg-rest-fixture). What is missing is a stateful, importable in-process REST catalog server that this repo's tests and downstream projects can use to drive a real client end to end (create namespace → create table → commit → load) without Docker, in the way httptest serves net/http.

Precedent in Apache Iceberg (Java)

• iceberg-api: schemas, types, expressions, partition specs; depends only on bundled Guava.
• iceberg-core: table metadata JSON, manifests, commit operations, REST client; Jackson, HTTP client, Avro.
• iceberg-arrow, iceberg-spark, and the other engine modules: vectorized readers and engine adapters.

Proposal

1. Isolate packages, no new modules yet. Move the Arrow-dependent code in the root package (variant, decimal128 conversions, geoarrow edge-interpolation constants) and the scan and execution code in table into engine-facing sub-packages, so that iceberg, the metadata operations in table, and catalog/rest compile with the standard library plus the pure-Go JSON and Avro path. Whether a separate go.mod follows is a later decision, per Enormous dependency tree — consider "drivers"? #696.
2. In-process REST catalog testkit. An in-memory implementation of the REST catalog spec (config endpoint, namespaces, tables, commits with requirement validation) exposed as an http.Handler, usable from httptest in this repo's tests and importable by downstream projects.

Next steps

I would like to contribute this incrementally:

1. The testkit first, since it is additive and unblocks container-less end-to-end client tests.
2. Then the package isolation, one dependency edge at a time, each PR keeping go build ./... and the existing tests green.

Would the maintainers be open to this direction?

[zeroshade]

Thanks for the proposal @slachiewicz I'm definitely open to this direction and in-fact it does fall somewhat into the planning we've been doing for a v1.0.0 release of iceberg-go. We were already planning on package reorganization, some of which was to create engine-facing sub-packages to better isolate the various dependencies and APIs.

I believe the testkit is definitely the right next step, but we should all coordinate together including with @laskoviymishka and @nssalian on the package isolation so that we're all on the same page for the different sub-packages and reorganization.

[laskoviymishka]

Thanks @zeroshade, happy to coordinate here. I did a quick prototype first to get real numbers before we settle on ordering.

What I found:

For _ "catalog/rest" on recent main, the main dependency groups are roughly:

• AWS SDK: ~55 packages
• arrow-go: ~44
• parquet: ~15
• substrait: ~11
• geoarrow: 1

So the biggest dependency for a REST/metadata-only consumer is actually AWS, pulled in only for SigV4. Removing Arrow alone does not get us close to a tiny binary; I get roughly 33–37 MB as the floor. The main Arrow wins are js/wasm support, dropping substrait, and a cleaner graph.

I also have two small changes working locally:

1. Drop pterm from table
   Use text/tabwriter for schema compatibility output. This removes atomicgo.dev/keyboard and makes table / catalog/rest build on js/wasm. No API change.

2. Make SigV4 optional
   Move AWS signing into catalog/rest/sigv4, with a small signer interface in catalog/rest. This removes AWS SDK from REST unless SigV4 is used. rest.WithAwsConfig changes here, so I think this belongs before v1.

For ordering, I’d do:

1. testkit first
2. pterm + substrait/Arrow isolation
3. SigV4 in parallel
4. decide separately whether a fully Arrow-free metadata core is actually a v1 goal

The last one is the harder part. VariantLiteral and DecimalLiteral expose parquet/Arrow types today, so removing the remaining dependency means changing the public Literal API. If zero-Arrow core is a v1 goal, we should decide that now; otherwise I’d stop before that.

Small scope-creep / FYI for the v1 discussion: since we are already looking at pre-v1 API breaks, there are a few other things probably worth tracking in a separate v1 cleanup issue rather than mixing into this one:

• io.IO context handling
• catalog capability interfaces for views / RegisterTable
• duplicate table write API names
• a couple of small public leaks / awkward names
• removing the deprecated compatibility shims that are still around

I would not block v1 on the full table package split or on removing Arrow from the scan path. Those feel like separate follow-ups.

I pushed the two prototypes as PRs (#2060 and #2061) so we have something concrete to review. @nssalian for visibility.

[zeroshade]

I don't think that a zero-Arrow core is a goal, personally. I prefer the consolidation of using the Arrow originated types for decimal/variant/geospatial/geometry rather than duplicating that, rather extensive, amount of work.

Most of the large parts of arrow-go will get optimized out when compiling cute the core iceberg package anyways if we're only using the types.

I also don't think we need to, or even should, have a non-arrow scan path. As you noted, AWS is the largest contributor to the size of the binary.