Draft design: query-scoped asynchronous Iceberg catalogs for DataFusion #3

Summary

Redesign the Iceberg DataFusion catalog integration around query-scoped asynchronous resolution.

A long-lived remote catalog should resolve only the tables referenced by one query, using an immutable request context supplied by the embedding application. Resolution should produce ordinary, query-local DataFusion catalog and table providers containing already-loaded Iceberg tables. Planning and execution should not perform another catalog lookup.

This issue is intended for the standalone Iceberg DataFusion integration project. The current prototype is alexanderbianchi/iceberg-rust#1, and contextual REST authentication is being developed in apache/iceberg-rust#3081.

Motivation

The existing integration models a remote Iceberg catalog as a conventional, long-lived DataFusion CatalogProvider:

1. catalog construction enumerates namespaces;
2. schema construction enumerates tables;
3. table providers are retained indefinitely;
4. IcebergTableProvider::scan reloads table metadata from the catalog;
5. scan file planning may happen later during physical execution.

This creates several problems for remote, multi-tenant catalogs:

• initial discovery may run without the query's authenticated identity;
• loading every namespace and table is unnecessarily expensive;
• persistent providers have undefined authorization, freshness, and eviction semantics;
• TableProvider::schema() and TableProvider::scan() can observe different table metadata;
• table metadata is refreshed during planning even when it was already resolved for that query;
• the embedding application cannot clearly control the request trust boundary;
• physical execution can accidentally retain catalog discovery and scan-planning responsibilities.

DataFusion's AsyncCatalogProvider and AsyncSchemaProvider describe the right broad model—resolve remote objects once before planning—but their current interface is insufficient for this session-aware integration. In particular, resolve() receives only &SessionConfig, not the DataFusion Session whose ID must define the Iceberg session scope. DataFusion also does not discover or invoke async catalog providers automatically; the embedding application must already collect references, call the resolver, and register the returned ordinary provider. The integration can therefore expose a session-aware asynchronous resolver today without claiming conformance to the current trait, then implement the trait when DataFusion exposes the required session information.

Design principles

The embedding application owns policy

The integration is a library. The embedding application owns:

• authentication and authorization policy;
• creation of trusted request context;
• catalog-name routing;
• extracting SQL or Substrait table references;
• calling asynchronous resolution before planning;
• registering the returned query-local provider;
• distributed credential or capability propagation;
• persistent caching, if any.

The integration should not define JWT claims, organization identifiers, attribution headers, or application-specific credential formats.

Resolve catalog state once per query

The intended lifecycle is:

long-lived SessionCatalog
        +
DataFusion Session
        +
portable Iceberg session metadata
        +
coordinator-local catalog credentials
        +
referenced TableReferences
        |
        v
session-aware asynchronous resolver
        |
        v
query-local CatalogProvider
  -> query-local SchemaProviders
    -> immutable TableProviders holding loaded Tables
        |
        v
logical and physical planning

The resolved provider is an immutable planning cache. It is not a persistent metadata cache and needs no refresh or eviction policy.

Catalog lookup belongs before planning

A catalog-backed TableProvider should hold the Table loaded during asynchronous resolution. TableProvider::scan() should not call Catalog::load_table or SessionCatalog::load_table.

This gives one stable metadata version across:

• schema();
• metadata-table planning;
• scan planning;
• insert planning.

A later refresh may still be required for optimistic commit validation and retry handling, but it should not silently change the table used to plan the query.

Scan planning and execution are separate concerns

The desired phase boundary is:

catalog resolution:
  authenticate and load Table

TableProvider::scan():
  select snapshot and plan FileScanTasks

ExecutionPlan::execute(partition):
  read only the preplanned tasks assigned to that partition

Iceberg's ArrowReaderBuilder can remain the file reader because it implements Iceberg-specific schema, delete, encryption, and field-ID behavior. However, TableScan::to_arrow() currently combines plan_files() and reading; the DataFusion integration should eventually invoke those stages separately.

A complete physical-scan redesign can be tracked independently, but this catalog redesign must not require catalog reloading from scan() or execution.

Proposed API shape

Provide a reusable resolver over one session-aware catalog:

pub struct IcebergSessionCatalogResolver {
    catalog: Arc<dyn SessionCatalog>,
}

impl IcebergSessionCatalogResolver {
    pub fn new(catalog: Arc<dyn SessionCatalog>) -> Self;
}

The IcebergOptions proposed in apache#3000 correctly identifies the need for a DataFusion-session carrier, but Iceberg's current SessionContext combines two things with different transport requirements:

1. request-scoped session metadata such as session ID, identity, and non-secret properties, which an application may legitimately propagate to workers;
2. a fully materialized catalog request context, including credentials such as a JWT, which should remain coordinator-local and be passed only to SessionCatalog operations.

The integration should represent those separately:

IcebergSessionMetadata
  session ID, identity, explicitly portable properties
  opaque typed DataFusion extension
  optionally serialized by an explicit application codec

CatalogRequestContext / iceberg::SessionContext
  IcebergSessionMetadata + catalog credentials
  constructed at the coordinator's SessionCatalog call boundary
  never serialized as worker session state

The exact core API can be additive: Iceberg could introduce a credential-free SessionMetadata value contained by SessionContext, while the DataFusion integration exposes the metadata as its typed extension and materializes SessionContext only when resolving or mutating catalog state. Until then, the integration can maintain the split itself.

Neither type should implement DataFusion's ConfigExtension or ExtensionOptions; there should be no string-based set, entries, or SQL SET surface. Portable metadata may opt into distributed propagation through an explicit typed codec such as the SessionExtensionCodec shape proposed in datafusion-contrib/datafusion-distributed#725: a stable versioned type URL plus opaque application-encoded bytes installed into the worker's typed SessionConfig extensions.

The codec should serialize IcebergSessionMetadata directly, not project or serialize the credential-bearing catalog request context. Catalog JWTs remain outside that type by construction. Any delegated worker capability should use its own explicit field or extension with documented transport, audience, scope, expiry, and redaction requirements.

The credential-bearing Iceberg SessionContext constructed during resolution is retained only by coordinator-side resolved providers for later contextual metadata operations such as commit refresh and update_table.

It then resolves each routed catalog after the DataFusion session exists but before logical planning:

let resolved = resolver
    .resolve_with_session(&references, &session_state, "lakehouse", credentials)
    .await?;

session_state
    .catalog_list()
    .register_catalog("lakehouse".to_owned(), resolved);

resolve_with_session() should:

1. ignore references belonging to other catalogs before requesting credentials;
2. read typed IcebergSessionMetadata from the DataFusion session;
3. derive the Iceberg session ID from Session::session_id() rather than accepting an independently configurable ID;
4. combine metadata with coordinator-local catalog credentials to materialize one immutable Arc<iceberg::SessionContext>;
5. fail before catalog I/O when required request state is missing or invalid;
6. load only referenced schemas and tables;
7. deduplicate repeated references;
8. perform no namespace or table enumeration;
9. return DataFusion's ordinary query-local CatalogProvider hierarchy;
10. retain the exact loaded Table and credential-bearing Arc<SessionContext> where later contextual commit operations require them.

There must be no anonymous or shared fallback context. A non-Iceberg query does not construct an Iceberg SessionContext at all.

Today: integration-specific resolver

The current DataFusion async catalog traits are helper abstractions rather than an engine-managed registry. DataFusion does not automatically call an AsyncCatalogProvider; the embedding application must explicitly:

1. extract referenced tables from SQL, Substrait, or another input;
2. call resolve();
3. register the returned ordinary CatalogProvider before planning.

The current AsyncCatalogProvider::resolve() receives &SessionConfig. It can read default catalog/schema options and opaque extensions, but it cannot read Session::session_id(). AsyncCatalogProvider::schema() receives neither. Implementing the current trait would therefore force one of the undesirable choices this design is intended to avoid: independently assigning an Iceberg session ID, pushing ID equality onto every application, constructing context eagerly, or falling back to anonymous state.

For now, the standalone integration should expose resolve_with_session(..., &dyn Session, ...) directly. This does not reduce interoperability after resolution: the result is an ordinary Arc<dyn CatalogProvider>, and all subsequent DataFusion logical and physical planning uses standard interfaces. The only non-standard operation is the explicit asynchronous pre-planning step that applications must perform with today's trait anyway.

With DataFusion changes

DataFusion should add a session-aware async-resolution entry point, ideally across AsyncSchemaProvider, AsyncCatalogProvider, and AsyncCatalogProviderList:

async fn resolve_with_session(
    &self,
    references: &[TableReference],
    session: &dyn Session,
    catalog_name: &str,
) -> Result<Arc<dyn CatalogProvider>>;

This can initially be additive, with a default implementation delegating to the existing config-only method using session.config(). Existing providers remain source-compatible, while providers that need session identity can override the new method. A future breaking release could make &dyn Session the primary input because it provides both session_id() and config().

Once this exists, the Iceberg resolver can implement the standard DataFusion trait without changing its lifecycle or application orchestration.

REST authentication

SessionCatalog context propagation and REST request authentication are separate responsibilities.

With apache/iceberg-rust#3081, a REST AuthManager can derive a contextual AuthSession for each catalog operation. This makes two authentication scopes explicit:

• node/catalog authentication is owned by the long-lived catalog and AuthManager (for example mTLS, a service/ISA identity, or a rotating service-token provider);
• query/session metadata comes from the typed, optionally portable IcebergSessionMetadata extension;
• query catalog credentials are combined with that metadata only at the coordinator's SessionCatalog boundary to create the full Iceberg SessionContext.

AuthManager::contextual_session combines those scopes for one catalog operation. Rotating node credentials should be evaluated at HTTP-request time rather than copied into every query context.

There must be no anonymous fallback for a referenced session catalog.

The credential-free session metadata may be sent to workers only through an explicitly registered typed codec. The resulting credential-bearing Iceberg SessionContext must not be encoded. Catalog resolution and commit should normally remain coordinator-local.

The boundary should follow the same property-ownership principle highlighted by apache/iceberg-rust#3105: catalog credentials are consumed by the catalog authentication layer and must not leak into generic FileIO or KMS property maps.

FileIO and storage credentials

Catalog authentication and storage authorization are distinct:

catalog request credentials
  -> authenticated load_table
  -> optional vended, table-scoped storage credentials
  -> Table/FileIO

The original catalog JWT or attribution should not be copied into FileIO. If the REST server vends storage credentials, the resolved Table should carry only the resulting storage capability.

apache/iceberg-rust#3129 proposes typed FileIO storage configuration with sensitive values marked explicitly. That work is complementary and becomes especially important if planned scans are serialized for distributed execution.

Writes and commits

Writes should use the same resolved table metadata for insert planning. Catalog commit remains a coordinator-side metadata operation and must preserve the request context.

The contextual path should use one transaction algorithm with contextual dispatch only for catalog operations:

resolved Table
  -> write data files
  -> refresh with SessionCatalog + SessionContext
  -> reapply/validate transaction requirements and updates
  -> update_table with the same SessionContext

The current prototype exposes Transaction::commit_with_session() while sharing the retry, action, requirement, and update logic with Transaction::commit().

Workers that only read or write data files should receive planned work and appropriately scoped storage capabilities, not the original catalog credentials.

Existing providers and compatibility

The standalone integration should distinguish three concepts rather than combine them in one provider:

1. Loaded/static table provider: wraps an already-loaded Table and performs no catalog operations.
2. Session catalog resolver: resolves authenticated query-local providers through an integration-specific resolve_with_session() today and the standard async catalog interface once it exposes &dyn Session.
3. Legacy plain catalog provider: retained only if compatibility requires it.

If the existing plain provider remains, its behavior should not force the session-aware path to support eager enumeration, anonymous fallback, scan-time rebinding, or persistent caching.

A future major version of the standalone integration should consider making asynchronous resolution canonical for remote catalogs and deprecating eager remote catalog construction.

Non-goals

• Defining an application-specific authentication scheme.
• Automatically parsing or forwarding arbitrary HTTP/gRPC headers.
• Automatically serializing request credentials to distributed workers.
• Adding a cross-query metadata or authorization cache.
• Performing eager authenticated namespace/table discovery.
• Adapting SessionCatalog + SessionContext into a broad Catalog implementation.
• Replacing Iceberg's Arrow reader before equivalent Iceberg semantics are available through a DataFusion-native file source.

Acceptance criteria

• A reusable asynchronous resolver resolves one SessionCatalog and returns an ordinary query-local CatalogProvider.
• The current implementation receives &dyn datafusion::catalog::Session and derives the Iceberg session ID from Session::session_id().
• The design documents why today's config-only AsyncCatalogProvider is not implemented and how to adopt a future session-aware method.
• Request-scoped session metadata and credential-bearing catalog request context are separate types or layers.
• Resolution combines typed session metadata with coordinator-local credentials and materializes one immutable Arc<SessionContext> at the SessionCatalog boundary.
• Neither uses ConfigExtension or ExtensionOptions, and neither exposes a string-based set/entries surface.
• The credential-bearing Iceberg SessionContext has no distributed encoding.
• Portable session metadata may use an explicitly registered, versioned typed extension codec such as the shape in datafusion-distributed#725.
• Catalog credentials are absent from the portable metadata type by construction.
• Missing context fails before any I/O for references routed to that catalog.
• References for unrelated catalogs do not require Iceberg context or cause Iceberg I/O.
• Only referenced tables are loaded; no namespace/table enumeration occurs.
• Repeated references are deduplicated during resolution.
• Resolved table and metadata-table providers retain the table loaded during resolution.
• TableProvider::scan() and insert planning perform no additional catalog load.
• Concurrent queries over one shared SessionCatalog cannot mix identities.
• Commit refresh and update_table use the same request context.
• The existing transaction retry and optimistic-concurrency behavior is preserved.
• Catalog credentials are not exposed as string-settable DataFusion configuration.
• Catalog credentials are not forwarded into FileIO properties.
• Documentation includes an end-to-end example: collect references, attach context, resolve, register, and plan.

Suggested implementation sequence

1. Add an integration-specific resolve_with_session() resolver and query-local providers without depending on today's config-only AsyncCatalogProvider contract.
2. Retain one resolved Table through scan and insert planning.
3. Add contextual transaction commit support without duplicating transaction logic.
4. Integrate contextual REST authentication from feat(rest_catalog::auth) Contextual Sessions apache/iceberg-rust#3081.
5. Add a runnable DataFusion example showing explicit pre-planning orchestration.
6. Separately move FileScanTask planning from physical execution into TableProvider::scan().
7. Define a typed and secure distributed representation for storage access, coordinated with FileIO should store a concrete storage config rather than raw hashmap. apache/iceberg-rust#3129.

Open questions

1. Should the standalone project retain the eager plain catalog provider indefinitely, deprecate it, or remove it at its next breaking release?
2. Should the project provide only a one-catalog resolver, or also an optional routing helper over multiple asynchronous providers?
3. Should contextual commit remain an explicit commit_with_session() method or become a more general transaction catalog-operation abstraction?
4. What portion of the scan-planning refactor should be required before the standalone project considers distributed physical-plan serialization supported?

Prototype validation

The current prototype reports:

• 94 unit tests passed;
• 9 integration tests passed;
• 1 doctest passed, with 5 existing doctests ignored;
• Clippy with -D warnings passed;
• formatting passed;
• Iceberg and DataFusion public API snapshots passed.

Tests cover missing context, exact Arc retention, no enumeration, stable metadata, request counts, contextual scans/inserts/commits, concurrent identity isolation, and unrelated catalog references.

AI disclosure

This design and the linked prototype were developed with assistance from an AI coding assistant and reviewed and validated by the contributor.