Draft: resolve Iceberg session catalogs per DataFusion query

[alexanderbianchi]

Purpose

Draft a query-scoped alternative to the scan/insert-time rebinding in apache/iceberg-rust#3000.

This personal-fork PR is based directly on Apache iceberg-rust main and contains no commits from apache#3000.

Model

Applications place Iceberg's existing Arc<iceberg::SessionContext> directly into DataFusion's typed SessionConfig extensions:

let context = Arc::new(
    iceberg::SessionContext::builder()
        .identity("alice".to_owned())
        .properties(properties)
        .credentials(credentials)
        .build(),
);
let config = SessionConfig::new().with_extension(context);

One shared resolver owns the session-aware catalog:

let iceberg = IcebergSessionCatalogProvider::new(shared_session_catalog);

Before SQL/Substrait planning, the caller resolves this catalog's references:

let provider = iceberg
    .resolve(&references, &config, "iceberg")
    .await?;

resolve retrieves the exact Arc<SessionContext> from SessionConfig, loads only the referenced tables under that immutable context, and returns an ordinary query-scoped CatalogProvider. The same context reaches initial table loading, metadata-table loading, scan refresh, insert loading, and commit/update.

Catalog-name routing remains the caller's responsibility. An application with multiple catalogs can register the returned provider in its query-local ordinary CatalogProviderList. DataFusion does not automatically invoke async providers.

Preserve the existing plain provider

The existing IcebergCatalogProvider::try_new(Arc<dyn Catalog>) and its eager namespace/table listing behavior are unchanged. Existing schema_names, table_names, table_exist, registration, and deregistration behavior and tests are preserved.

The new session-aware path binds the request context into private AsyncCatalogProvider/AsyncSchemaProvider implementations, then delegates grouping, deduplication, and immutable query-local cache construction to DataFusion's default AsyncCatalogProvider::resolve. No mutable custom provider maps are maintained by this integration.

Shared table and commit implementation

The existing IcebergCatalogProvider remains plain-only and unchanged.

A single private IcebergTableSource represents the two valid table-provider lifetimes:

enum IcebergTableSource {
    RefreshingCatalog {
        catalog: Arc<dyn Catalog>,
        table_ident: TableIdent,
    },
    ResolvedSession {
        catalog: Arc<dyn SessionCatalog>,
        context: Arc<SessionContext>,
        table: Table,
    },
}

This collapses the former independent catalog-access and metadata-lifetime modes, making invalid combinations unrepresentable. It deliberately does not implement Catalog, so there is no SessionCatalog + SessionContext -> Catalog adapter.

Existing plain providers continue refreshing metadata on every scan/write. Query-resolved session providers retain and reuse the exact Table loaded during async resolution for metadata tables, scans, and insert planning. This avoids redundant requests and prevents schema/metadata drift between logical and physical planning.

Transaction::commit and the new Transaction::commit_with_session are thin entry points into one shared private transaction algorithm. Refresh, retries, action application, requirements, updates, and final commit construction are identical; only contextual load_table and update_table dispatch differs.

No persistent session cache

The session-aware resolver does not enumerate all namespaces/tables and does not retain cross-query providers or metadata. Persistent cache scope, authorization keys freshness, invalidation, and vended credentials remain the responsibility of SessionCatalog users and concrete REST/catalog implementations.

DataFusion trait behavior

IcebergSessionCatalogProvider implements AsyncCatalogProvider, which represents one catalog. Its resolve(references, config, catalog_name) is the authenticated entry point.

The required schema() method cannot safely resolve a session catalog because DataFusion does not pass SessionConfig to it. It therefore reports that resolve() is required rather than introducing an anonymous fallback. References belonging to another catalog return an empty resolved provider without requiring an Iceberg context.

Authentication boundary

This change propagates Iceberg SessionContext; it does not turn credentials into HTTP headers. The contextual REST authentication hook proposed in #3081, or an equivalent context-aware REST implementation, remains required.

With apache#3081:

• query JWT/attribution live in immutable SessionContext credentials;
• organization and other non-sensitive attributes may use identity/properties;
• the rotating service bearer comes from the shared auth session/token provider at request time;
• /v1/config uses service/bootstrap authentication;
• subsequent query catalog operations use contextual authentication.

No application-specific JWT, attribution, or organization fields are introduced in OSS.

Validation

• cargo test -p iceberg-datafusion: 94 unit tests + 9 integration tests + 1 doctest passed; 5 existing doctests ignored.
• cargo clippy -p iceberg-datafusion --all-targets -- -D warnings: passed.
• cargo fmt -p iceberg-datafusion -- --check: passed.
• cargo public-api -p iceberg-datafusion --all-features -ss: matches the updated snapshot.

New tests verify:

• missing SessionContext fails before catalog I/O instead of falling back anonymously;
• DataFusion returns the exact Arc<SessionContext> installed by the application;
• only referenced tables are loaded, with no namespace/table enumeration;
• query-resolved scans and metadata tables perform no load after resolution;
• insert planning reuses resolved metadata while commit refreshes exactly once before contextual update;
• SQL scan and insert/commit retain identity, properties, credentials, and session ID through update_table;
• concurrent resolutions over one shared SessionCatalog do not mix identities;
• unrelated catalog references require no Iceberg context or I/O.

Scope and limitations

• The caller must collect SQL/Substrait TableReferences, call resolve, and install the returned provider before planning.
• This does not add feat(rest_catalog::auth) Contextual Sessions apache/iceberg-rust#3081, a REST auth implementation, persistent caching, or distributed-worker credential propagation.
• Performance has not yet been benchmarked; whole-catalog discovery is avoided by construction.

Change size

Production Rust changes are approximately +334/-44. Existing plain catalog/schema behavior remains intact; the new code adds DataFusion-native async resolution, one table-source lifecycle abstraction, and shared contextual transaction dispatch. The remaining lines are tests and generated API output.

AI disclosure

Implemented and validated with an AI coding assistant; all commands reported above were run locally.

[DerGut]

Nice I think this may be useful for usptream too! API-wise, I've also played with this but a pub fn might be simpler

impl dyn SessionCatalog {
    /// Bind this catalog to a session, exposing the ordinary Catalog API.
    ///
    /// # Example
    /// ```
    /// # fn into_catalog(session_catalog: Arc<dyn SessionCatalog>, id: String) {
    /// let session = SessionContext::builder().session_id(id).build();
    ///
    /// // Use the plain catalog API for the duration of this session.
    /// let catalog = session_catalog.into_catalog(session);
    /// # let _ = catalog;
    /// # }
    /// ```
    pub fn into_catalog(self: Arc<Self>, session: SessionContext) -> Arc<dyn Catalog> {
        Arc::new(SessionBoundCatalog {
            inner: self,
            session,
        })
    }
}

[alexanderbianchi]

What I was trying to play with was is there was a way to kinda work around datafusion's (limited) catalog API and just have it handled in the service layer. apache#3081 is still going to be needed from what I can tell. There is this trait

pub trait AsyncCatalogProviderList: Send + Sync {
    // Required method
    fn catalog<'life0, 'life1, 'async_trait>(
        &'life0 self,
        name: &'life1 str,
    ) -> Pin<Box<dyn Future<Output = Result<Option<Arc<dyn AsyncCatalogProvider>>, DataFusionError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             Self: 'async_trait;

    // Provided method
    fn resolve<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        references: &'life1 [TableReference],
        config: &'life2 SessionConfig,
    ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn CatalogProviderList>, DataFusionError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait,
             Self: 'async_trait { ... }
}

which I think we could use build the "bound" catalog.

[alexanderbianchi]

https://docs.rs/datafusion/latest/datafusion/catalog/trait.AsyncCatalogProvider.html
maybe should just do the provider not the List

[alexanderbianchi]

Hm... At first I didn't like the enum but it's growing on me.

[alexanderbianchi]

Lots of stripping of the schema provider here isn't great

[alexanderbianchi]

I think this might be 1 extra level of indirection I think we could probably do something closer to the actual datafusion catalog provider

[alexanderbianchi]

I don't want to have to touch this file

[alexanderbianchi]

why do we ever have to load table after async provider

[alexanderbianchi]

AI is informing me it's becuase the table provider currently reloads the table whenever it's called so table provider are setup to refresh. THis... seems silly to me. SHould be table provider per request