Draft design: separate portable session metadata from catalog request credentials

[alexanderbianchi]

Summary

Separate portable, request-scoped session metadata from the fully materialized, credential-bearing context passed to SessionCatalog operations.

The current SessionContext stores session identity, properties, and credentials together. These values have different lifecycle and transport requirements: an embedding application may need session metadata on execution workers, while catalog credentials such as a JWT should normally remain coordinator-local.

Portable session metadata

pub struct SessionMetadata {
    session_id: String,
    identity: Option<String>,
    properties: HashMap<String, String>,
}

This is request-scoped but credential-free. An embedding application can choose to propagate it to workers.

"Portable" means that this type is structurally separated from catalog credentials. It should not imply automatic serialization or propagation: identity and properties may still be sensitive, and the embedding application remains responsible for deciding whether and how to encode them.

Fully materialized catalog request context

pub struct SessionContext {
    metadata: SessionMetadata,
    credentials: HashMap<String, SensitiveString>,
}

This is constructed only where a SessionCatalog operation is about to occur:

SessionMetadata
  + coordinator-local credentials
  → SessionContext
  → SessionCatalog::load_table/update_table

Only the second type contains the JWT or other catalog credential material.

Ideally Iceberg core would make the separation additive:

impl SessionContext {
    pub fn metadata(&self) -> &SessionMetadata;
}

and support construction from metadata:

SessionContext::builder()
    .metadata(metadata)
    .credentials(credentials)
    .build()

Existing SessionContext accessors such as session_id(), identity(), properties(), and credentials() can remain available, with the metadata accessors delegating to the contained SessionMetadata. Existing builder methods can remain for compatibility while the new metadata-based construction path is added.

DataFusion session ID invariant

The DataFusion integration should make the safe default:

datafusion Session::session_id()
  == Iceberg SessionMetadata::session_id()
  == Iceberg SessionContext::session_id()

This was a useful property of the DataFusion-specific options proposed in apache#3000: the options did not accept a session ID, and the integration constructed SessionContext with session.session_id(). Installing a complete caller-built Iceberg SessionContext instead would let an operator accidentally reuse one Iceberg session ID across distinct DataFusion sessions, which is unsafe for auth managers or catalogs that cache contextual state by session ID.

Equality is an integration default rather than a universal Iceberg requirement: an embedding application may intentionally define a different logical session scope. The important requirements are stable scope and prevention of one ID being reused with different credentials.

DataFusion's current AsyncCatalogProvider::resolve() receives &SessionConfig, not &dyn Session, so it cannot itself read Session::session_id(). The standalone integration must resolve this API gap explicitly, for example by resolving against an already-created DataFusion session/state, having the application assign one shared session ID to both objects, or proposing that the async resolver receive a DataFusion Session. It should not silently generate an unrelated Iceberg ID.

How AuthManager plugs in

This separation complements the contextual REST authentication hook proposed in apache/iceberg-rust#3081.

There are two authentication scopes:

• the long-lived AuthManager and catalog-wide AuthSession own node/catalog authentication, such as mTLS, service identity, or a rotating service-token provider;
• the fully materialized SessionContext contains credentials specific to one request/session, such as a user JWT or delegation token.

For each SessionCatalog operation, the REST catalog passes the complete context to the auth manager:

SessionCatalog::load_table(&context, table_ident)
  → AuthManager::contextual_session(&context, catalog_session)
  → contextual AuthSession
  → AuthSession::authenticate(&mut HttpRequest)
  → REST catalog request

A contextual auth manager can combine the two scopes without storing mutable per-user state in the shared manager:

async fn contextual_session(
    &self,
    context: &SessionContext,
    catalog_session: Arc<dyn AuthSession>,
) -> Result<Arc<dyn AuthSession>> {
    let request_credential = context
        .credentials()
        .get("request-token")
        .cloned();

    Ok(Arc::new(ContextualAuthSession {
        catalog_session,
        request_credential,
        service_token_provider: Arc::clone(&self.service_token_provider),
    }))
}

The returned AuthSession applies catalog-wide authentication and request-specific authentication to the outgoing HTTP request. Rotating service credentials remain in the long-lived provider and are evaluated at request time; they are not copied into every SessionContext.

SessionMetadata alone is not sufficient to authenticate a catalog request and should not expose the credential map. Code that only receives metadata cannot accidentally obtain or serialize the catalog JWT.

DataFusion and distributed execution

A DataFusion integration could install SessionMetadata as an opaque typed SessionConfig extension. If workers require it, the embedding distributed framework can explicitly opt that type into propagation with a versioned typed codec, such as the SessionExtensionCodec shape proposed in datafusion-contrib/datafusion-distributed#725.

The credential-bearing SessionContext should instead be constructed on the coordinator immediately before asynchronous catalog resolution or metadata mutation. It should not have a default distributed encoding.

coordinator SessionConfig
  └── SessionMetadata
        ├── optional explicit propagation to workers
        └── + coordinator-local credentials
              → SessionContext
              → SessionCatalog / AuthManager

Non-goals

• Automatically serializing or propagating SessionMetadata.
• Making session metadata user-settable through string configuration or SQL SET.
• Defining application-specific JWT, identity, attribution, or organization fields.
• Sending catalog credentials to execution workers.
• Moving rotating node/service credentials out of AuthManager.

Acceptance criteria

• Add a public credential-free SessionMetadata type containing session ID, identity, and properties.
• Store SessionMetadata within SessionContext alongside its credential map.
• Add SessionContext::metadata().
• Support constructing SessionContext from existing SessionMetadata and credentials.
• The DataFusion integration derives Iceberg's session ID from the DataFusion session by default, or otherwise requires an explicit shared logical session ID.
• Document how async pre-planning resolution obtains that ID despite the current AsyncCatalogProvider::resolve(&SessionConfig, ...) signature.
• Preserve the existing SessionContext accessors and a compatible builder path.
• Keep credentials represented as SensitiveString values and excluded from SessionMetadata.
• Document that metadata propagation is explicit application policy, not automatic behavior.
• Document how feat(rest_catalog::auth) Contextual Sessions apache/iceberg-rust#3081's AuthManager::contextual_session combines catalog-wide authentication with the fully materialized request context.
• Add tests proving metadata can be cloned/reused without exposing credentials and that full contexts retain existing session ID, identity, property, and credential behavior.

Related work

• apache/iceberg-rust#3081: contextual REST authentication sessions.
• apache/iceberg-rust#3000: DataFusion session-catalog integration and session options.
• datafusion-contrib/datafusion-distributed#725: explicit typed session-extension serialization.

AI disclosure

This design draft was developed with assistance from an AI coding assistant and reviewed by the contributor.