Document DataFusion integration refactor notes

[alexanderbianchi]

Captures architectural notes and a proposed refactor sequence for the Iceberg DataFusion integration.

This is a draft for safekeeping and further iteration.

There are 3 actors to note

1. The SQL end user.
2. The application embedding DataFusion.
3. The Iceberg/DataFusion integration.

A SQL user generally should not deal with sessions or credentials. But the
caller constructing and registering IcebergCatalogProvider is not that SQL
user—it is the embedding application or framework.

That application already controls:

• SessionConfig;
• DataFusion SessionContext creation;
• catalog registration;
• request authentication;
• trusted versus user-controlled metadata;
• query planning and execution placement;
• distributed credential propagation.

DataFusion is not Trino or Spark. It does not provide a standard
authenticated-session model, deployment architecture, coordinator/worker
boundary, or credential propagation protocol. Those are responsibilities of
systems built around it, such as df-executor, Spice, or Sail.

This makes a generic mapping like:

  user-settable DataFusion options → authenticated Iceberg identity

a questionable default. Iceberg cannot know:

• which values were authenticated;
• which values came from SQL SET;
• whether a session maps to one request, user, tenant, or connection;
• whether workers should receive those credentials;
• whether a reused session ID has stable authorization.

The safer integration contract is:

  Embedding application authenticates request
      ↓
  Application constructs trusted SessionContext
      ↓
  Application inserts it into this query’s SessionConfig
      ↓
  Iceberg consumes it without interpreting application authentication

A single-tenant application can still install an empty or service-level
context programmatically. A higher-level framework can provide convenience
wrappers. But the low-level integration should not weaken the trust
boundary to create an OOTB experience DataFusion itself does not promise.