SS-471 Add support for vending (and refreshing) credentials for Iceberg catalogs backed by Azure Data Lake Storage (V2)

[patrickwwbutler]

Implements the VendedCredential trait for the newly created CustomAzdlsCredentialLoader added to the MZ iceberg-rust fork in MaterializeInc/iceberg-rust#9

Then plugs in a CustomAzdlsCredentialLoader with the previously created VendedCredentialLoader using this trait implementation into the OpenDalStorageFactory::Azdls constructor, allowing opendal to call it for fresh vended credentials from the catalog.

Tested manually in our Azure Databricks workspace, with optimistic plans to add CI testing later on.

Also closes SS-173

[def-]

QA LLM Review

1. MEDIUM -- ADLS from_vended rejects catalogs that alias one SAS token under several adls.sas-token.* keys

src/storage-types/src/connections/iceberg_credentials.rs:134

Vending catalogs routinely publish the same SAS token under more than one adls.sas-token.<suffix> key for client compatibility. The new loop treats the second such key as a fatal ambiguity, so against those catalogs the loader never yields a credential and every ADLS storage request fails — a regression, because before this PR the same response worked through the static props.

Details

Apache Polaris AzureCredentialsStorageIntegration.handleAzureCredential emits, for a single ADLS location, all of:

• adls.sas-token.<account>.dfs.core.windows.net → token
• adls.sas-token.<account> → the same token (its comment: "Iceberg 1.7.x may expect the credential key to not be suffixed with endpoint")
• adls.sas-token (bare) → the same token
• adls.sas-token-expires-at-ms.<account>.dfs.core.windows.net

The first two both satisfy prop.starts_with("adls.sas-token."), so from_vended returns credential_invalid on the second one it happens to see. reqsign_core::ProvideCredentialChain::provide_credential swallows provider errors and moves on (reqsign-core-3.3.x api.rs:494), and the fork's azdls_config_build replaces the whole chain when a loader is installed rather than prepending to it, so there is no fallback: the chain returns None and the operator fails with a generic "signing credential" error. The real cause only shows up in the warn! inside VendedCredentialLoader::provide_credential.

Note also that the bare adls.sas-token key (no trailing dot) is what iceberg-rust's azdls_config_parse reads (ADLS_SAS_TOKEN = "adls.sas-token"), and the REST catalog merges cred.config into the per-prefix FileIO props. That is why Polaris + ADLS worked pre-PR and stops working now.

Databricks/Unity Catalog appears to emit only the host-suffixed key, which is consistent with the manual testing reported in the PR description.

Suggested fix: only the aliases disagreeing on the token value are genuinely ambiguous, and the bare key is worth honoring as a fallback.

let mut sas_token: Option<&String> = None;
for (prop, val) in vended.config.iter() {
    if prop.starts_with("adls.sas-token.") {
        // Catalogs alias one token under several keys (host-suffixed, account-suffixed,
        // bare) for client compatibility. Only differing values are ambiguous.
        if sas_token.is_some_and(|seen| seen != val) {
            return Err(reqsign_core::Error::credential_invalid(format!(
                "vended Iceberg storage credential for prefix {} has conflicting \
                 adls.sas-token.<account> properties",
                vended.prefix
            )));
        }
        sas_token = Some(val);
    }
    // ... expiry handling unchanged
}
let sas_token = sas_token.or_else(|| vended.config.get("adls.sas-token"));

A unit test alongside test_aws_credential_from_vended / test_gcs_credential_from_vended covering the Polaris key set would pin this down; there is currently no test for the ADLS impl.

2. LOW -- TODO(SS-449) says ADLS has no loader, directly above the code that installs one

src/storage-types/src/connections.rs:1228

The comment states that "vended credentials here still expire without one" and that wiring one up "needs a VendedCredential mapping from the catalog's adls.sas-token.<account> prop onto Credential::SasToken" — which is exactly what this PR adds two lines below and in iceberg_credentials.rs:126. Left as is, the next reader will believe ADLS refresh is unimplemented.

3. LOW -- the iceberg-rust rev bump is not a fast-forward and silently reverts fork PR #7

Cargo.toml:679

eafda25 is not a descendant of the currently pinned 985929c (their merge base is upstream 2cf6128), so the bump drops fork commit 5c5c189 "make Transaction::commit internals public" along with picking up the ADLS work. iceberg::transaction::TransactionAction, RowDeltaAction and TableCommit's public builder go back to pub(crate) — visible in the crates/iceberg/public-api.txt delta.

Details

Nothing in this tree names those items today (src/storage/src/sink/iceberg.rs:108 only uses ApplyTransactionAction and Transaction, both still public), so there is no functional impact. Flagging because a rev bump shows up in the diff as one hex string and the revert is otherwise invisible: worth confirming it was intentional rather than a rebase casualty.

[martykulma]

aside from the comment, the rest looks good! thanks @patrickwwbutler