mixermt · Sep 13, 2026 · Sep 13, 2026 · Sep 13, 2026 · Sep 13, 2026 · Sep 15, 2026
diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml
@@ -467,6 +467,7 @@ jobs:
               org.apache.comet.CometVariantProjectionSuite
               org.apache.spark.sql.CometVariantShreddingSuite
               org.apache.comet.CometIcebergNativeSuite
+              org.apache.comet.CometIcebergHdfsSuite
               org.apache.comet.CometIcebergEncryptionSuite
               org.apache.comet.CometIcebergRewriteActionSuite
               org.apache.comet.CometIcebergWriteActionSuite

diff --git a/.github/workflows/pr_build_macos.yml b/.github/workflows/pr_build_macos.yml
@@ -171,6 +171,7 @@ jobs:
               org.apache.comet.CometVariantProjectionSuite
               org.apache.spark.sql.CometVariantShreddingSuite
               org.apache.comet.CometIcebergNativeSuite
+              org.apache.comet.CometIcebergHdfsSuite
               org.apache.comet.CometIcebergEncryptionSuite
               org.apache.comet.CometIcebergRewriteActionSuite
               org.apache.comet.CometIcebergWriteActionSuite

diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md
@@ -67,10 +67,7 @@ would remove JVM round-trips beyond those the lambda work above addresses.
 Comet's native Iceberg scans read V3 tables, including encrypted tables ([#4991]) and tables with deletion vectors
 ([#5853]). We want to add the remaining V3 features so that these scans don't fall back to Spark: row lineage
 metadata columns, column default values, and the new V3 types (`variant`, `geometry`, `geography`, and `unknown`).
-The work is tracked in [#3376], and upstream `iceberg-rust` support in [iceberg-rust #2411]. Native Iceberg scans
-also don't support HDFS-backed tables today: Comet's native Iceberg storage layer handles only local files, S3 and
-S3-compatible stores, GCS, and OSS, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`.
-We're scoping what that work would take.
+The work is tracked in [#3376], and upstream `iceberg-rust` support in [iceberg-rust #2411].
 
 [#3376]: https://github.com/apache/datafusion-comet/issues/3376
 [#4991]: https://github.com/apache/datafusion-comet/pull/4991

diff --git a/docs/source/contributor-guide/s3-credential-provider-design.md b/docs/source/contributor-guide/s3-credential-provider-design.md
@@ -142,7 +142,7 @@ The 5-minute fallback on the Iceberg path bounds how long reqsign reuses a crede
 
 ## Property-bag handling on the Iceberg path
 
-The full unfiltered FileIO property bag crosses JNI as `catalog_properties`. The storage-prefix filter (`s3.`/`gcs.`/`adls.`/`client.`/`opendal.`) is applied native-side in `iceberg_common.rs::build_file_io` immediately before `FileIOBuilder.with_prop`. This means the bridge sees `credentials.uri`, OAuth tokens, and any vendor-custom keys with no parallel field on the operator and no driver-side broadcast. Vendors set their own keys on the catalog config and read them back inside `initialize(Map)`.
+The full unfiltered FileIO property bag crosses JNI as `catalog_properties`. The storage-prefix filter (`s3.`/`gcs.`/`adls.`/`client.`/`opendal.`/`hdfs.`/`hadoop.`) is applied native-side in `iceberg_common.rs::build_file_io` immediately before `FileIOBuilder.with_prop`. This means the bridge sees `credentials.uri`, OAuth tokens, and any vendor-custom keys with no parallel field on the operator and no driver-side broadcast. Vendors set their own keys on the catalog config and read them back inside `initialize(Map)`.
 
 `IcebergScanExec` derives a redacting `Debug`, and the `FileIO` cache key's `Debug` omits the property bag, so plan dumps and tracing do not leak it.
 

diff --git a/docs/source/user-guide/latest/datasources.md b/docs/source/user-guide/latest/datasources.md
@@ -112,8 +112,9 @@ attaches its own worker threads, so a worker that has read from HDFS can crash t
 the HDFS read itself, typically while an unrelated query is running.
 ```
 
-Native Iceberg scans do not support HDFS-backed tables; those scans fall back to Spark. See the
-[Comet and Iceberg Guide](iceberg.md).
+Native Iceberg scans and writes on `hdfs://` tables do not go through `libhdfs`. They use iceberg-rust's
+own pure-Rust HDFS client, which is part of every Comet build and is configured separately from the
+settings on this page. See [Object store configuration (HDFS)](iceberg.md#object-store-configuration-hdfs).
 
 ### Building Comet with HDFS support
 

diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md
@@ -193,7 +193,7 @@ A write is eligible only when ALL of the following hold:
 | `write.metadata.metrics.*`                                                                                                                  | any value (manifest metrics are re-derived on the JVM with Iceberg's own logic)                                                                                                                                                                                                                                                                                                                                                                                                 |
 | `write.spark.fanout.enabled`                                                                                                                | any value (the native writer implements both clustered and fanout modes)                                                                                                                                                                                                                                                                                                                                                                                                        |
 | `write.target-file-size-bytes`                                                                                                              | any value (the two writers can choose different roll points; see accepted divergences)                                                                                                                                                                                                                                                                                                                                                                                          |
-| data location URI scheme                                                                                                                    | `file`, `memory`, `s3`, `s3a`, `gs`, matched case-sensitively (`S3://` falls back). `s3`, `s3a` and `gs` need a bucket in the authority (`s3://bucket/...`), so a hostless form such as `s3:/bucket/key` falls back. `gs` only when the `FileIO` opening the data location is a `GCSFileIO`; see below                                                                                                                                                                          |
+| data location URI scheme                                                                                                                    | `file`, `memory`, `s3`, `s3a`, `gs`, `hdfs`, matched case-sensitively (`S3://` falls back). `s3`, `s3a`, `gs` and `hdfs` need an authority (`s3://bucket/...`, `hdfs://namenode:8020/...`), so a hostless form such as `s3:/bucket/key` or `hdfs:///path` falls back. `gs` only when the `FileIO` opening the data location is a `GCSFileIO`, and `hdfs` only when the native HDFS client can reach the NameNode; see below                                                                                                                                                                          |
 | resolved `table.locationProvider()`                                                                                                         | Iceberg's built-in `DefaultLocationProvider`                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | Hadoop S3A settings for an `s3` / `s3a` data location                                                                                       | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back                                                                                                                                                                                                                                                       |
 | Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location                                                                              | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` / `client.*` properties are also forwarded; unsupported Iceberg-defined S3 settings still fall back                 |
@@ -226,6 +226,20 @@ could resolve a different storage identity or endpoint than the JVM writer would
 combination falls back; a `GCSFileIO` carries its `gcs.*` settings in `FileIO.properties()`,
 which are forwarded.
 
+An `hdfs` data location is checked at planning for whether the native HDFS client could reach its
+NameNode. A location with no authority (`hdfs:///...`) falls back, even when `hdfs.name-node`,
+`hdfs.host`, `hdfs.port` or `hadoop.fs.defaultFS` is set. A catalog `hdfs.name-node` or
+`hdfs.name-node.<nameservice>` entry that is not `host:port` also falls back. So does a portless
+authority (a nameservice) that neither the Hadoop configuration of the table's `FileIO`
+(`dfs.ha.namenodes.<nameservice>` and a `host:port` `dfs.namenode.rpc-address.<nameservice>.<nn>`
+for each NameNode), which is the one the native writer's declaration is derived from, nor the
+catalog (`hdfs.name-node.<nameservice>`) declares, as do the other HDFS cases listed in the
+section linked below. A portless authority
+that the Hadoop configuration does not name as a nameservice is a plain host on port 8020, as for
+the JVM client. Each is reported as a fall-back reason like any other, rather than surfacing as a
+failed task. See [Object store configuration (HDFS)](iceberg.md#object-store-configuration-hdfs)
+for how the NameNode list is derived.
+
 For an `s3` or `s3a` data location, the gate also inspects both the table FileIO's effective Hadoop
 configuration and `table.io().properties()`. These are separate allowlists because Hadoop S3A
 keys are translated before they reach iceberg-rust, while Iceberg `FileIO` keys are forwarded