Support Iceberg tables on HDFS in the native scan #5894

What is the problem the feature request solves?

Comet's native Iceberg scan cannot read a table whose data lives on HDFS. hdfs:// is not in the scheme allowlist that mirrors storage_factory_for, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.

This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via fs.comet.libhdfs.schemes). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.

Describe the potential solution

Route hdfs:// to iceberg-rust's hdfs-native OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.

Comet-side work:

• Add an hdfs arm to storage_factory_for and admit the scheme in the JVM gates (CometScanRule.icebergReadableSchemes, and CometIcebergNativeWrite.SupportedStorageSchemes for the write path).
• Forward the hdfs./hadoop. property prefixes to the native FileIO.
• Resolve the NameNode endpoints for HA clusters. opendal's HdfsNativeBuilder never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated hdfs.name-node value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real host:port. An HA location reads hdfs://<nameservice>/..., and a nameservice is not a routable host, so the endpoints have to be derived from dfs.ha.namenodes.<ns> / dfs.namenode.rpc-address.<ns>.<nn> in the session Hadoop configuration.
• Decline a scan whose data/delete files span more than one HDFS authority: one hdfs.name-node per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.

Additional context

Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same $HADOOP_CONF_DIR XML but hold separate connections and separate Kerberos state.

Blocked on apache/iceberg-rust#3111.