feat(storage): Add HDFS support via opendal services-hdfs-native

[mixermt]

Which issue does this PR close?

• Closes Support tables which live on HDFS #2440.
• Part of HDFS support in iceberg-rust #1130.

This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current Storage-trait layout and updated for opendal 0.58, where services-hdfs-native changed behavior in ways that required design changes (details below).

What changes are included in this PR?

Adds an opendal-hdfs-native cargo feature to iceberg-storage-opendal, with OpenDalStorageFactory::HdfsNative / OpenDalStorage::HdfsNative variants and hdfs:// routing in OpenDalResolvingStorage. The backend uses OpenDAL's services-hdfs-native (pure-Rust HDFS RPC via hdfs-native — no JNI/libhdfs). The feature is experimental and not part of opendal-all.

NameNode resolution (differs from #2441, forced by opendal's hdfs-native service, where name_node is mandatory and, since 0.56 (apache/opendal#7248), its comma-split list is the HA mechanism):

1. An authority with a port (hdfs://host:port/path) is used as is; a configured hdfs.name-node never overrides it. Userinfo and port 0 in an authority are rejected with DataInvalid (every error the HDFS backend raises for a path or NameNode value masks everything between the scheme and the last @, e.g. hdfs://***@nn:8020/x).
2. A portless authority (hdfs://ns1/path) must be a declared nameservice: hdfs.name-node.<nameservice> takes a single endpoint or a comma-separated list for HA failover and is sugar for Hadoop's dfs.ha.namenodes.<nameservice> / dfs.namenode.rpc-address.<nameservice>.<id>, which are honored as well when passed through hadoop.*. Unlike Hadoop, an undeclared one is an error naming the key to set, rather than being resolved from $HADOOP_CONF_DIR or dialed on port 8020: opendal always builds the client as hdfs://nameservice from an explicit list, so hdfs-native never sees the path's authority (noted in the README; lifting it needs opendal to accept a logical nameservice as name_node). Every NameNode spelling, configured or from a path, is normalized to hdfs://host:port so all sources share cache keys; other schemes, userinfo, paths, port 0 and unbracketed IPv6 are rejected, and hdfs-native has no default port, so a portless NameNode can only be a logical name.
3. An authority-less path (hdfs:///path) resolves through plain hdfs.name-node (new HDFS_NAME_NODE / HDFS_HADOOP_CONF_PREFIX constants in iceberg::io), else fs.defaultFS: set from PyIceberg's hdfs.host/hdfs.port (new HDFS_HOST / HDFS_PORT constants, port defaults to 8020, validated at config time; as in PyIceberg they apply only without a path authority) or from hadoop.fs.defaultFS. Anything unresolvable is rejected at resolution time with an error naming the property to set.

hadoop.-prefixed properties pass through to the HDFS client config with the prefix stripped, overriding $HADOOP_CONF_DIR values (mirroring the hadoop. catalog-property convention of the Java integrations); hdfs-native still loads core-site.xml/hdfs-site.xml from $HADOOP_CONF_DIR/$HADOOP_HOME for everything else, and Kerberos works via libgssapi_krb5 (runtime dlopen). PyIceberg's hdfs.user/hdfs.kerberos_ticket have no equivalent in opendal's config and log a warning.

Operators are cached per effective NameNode inside an opaque HdfsNativeStorage (built only by the factories) and built on a blocking thread, since the build reads the Hadoop XML synchronously. Each cached client is bound to the tokio runtime that built it; the cache notices when that runtime is gone and rebuilds the operator, because hdfs-native panics when spawning onto a dead runtime. A poisoned cache lock is recovered rather than failing every later call. Without a tokio runtime, create_operator returns FeatureUnsupported instead of panicking (every backend's I/O needs one for opendal's timeout layer).

Relative paths are returned opendal-style without a leading / — opendal::Deleter::delete (used by delete_stream) rejects leading slashes, which an integration test caught.

Test infrastructure: single-node HDFS docker fixture (apache/hadoop:3.5.0, host networking — required because hdfs-native dials DataNodes by their registered IP, unroutable on a bridge). The DataNode healthcheck gates on NameNode registration so --wait means writable. The fixture is opt-in (profiles: [hdfs], started with COMPOSE_PROFILES=hdfs make docker-up) and the tests self-skip unless ICEBERG_TEST_HDFS_ENDPOINT is set, like the HF tests. A dedicated Tests (hdfs) CI job starts only the two HDFS containers and sets the endpoint, so the default suite no longer pulls the Hadoop image.

Are these changes tested?

• 37 unit tests across config parsing and normalization (per-entry trimming, port validation, hdfs.host/hdfs.port into fs.defaultFS, rejected and ignored keys), path parsing (authority/port/IPv6/authority-less/non-hierarchical), NameNode precedence and nameservice declarations, spelling normalization, operator caching (first-insert-wins for racing builds, rebuild after runtime shutdown, sentinel cleanup, poisoned-lock recovery), userinfo masking in errors, serde round-trip, batch keys, relativize, and scheme resolution.
• 14 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty, +two NameNode spellings), authority-less paths through hdfs.name-node and hdfs.host/hdfs.port, resolving storage over hdfs://, and the HA flow (logical authority in the path + hdfs.name-node property), plus a fixture-free test that reuses a FileIO from a second runtime after the first is dropped (it panics on the old cache). The docker-backed suite runs in the Tests (hdfs) CI job and self-skips elsewhere unless ICEBERG_TEST_HDFS_ENDPOINT is set.
• make check (incl. cargo-deny licenses), full-workspace unit/doc tests, MSRV, the public API check, and the s3/gcs/hms/resolving integration suites pass locally; public-api.txt regenerated for both crates.

AI Disclosure

Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the locked opendal 0.58.1 / hdfs-native 0.14.6 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what hdfs://<nameservice> paths can do without the property), and the Windows --all-features build of hdfs-native, which I could only verify via CI.

🤖 Generated with Claude Code

[blackmwk]

I found two correctness issues that need to be addressed before merge: malformed hdfs: URLs can panic, and bulk deletion can conflate NameNodes running on different ports. Details and suggested regression coverage are inline. Please also resolve the two existing review threads.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Rust maintainer. After you've addressed the points above and pushed an update, an Apache Iceberg Rust maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Rust handles maintainer review: CONTRIBUTING.md.

[blackmwk]

Thanks @mixermt for this pr. I think the variant should be named to Hdfs-native rather than hdfs. Hdfs should be reserved for non native hdfs support.

[mixermt]

@blackmwk
I've addressed all the recent feedback.
Could you please give this one last review? 🙏

[comphead]

Reviewed the backend against the locked opendal-service-hdfs-native 0.58.1 and hdfs-native 0.14.5 sources, and against the existing hf/oss/azdls backends in this crate.

The overall shape fits the crate well: *_config_parse / *_batch_key / (Operator, &str) mirror hf, and the dedicated batch_key arm is genuinely needed, since host-only keying would merge NameNodes that differ only by port. The operator cache is justified too, given each client re-reads the Hadoop XML and opens RPC connections.

Leaving 5 inline comments: 2 blockers (test gating versus what the description claims, and blocking I/O under the cache write lock) and 3 majors (the cache leaking into the public API, duplicated NameNode-precedence logic, and empty-name-node handling).

I also have a handful of minor simplification notes and two questions (operator caching pins the Client to the tokio runtime that first built it, since hdfs-native captures Handle::try_current() eagerly; and the hadoop. prefix scope). Happy to add those if useful, but they seemed like noise next to the above.

[mixermt]

Thanks @comphead for the thorough review — all five items are addressed in d17ae38 and answered inline. The security_audit failure was RUSTSEC-2026-0285 in rustls 0.23.41, which is main's current pin (AWS SDK chain, unrelated to hdfs-native; tracked in #3222). Bumped to 0.23.45 in b8cc860, the same fix #3165 applied. The minor simplification notes you held back are welcome — please send them.

"Bumped to 0.23.45 in b8cc860, the same fix #3165 applied" if we keep it without it CI will stay red, tell me if we want to revert this under this PR

[mixermt]

@comphead @blackmwk any other comments?
I really want to push this forward

I'm testing this with datafusion-comet, so far no issues in this code.

[mixermt]

Thanks @laskoviymishka. All 10 threads are addressed in 22db8e6..9c62f4c (merge of main plus 9 commits), each with an inline reply. Behavior changes: the operator cache rebuilds entries whose tokio runtime is gone instead of hdfs-native panicking (regression test included); NameNode resolution follows Hadoop, so a host:port authority is never overridden and portless names must resolve through hdfs.name-node; the HDFS variant wraps one opaque HdfsNativeStorage; operator builds run under spawn_blocking; the HDFS integration tests moved to a dedicated CI job; ignored or malformed properties now warn or error instead of being dropped. Description updated; full fixture suite green locally.

[laskoviymishka]

Thanks for the quick turnaround — a lot of the round-1 list landed cleanly. The hdfs: URL panic is gone (nice coverage on hdfs:x/hdfs:/x/hdfs:), batch deletes key on the effective NameNode so the port conflation is fixed, relativize_path and create_operator now share hdfs_native_effective_name_node, the HA list is trimmed per entry, the variant is HdfsNative, and the cache no longer leaks into the public API. On my IPv6 flag from last round — you were right and I was wrong: Url::host_str() does keep the brackets, and the hdfs://[::1]:8020 tests pin it.

Two things still open, one of them new. My executor-blocking point got addressed by moving the XML read into spawn_blocking — but that trades it for a panic when there's no tokio runtime, and it leaves the sentinel: None / "built outside any runtime" branch dead in production. That one I'd like resolved before merge. The portless-authority / logical-nameservice resolution is the carried-over half of my hdfs.name-node-vs-PyIceberg note, now sharper: with hdfs.name-node set every portless authority routes to the one configured cluster (risky for delete_*), and a nameservice living only in $HADOOP_CONF_DIR gets rejected — both hinge on how opendal 0.58.1 treats a bare logical name_node, which none of us could confirm. Details inline.

Everything else is a tightening nit (the hdfs_native_has_port validation + cache-key normalization). Sort the spawn_blocking panic and settle the portless-authority question and I think this is basically there.

[mixermt]

Thanks @laskoviymishka. The three threads from this round are addressed in 9c62f4c..ce78ee3, each with a reply inline: a missing tokio runtime is now a FeatureUnsupported error instead of a panic (and the unreachable branch is gone); every NameNode spelling is normalized to hdfs://host:port with the malformed shapes rejected; and nameservices can be declared with hdfs.name-node.<nameservice> (Hadoop's own keys underneath), with plain hdfs.name-node as the documented single-cluster default. Description updated; full checks and the fixture suite green locally.

[laskoviymishka]

Thanks for the turnaround. Everything from round 1 is in now: IPv6 round-trips, shared hdfs_native_effective_name_node, HA list trimming, XML reads on spawn_blocking, and the @blackmwk / @comphead blockers are resolved.

One correctness issue left before I’d approve: with plain hdfs.name-node set, an unknown or mistyped authority falls back to that cluster. In a multi-cluster setup that can send writes/deletes to the wrong NameNode. hdfs_native_parse_path also allows userinfo and port 0, and hdfs://nn:0/... hits the same fallback. I think unknown authorities should error, not guess.

Non-blocking follow-ups:

• nameservices / fs.defaultFS from $HADOOP_CONF_DIR XML are not visible to the resolver today. It already fails closed, so a doc note on HDFS_NAME_NODE is enough for now.
• validate hdfs.host / hdfs.port during parsing.

Fix the fallback and the parse_path gap and I’m happy to approve.

[mixermt]

Thanks @laskoviymishka. Both blocking items and the non-blocking ones from this round are in ce78ee3..b962408: logical nameservices must be declared (plain hdfs.name-node serves authority-less paths only), userinfo and port 0 are rejected in path authorities, the composed hdfs.host/hdfs.port is validated, and the HDFS_NAME_NODE doc covers nameservices that live only in $HADOOP_CONF_DIR. I also tightened the test suite after a review pass: three redundant tests dropped, the warnings and the password redaction asserted directly, and fixture coverage added for authority-less paths. Description updated; full checks and the fixture suite green locally.

[laskoviymishka]

This is in good shape, most of items from iteration landed. IPv6 round-trips (url.host_str keeping the brackets, with the [::1]:8020 test), relativize_path/create_operator/batch_key all go through hdfs_native_effective_name_node so they agree on authority-less paths, the HA list is trimmed per entry, and the XML read is in spawn_blocking. The shared set @blackmwk and @comphead had open — the hdfs: panic, port in the batch keys, the HdfsNative rename, the public-API cache leak — is in too. Thanks for working through all of it.

Good to land from me. I've left a few followup nits inline, none of them blocking:

• The portless-authority path: hdfs://nameservice1/... (the common HA shape, where the nameservice is defined only in hdfs-site.xml under HADOOP_CONF_DIR) errors in the resolver instead of being handed to hdfs-native, which already reads the conf dir and resolves it. Worth passing undeclared portless authorities straight through so those clusters can read their own tables — or, if the strict behavior is deliberate for now, dropping the "as in Hadoop" doc line, since it diverges there.
• The parse-error strings echo the full path (so a user:pw@… path leaks the password) despite the comment promising otherwise.
• A poisoned cache lock turns into permanent HDFS failure — recover the guard instead. Separate nuance from the lock thread I confirmed was fine last round, not a reversal.
• The runtime-rebuild test can go green without actually dialing from the second runtime.

I'll keep the PR soaking for at least a day before I merge, so there's room to fold any of these in now if you'd rather not leave them as follow-ons.

[laskoviymishka]

Ah, there is a blocking change_request from @blackmwk , so i'll wait for him to stamp before merging at least.