Repository navigation

feat(storage): Add HDFS support via opendal services-hdfs-native - #3111

Open

mixermt wants to merge 68 commits into

apache:mainfrom

mixermt:feat-storage-hdfs-native

Open

feat(storage): Add HDFS support via opendal services-hdfs-native#3111
mixermt wants to merge 68 commits into
apache:mainfrom
mixermt:feat-storage-hdfs-native

Commits on Aug 29, 2026
- feat(storage): Add HDFS support via opendal services-hdfs-native
  
  committed
  6679a19
Commits on Aug 31, 2026
Commits on Sep 1, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  4141f4e
Commits on Sep 3, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  2578c03
Commits on Sep 5, 2026
- Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native
  mixermt
  committed
  13911c5
- refactor(storage): rename hdfs module to hdfs_native, drop unused HdfsConfig
  
  mixermt
  and
  claude
  committed
  a4178fe
Commits on Sep 8, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  a9a6786
Commits on Sep 12, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  806bd89
Commits on Sep 13, 2026
- ci: retrigger after setup-python manifest timeout on macos runner
  
  mixermt
  and
  claude
  committed
  9d7d2d8
Commits on Sep 16, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  c561cfc
Commits on Sep 17, 2026
- fix(storage): address review — cache newtype, build outside lock, single NameNode rule, env-gated HDFS tests
  
  mixermt
  and
  claude
  committed
  d17ae38
- chore(deps): bump rustls to 0.23.45 (RUSTSEC-2026-0285)
  
  mixermt
  and
  claude
  committed
  b8cc860
Commits on Sep 18, 2026
- Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native
  mixermt
  committed
  587688c
Commits on Sep 22, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  72305b6
Commits on Sep 25, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  f8b6803
Commits on Sep 28, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  ffee268
Commits on Oct 1, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  eeca0d1
Commits on Oct 3, 2026
- Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native
  
  mixermt
  and
  claude
  committed
  2068e83
- test(storage): pin IPv6 authority handling in HDFS path parsing
  
  mixermt
  and
  claude
  committed
  297afd6
- fix(storage): resolve the HDFS NameNode in relativize_path too
  
  mixermt
  and
  claude
  committed
  c018cd1
- docs(storage): state the cost of the synchronous HDFS operator build
  
  mixermt
  and
  claude
  committed
  556a0f2
- docs(storage): describe the HDFS operator cache lifecycle
  
  mixermt
  and
  claude
  committed
  d3ef172
- fix(storage): normalize each entry of the hdfs.name-node list
  
  mixermt
  and
  claude
  committed
  62ea36d
- feat(storage): accept PyIceberg's hdfs.host/hdfs.port for authority-less HDFS paths
  
  mixermt
  and
  claude
  committed
  3ab457a
- fix(storage): require a port in every hdfs.name-node entry
  
  mixermt
  and
  claude
  committed
  74bb9b1
- docs(storage): state the runtime binding of cached HDFS operators
  
  mixermt
  and
  claude
  committed
  0928265
- test(storage): cover two-NameNode delete_stream batches and first-insert-wins caching
  
  mixermt
  and
  claude
  committed
  089731f
- test(utils): name the HDFS port constant like its siblings
  
  mixermt
  and
  claude
  committed
  22db8e6
Commits on Oct 6, 2026
- Merge remote-tracking branch 'apache/main' into feat-storage-hdfs-native
  mixermt
  committed
  2f6a57d
- fix(storage): rebuild cached HDFS operators once their runtime is gone
  
  mixermt
  and
  claude
  committed
  629cfc8
- fix(storage): resolve HDFS NameNodes like Hadoop does
  
  mixermt
  and
  claude
  committed
  e38bab2
- refactor(storage): keep the HDFS operator cache out of the public API
  
  mixermt
  and
  claude
  committed
  cfe4ee7
- perf(storage): build HDFS operators on a blocking thread
  
  mixermt
  and
  claude
  committed
  6603348
- ci: run the HDFS integration tests in their own job
  
  mixermt
  and
  claude
  committed
  ed23ab6
- fix(storage): stop dropping HDFS properties silently
  
  mixermt
  and
  claude
  committed
  cc9dd91
- test(storage): derive the second NameNode spelling from the endpoint host
  
  mixermt
  and
  claude
  committed
  f83db4e
- chore: record tracing as a dependency of iceberg-storage-opendal in Cargo.lock
  
  mixermt
  and
  claude
  committed
  9c62f4c
- fix(storage): report a missing tokio runtime instead of panicking in HDFS create_operator
  
  mixermt
  and
  claude
  committed
  a0d2194
- fix(storage): normalize every HDFS NameNode spelling to hdfs://host:port
  
  mixermt
  and
  claude
  committed
  ca92240
- feat(storage): declare HDFS nameservices with hdfs.name-node.<nameservice>
  
  mixermt
  and
  claude
  committed
  ce78ee3
- fix(storage): require a declaration for every logical HDFS nameservice
  
  mixermt
  and
  claude
  committed
  4a9cc25
- fix(storage): reject userinfo and port 0 in HDFS path authorities
  
  mixermt
  and
  claude
  committed
  5870769
- fix(storage): validate the NameNode composed from hdfs.host and hdfs.port
  
  mixermt
  and
  claude
  committed
  b72d795
- test(storage): use an authority-less path in the cross-runtime HDFS test
  
  mixermt
  and
  claude
  committed
  7f2dc7f
- test(storage): tighten the HDFS tests after review
  
  mixermt
  and
  claude
  committed
  b962408
Commits on Oct 7, 2026
- Merge branch 'main' into feat-storage-hdfs-native
  mixermt
  authored
  ef451b6
Commits on Oct 8, 2026
- fix(storage): mask userinfo in HDFS errors that echo a path or NameNode
  
  mixermt
  and
  claude
  committed
  1276d2d
- fix(storage): recover a poisoned HDFS operator cache lock
  
  mixermt
  and
  claude
  committed
  1fd3492
- test(storage): flush the fake NameNode before sampling dials
  
  mixermt
  and
  claude
  committed
  894a8ed
- docs(storage): state where portless HDFS authorities differ from Hadoop
  
  mixermt
  and
  claude
  committed
  30d74a8
Commits on Oct 9, 2026
- refactor(storage): rename the HDFS config module to hdfs_native
  
  mixermt
  and
  claude
  committed
  86d81b4
- test(storage): move the HDFS tests out of lib.rs into hdfs_native
  
  mixermt
  and
  claude
  committed
  a347b84
- refactor(storage): read HDFS properties through #[derive(Properties)]
  
  mixermt
  and
  claude
  committed
  39b3cc5
- ci: run the HDFS tests in the default suite like the other fixtures
  
  mixermt
  and
  claude
  committed
  4ab34ee
- fix(storage): build HDFS clients in a crate-owned runtime
  
  mixermt
  and
  claude
  committed
  0bf5d5a
- refactor(storage): type HDFS path authorities and validate declarations at parse
  
  mixermt
  and
  claude
  committed
  f092ea2
- refactor(storage): drop index slicing and combinator chains in the HDFS backend
  
  mixermt
  and
  claude
  committed
  26aba03
- fix(storage): attach the hdfs.port parse error as the source
  
  mixermt
  and
  claude
  committed
  80f2149
- fix(storage): share one HDFS client across host name spellings
  
  mixermt
  and
  claude
  committed
  323c94b
- docs(storage): note the upstream gap behind the HDFS client runtime
  
  mixermt
  and
  claude
  committed
  63dbed2
- test(storage): make the HDFS HA test fail over for real
  
  mixermt
  and
  claude
  committed
  7ac1f45
- Merge apache/main into feat-storage-hdfs-native
  
  mixermt
  and
  claude
  committed
  fdd9562
- chore: retrigger CI
  
  mixermt
  and
  claude
  committed
  8aba12b
Commits on Oct 10, 2026
- chore: retrigger CI
  
  mixermt
  and
  claude
  committed
  650e793
- docs(storage): document how to enable and configure the HDFS backend
  
  mixermt
  and
  claude
  committed
  0145ca6

Commits

Commits on Aug 29, 2026

Commits on Aug 31, 2026

Commits on Sep 1, 2026

Commits on Sep 3, 2026

Commits on Sep 5, 2026

Commits on Sep 8, 2026

Commits on Sep 12, 2026

Commits on Sep 13, 2026

Commits on Sep 16, 2026

Commits on Sep 17, 2026

Commits on Sep 18, 2026

Commits on Sep 22, 2026

Commits on Sep 25, 2026

Commits on Sep 28, 2026

Commits on Oct 1, 2026

Commits on Oct 3, 2026

Commits on Oct 6, 2026

Commits on Oct 7, 2026

Commits on Oct 8, 2026

Commits on Oct 9, 2026

Commits on Oct 10, 2026