Support HDFS as a storage backend

[hfutatzhanghb]

Background

HDFS is widely used in enterprise data platforms, and many teams want to use Lance as their columnar format on top of HDFS. Full HDFS support is currently blocked by two upstream dependencies.

Related PR

• lance-format/lance#5472 — HDFS support for Lance

Blocking dependencies

This PR is currently blocked by two upstream issues:

1. hdrs — missing HDFS copy support

The hdrs crate (used by Lance's HDFS object store provider) does not implement the copy operation. This causes failures when Lance needs to copy objects on HDFS, for example during optimize_indice on inverted indexes.

• Upstream PR: Xuanwo/hdrs#143

2. opendal — rename_with_if_not_exist

HdfsSafeCommitHandler need this.

• Upstream PR: apache/opendal#7815

Call for help

We'd appreciate the community's help in reviewing and merging the two upstream PRs so that HDFS support can land in Lance. If there are alternative approaches or workarounds, please share them in this issue or the PR discussion.

[hfutatzhanghb]

@Xuanwo @jiaoew1991 @BubbleCal Hi, I would appreciate if you can help me push this feature! Thanks very much.

[wjones127]

Hello! I think the direction we want to go is bringing https://crates.io/crates/lance-hdfs-provider into this monorepo, and then shipping that in pylance. (We have permission from the author.) Does that sound like a good plan to you?

[hfutatzhanghb]

Hello! I think the direction we want to go is bringing https://crates.io/crates/lance-hdfs-provider into this monorepo, and then shipping that in pylance. (We have permission from the author.) Does that sound like a good plan to you?

Hi, @wjones127 . Sorry, I don't get it. Do you mean merge https://crates.io/crates/lance-hdfs-provider into my branch and finally merge into lance's main branch? If that, i am willing to do it. Hope to receive from you~ Thanks!

[wjones127]

I mean that we'd like to:

1. Move that crate into our workspace. It will still be a separate crate, but it will be published at the same time as our other crates.
2. Make lance-hdfs-provider an optional but default dependency of the pylance package. If we make it default, it will ship as supported in the wheels.

Is that clearer?

Part of what I am asking is: can we use lance-hdfs-provider as-is, without any code changes? Or are there some code changes required?

[hfutatzhanghb]

@wjones127 Thanks. Got it.

1. lance-hdfs-provider seems not have a safe commithandler which is essential for transaction commit. (feat: support HDFS object store #5472 has implemented it)
2. BTW. I wonder why we don't adopt the way like feat: support HDFS object store #5472 ? I think that way is better than provide a crate. Just like oss、gcp、tos and other storages.

What' your opinions?

[wjones127]

1. lance-hdfs-provider seems not have a safe commithandler which is essential for transaction commit. (feat: support HDFS object store #5472 has implemented it)

Can we add that to the crate?

2. BTW. I wonder why we don't adopt the way like feat: support HDFS object store #5472 ? I think that way is better than provide a crate. Just like oss、gcp、tos and other storages.

I don't really love the current situation. As the trend continues, we'll have code specific to every cloud vendor in existence baked into our repo. I'd much prefer it if we could get to a point where each vendor had a separate crate. And I especially don't love that we enable all of them by default:

lance/rust/lance/Cargo.toml

Line 141 in 104ef4f

default = ["aws", "azure", "gcp", "oss", "huggingface", "tencent", "tos", "goosefs", "geo"]

I do get there's something easy about making them a feature. It's on some level easier for users to turn on and off. Plus there's the Python issue: we build those binaries so we have to choose what gets built. That doesn't make it easy.

[hfutatzhanghb]

1. Yes, we can add it into that crate. 2. Great idea. Totally agree. And i think maybe we can refactor current repo to move all object storages to independent crates in the future.

…

---- Replied Message ---- | From | Will ***@***.***> | | Date | 07/07/2026 23:26 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [lance-format/lance] Support HDFS as a storage backend (Issue #7584) | wjones127 left a comment (lance-format/lance#7584) lance-hdfs-provider seems not have a safe commithandler which is essential for transaction commit. (feat: support HDFS object store #5472 has implemented it) Can we add that to the crate? 2. BTW. I wonder why we don't adopt the way like feat: support HDFS object store #5472 ? I think that way is better than provide a crate. Just like oss、gcp、tos and other storages. I don't really love the current situation. As the trend continues, we'll have code specific to every cloud vendor in existence baked into our repo. I'd much prefer it if we could get to a point where each vendor had a separate crate. And I especially don't love that we enable all of them by default: https://github.com/lance-format/lance/blob/104ef4f7b612c8aaee2417fe9291caea3a694931/rust/lance/Cargo.toml#L141 I do get there's something easy about making them a feature. It's on some level easier for users to turn on and off. Plus there's the Python issue: we build those binaries so we have to choose what gets built. That doesn't make it easy. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[hfutatzhanghb]

Hi. @wjones127 I will keep on pushing this feature.

1. For Safe CommitHanlder: We need OpenDAL release a new version which contains feat(core): add conditional rename option apache/opendal#7815 (have been merged) and we use that new version in lance-hdfs-provider.
2. For some optimize_indices operations: We need Hdrs release a version which contains feat: Add copy file support for client Xuanwo/hdrs#143 (ready for merging).

When ready we can continue to implement lance-hdfs-provider.

[hfutatzhanghb]

fMeow/lance-hdfs-provider#20 (comment)

I have uploaded the new patch in above link. But could we build a standard pipeline for that repo first? Thanks!

[hfutatzhanghb]

Hi, @wjones127 @Xuanwo . Since the project fMeow/lance-hdfs-provider#20 push forward slowly. Could we make hdfs provider into lance main repo just this time ?

[wjones127]

Yes, I'm open to adding the crate to this repo. But keep it a separate crate as part of the workspace, not a feature in the main binary.

[hfutatzhanghb]

Yes, I'm open to adding the crate to this repo. But keep it a separate crate as part of the workspace, not a feature in the main binary.

OK, Got it. Sir, could you give me some permissions on project fMeow/lance-hdfs-provider#20 ? I have no write permissions which block me to push hdfs feature forward. Thanks.