new feature: in-memory distributed buffer as OpenDAL layer

[Kinrany]

Feature Description

A layer that allows for multiple processes connected to the same underlying OpenDAL store to pick a single writer and route all writes to that process, transparently.

Problem and Solution

I see it as a common issue that writes to object storage have to be coordinated.

Another common issue is that writes to object storage take significant time. A common solution to that is buffering in memory and replicating to multiple availability zones.

It feels (naively so perhaps) possible for an OpenDAL layer to transparently connect to other nodes in a cluster, relay writes to the leader, wait for the cluster to confirm.

Additional Context

I believe Cloudflare Durable Objects use a similar design.

Originally I started thinking about this in the context of SlateDB, which I believe is considering using OpenDAL internally.

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.