fsspec-rs

fsspec-compatible filesystem backends with Rust acceleration.

Overview

fsspec-rs provides Python filesystem classes that inherit from fsspec.spec.AbstractFileSystem while delegating core operations to Rust. They can be used anywhere fsspec filesystems are accepted, including pandas, dask, xarray, and direct fsspec.open() calls.

The package currently includes:

Backend	Protocol	Python class	Replaces
Local filesystem	file-rs, local-rs	fsspec_rs.LocalFileSystem	fsspec.implementations.local.LocalFileSystem
Amazon S3	s3-rs	fsspec_rs.S3FileSystem	s3fs.S3FileSystem

Install

pip install fsspec-rs

Quick start

import fsspec
from fsspec_rs import LocalFileSystem
from fsspec_rs import S3FileSystem

fs = LocalFileSystem()
fs.pipe_file("/tmp/example.txt", b"hello")
print(fs.cat_file("/tmp/example.txt"))

s3 = S3FileSystem(bucket="my-bucket", region="us-east-1")
s3.pipe_file("path/to/output.txt", b"hello")

with fsspec.open("file-rs:///tmp/example.txt", "rb") as f:
    print(f.read())

Documentation

• Filesystems: local and S3 usage, configuration, caching, and path conventions.
• API reference: Python API generated with yardang/Sphinx.
• Rust traits: Rust FileSystem, experimental AsyncFileSystem, FsFile, and cache traits.

Build docs locally with:

yardang build