timeseries-table-format

Stop managing Parquet files. Start managing time-series tables.

A Rust-native table format with coverage tracking, overlap-safe ingestion,
and DataFusion SQL for local time-series data.

Documentation | Python | Rust

Early MVP: APIs and on-disk layouts may change before v1.0.

Built for time-series data

timeseries-table-format turns Arrow data into managed, append-only tables. It tracks which chronological windows exist for each entity, rejects overlapping appends, and exposes the result through DataFusion SQL.

Use a Timestamp, Int64, or UInt64 column as the ordered index. The table format handles metadata, transactions, coverage, and segment discovery. Index granularity defines logical intervals, and each complete entity identity may have at most one row per interval, both within and across appends.

Need	Built-in support
Know whether a time range is covered	Coverage indexes and gap queries
Enforce one row per identity and interval	Validation within and across appends
Query many Parquet segments	DataFusion SQL with segment pruning
Inspect or remove interrupted-write artifacts	Retention-aware vacuum with dry-run and apply modes
Run without Spark or a database server	Rust core, Python package, and CLI

It is a good fit for market data, sensor pipelines, backtesting systems, and other incremental time-series workloads that live on a local filesystem.

A taste of the Python API

pip install timeseries-table-format

import pyarrow as pa
import pyarrow.parquet as pq
import timeseries_table_format as ttf

table = ttf.TimeSeriesTable.create(
    table_root="prices",
    index_column="ts",
    index_type="timestamp",
    index_granularity="1h",
    entity_columns=["symbol"],
)
parquet_file = pq.ParquetFile("prices.parquet")
report = table.append(
    pa.RecordBatchReader.from_batches(
        parquet_file.schema_arrow,
        parquet_file.iter_batches(),
    )
)
version = report.committed_version

session = ttf.Session()
session.register_tstable("prices", "prices")
result = session.sql("SELECT * FROM prices ORDER BY ts")

The Python documentation walks through installation, ingestion, and queries.

Performance

In the repository's 73 million row NYC taxi benchmark, bulk ingestion was 7.7x faster than ClickHouse and 27x faster than PostgreSQL on the tested hardware and configuration.

See the benchmark methodology and results for the workloads, environment, and full comparison.

Interfaces

Interface	Where to start
Python	Python documentation and PyPI
Rust	Engine guide and docs.rs
CLI	CLI reference
DataFusion	Integration guide

Current scope

The current release focuses on local, append-only tables. It does not yet support object storage, compaction, schema evolution, row updates, merges, or time-travel queries.

For design details, read How I built this or view the architecture diagram.

Contributing

Contributions and bug reports are welcome. See the repository's existing issues and development documentation before starting a larger change.

License

MIT. See LICENSE.