Note
Made with ❤️ by @clflushopt, @alamb and @kevinjqliu
Originally located at clflushopt/tpchgen-rs, the project is now maintained at
datafusion-contrib/tpcgen-rs
Modern, blazing fast and easy to use TPC-H and TPC-DS benchmark data generators.
Democratize the comparison of analytical systems with tools for easily and efficiently generating benchmark data for common benchmarks in modern formats such as Apache Parquet, Apache Arrow, and CSV.
- Blazing Speed 🚀
- Easy to Use: CLI or embeddable libraries
- Obsessively Tested 📋
- Resource Efficient: multi-core and constant memory use 🧠
tpcgen-cli is distributed as a Python package. Run it with uvx:
uvx tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpch
uvx tpcgen-cli tpcds parquet -s 1 --output-dir /tmp/tpcdsOr install it with pip:
python -m pip install tpcgen-cli
tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpchSee the tpcgen-cli README for more examples.
| Project | Description |
|---|---|
tpcgen-cli |
Command line tool to generate TPC-H and TPC-DS data in multiple formats |
tpchgen-cli |
Command line tool to generate only TPC-H data in multiple formats (kept for backwards compatibility) |
tpchgen |
Rust library to generate TPC-H data (zero dependencies) |
tpchgen-arrow |
Rust library to generate TPC-H data in Apache Arrow format |
tpcdsgen |
Rust library to generate TPC-DS data (zero dependencies) |
tpcdsgen-arrow |
Rust library to generate TPC-DS data in Apache Arrow format |
Time to generate TPC-H Scale Factor 100 data in Parquet format. See the performance spreadsheet for details and more comparisons.
We go to great lengths to ensure our data generators produce the same exact bytes as the reference implementations, both for single and multi-part output. Our multi-level strategy for ensuring correctness is detailed in TESTING.md and includes comparing the output of our generators to the original reference implementations as part of CI.
Pull requests are welcome. For major changes, please open an issue first for discussion. See our contributors guide for more details.
Please see architecture guide for details on how the code is structured.
The project is licensed under the APACHE 2.0 license.