Catalog Capture

Independent, community-maintained capture tool (catalog_capture) that writes live market data into Nautilus Trader Rust ParquetDataCatalog layouts for direct backtest use.

Not affiliated with or endorsed by Nautech Systems Pty Ltd. See TRADEMARK.md and NOTICE.

venues → catalog-capture-cli + TOML
      → {catalog}/data/{type}/{instrument_id}/…parquet
      → ParquetDataCatalog / Rust backtest

Status

0.1.x — early open source, research / operator oriented.

• Multi-venue mainnet capture (public market data) has been run unattended for multi-day windows with scheduled segment sealing and Parquet readback.
• APIs and TOML may still evolve; pin a commit for production jobs.
• Best-effort support — see SECURITY.md and ROADMAP.md.

Features

• One product binary — catalog-capture-cli (TOML configs only)
• Rust catalog layout only — rust_canonical_only (Nautilus ParquetDataCatalog)
• Venues — Binance Futures, Deribit, Bybit, OKX, Hyperliquid (venue-* features)
• Subscription lifecycle — optional universe refresh with old subscriptions removed and new subscriptions added
• Order book checkpoints — managed Nautilus books checkpointed hourly and at segment boundaries
• Segment lifecycle — append *.parquet.part, seal on schedule (default 06:00 UTC)
• Mainnet-oriented examples — public data by default (no keys in TOML)

Quick start

Requires Rust 1.98.1 and sibling ../nautilus_trader. Run from the repo root.

# Clone + checkout the Nautilus revision pinned by CI (recommended)
make bootstrap-deps

# Cloud / multi-venue capture: only link venues you need (smaller, faster)
make build-release-capture
# features: venue-binance,venue-deribit,venue-hyperliquid,venue-lighter

./bin/catalog-capture-cli validate \
  --config examples/capture.multi-venue-mainnet.toml

# Long-running mainnet capture (logs under ./logs/)
./scripts/run-mainnet-capture.sh
# default config: examples/capture.multi-venue-mainnet.toml

Short smoke:

CAPTURE_SECONDS=120 ./scripts/run-mainnet-capture.sh

Data: ./data/… (gitignored). Catalog layout: docs/concepts/catalog_layout.md.

Recommended configs

Config	Use
examples/capture.multi-venue-mainnet.toml	Multi-venue instruments, quotes, trades and order-book data
examples/capture.hyperliquid-hip4-btc-daily.toml	Hyperliquid rolling-universe example
examples/capture.deribit-btc-book-summary.toml	Deribit BookSummary only (1s poll)

More: examples/README.md.

Build options

Command	When
make build-release-capture	Preferred for multi-venue mainnet capture
make build-release	All venues (all-venues)
make build-slim FEATURES=venue-deribit	Single-venue debug
make build-release-small	Smaller binary, slower compile
make clean / make clean-all-targets	Free disk (clean-all also clears ../nautilus_trader/target)

Build size notes: docs/how_to/build_size.md.

Cloud / unattended

# Full runbook
# docs/how_to/cloud_capture.md

mkdir -p logs
nohup env NAUTILUS_LOG='stdout=Info;is_colored=false' \
  ./bin/catalog-capture-cli run \
  --config examples/capture.multi-venue-mainnet.toml \
  > logs/capture.log 2>&1 &
echo $! > logs/capture.pid

# Metrics (if enabled in TOML): http://127.0.0.1:9108/metrics
# Stop gracefully: kill -TERM "$(cat logs/capture.pid)"

Documentation

Section	Path
Doc map	docs/index.md
Install	docs/getting_started/installation.md
Cloud capture	docs/how_to/cloud_capture.md
Catalog layout	docs/concepts/catalog_layout.md
Dynamic universe / subscription lifecycle	docs/how_to/hip4_capture.md
Build size	docs/how_to/build_size.md
CLI	docs/reference/cli.md

Credentials

Public capture by default (unset API env vars). Optional key+secret in env only — see credentials.

Development

make bootstrap-deps
make install-tools
pip install pre-commit && pre-commit install
make test && make clippy && make cargo-deny

See CONTRIBUTING.md, SECURITY.md, ROADMAP.md.

License

LGPL-3.0-or-later. Links against Nautilus Trader (LGPL). NOTICE.