AGENTS.md

AGENTS.md – FlowDB

Build & Test Commands

• Build (debug): cargo build
• Build all targets: cargo build --tests
• Run all tests: cargo test
• Run only library tests: cargo test --lib
• Run the fuzz suite: cargo test --test fuzz_tests
• Run benches: cargo bench

Lock Ordering

To avoid deadlocks, all background tasks must acquire locks in this order:

cache shard locks → manifest → index → worker (WritePipeline)

• Flush path: manifest → index (releases between)
• GC path: index → manifest (separate scopes)
• Compaction path: manifest → index (separate scopes)
• Never hold two locks from this set simultaneously unless the order above is strictly followed.

Architecture (v0.3.0+)

FlowDB is a pure embedded storage engine with a fully synchronous API. There is no async runtime dependency — no Tokio, no async-std, no smol. Background maintenance (flush, compaction, GC, WAL sync) runs on a plain std::thread::spawn thread managed by [MaintenanceHandle].

All public Engine methods are plain fn — no async, no .await.

Coverage

This project enforces ≥ 90% line/region/function coverage across the entire crate (including the CLI binary).

Run coverage locally:

cargo llvm-cov                       # text summary
cargo llvm-cov --html --open         # interactive HTML report
cargo llvm-cov --summary-only        # just the totals

How we hit 90%+ across the whole crate:

1. The fn main() entry point in src/bin/flowdb-stress.rs is tagged with #[cfg_attr(coverage_nightly, coverage(off))] because it is pure process bootstrap and is exercised via integration tests, not unit tests. The file enables the unstable coverage_attribute feature only under cfg(coverage_nightly), which cargo-llvm-cov sets automatically on nightly toolchains.

2. All logic in the binary (CLI parsing, all bench_* / format_* / print_* helpers in the stress tool) is factored into free functions with unit tests in the same file's mod tests.

3. Cargo.toml declares the lints.rust.unexpected_cfgs table so the cfg(coverage) / cfg(coverage_nightly) cfgs don't trigger warnings.

4. Engine background maintenance (spawn_background_maintenance, auto_background = true) is covered by engine::tests::test_engine_auto_background_starts_maintenance and engine::tests::test_spawn_background_maintenance_explicit.

If you add new code, please add corresponding tests and run cargo llvm-cov --summary-only before committing — the TOTAL line coverage must stay at or above 95%.

Project Layout

src/
  lib.rs              – public API surface (Config, Engine, Record, Query, ...)
  engine.rs           – Engine + ScanIterator + MaintenanceHandle (the core)
  memtable.rs         – in-memory write buffer (Vec-based active, BTreeMap frozen)
  wal.rs              – write-ahead log (FxHash checksums, buffered writes)
  sstable.rs          – on-disk sorted-string table reader/writer
  block_meta_index.rs – fine-grained block-level index
  bloom.rs            – bloom filter for SST point queries
  cache.rs            – block cache (LRU, 64 shards)
  compaction.rs       – size-tiered compaction (streaming heap merge)
  gc.rs               – expired-SST garbage collection
  manifest.rs         – append-only manifest log (JSON)
  record.rs           – Record / InternalRecord / Query / Config / ScanRange types
  write_worker.rs     – single-writer worker driving WAL + memtable
  stats.rs            – engine stats + Prometheus exporter
  error.rs            – FlowError / Result
  jsondb/             – IndexedDB-compatible JSON document layer
    mod.rs            – JsonDB, Transaction, full API
    encoding.rs       – key/value encoding, JSON field extraction
    schema.rs         – StoreDef / IndexDef / schema persistence
  bin/
    flowdb-stress.rs  – `flowdb-stress` benchmarking binary
tests/
  fuzz_tests.rs       – property/fuzz tests with arbitrary (6 suites)
benches/
  flowdb_bench.rs     – criterion micro-benchmarks
examples/
  (standalone example binaries: basic_engine, basic_jsondb, bench-storage,
   supabase-server, supabase_example)

bindings/node/         – napi-rs native Node.js addon
  src/lib.rs           – #[napi] Rust bindings (FlowDb, JsTransaction)
  Cargo.toml           – napi crate (workspace member)
  build.rs             – napi-build setup
  index.js             – Platform-specific .node loader
  package.json         – npm package config
  ts/                  – TypeScript wrapper source
    index.ts, jsondb.ts, transaction.ts, types.ts

napi-rs Node.js Bindings

bindings/node/ is a Cargo workspace member that builds a native Node.js addon.

Build (release)

cd bindings/node && napi build --platform --release

Test

cd bindings/node && node -e "const m=require('.'); m.FlowDB.open({dataDir:'/tmp/t',createIfMissing:true}).close()"

Cross-platform publishing (local, no GitHub CI)

All 5 targets are built locally — Mac mini handles the two darwin targets, a Linux x86_64 machine handles linux-gnu ×2 and windows-msvc via SSH.

# 1. Mac mini: build darwin-arm64 + darwin-x64
cd bindings/node && ./scripts/build-all.sh

# 2. Linux SSH: build linux-x64 + linux-arm64 + windows-msvc
LINUX_SSH=user@linux-host ./scripts/remote-build.sh

# 3. Generate platform subpackages
node scripts/publish-platforms.js

# 4. Publish main package + 5 platform subpackages
npm publish && for p in npm/*/; do (cd "$p" && npm publish); done

The TypeScript wrapper in ts/ is compiled with tsc to dist/. The index.js loader handles platform detection.

Test Conventions

• Inline #[cfg(test)] mod tests per module for fine-grained unit tests.
• All tests are plain #[test] fn — no #[tokio::test], no async.
• Concurrency tests use std::thread::spawn (not tokio::spawn).
• Tests use tempfile::TempDir for filesystem isolation. The lone exception is src/bin/flowdb-stress.rs's make_temp_dir() helper, which uses an atomic counter to give each parallel test its own directory.
• Tests must clean up after themselves (engine.shutdown().unwrap() for engines owned by value; engine.flush() for engines behind an Arc that cannot be moved out).