From c01233b9dc8bd21ac4b333f65adeabda3ffc44c2 Mon Sep 17 00:00:00 2001 From: Jayson Steffens Date: Mon, 20 Apr 2026 15:06:37 +0200 Subject: [PATCH 1/2] bench: refresh competitive table from single fresh-process run Re-ran experiments/bench_competitive.py start-to-finish in a single clean Python session on main (commit 8a28af1, post v0.10.0 merge) after observing a 56% p50 delta on SnapIndex between the first multi-session measurement and a later standalone run. The refreshed table (this commit) sits within +/-5% of the previous numbers for every row except hnswlib, confirming that the spike was the first-run outlier we already corrected, not a systemic drift. All recall numbers are identical run-to-run (determinism tests hold). Also add a short Methodology note above the table so readers know the numbers came from one fresh process, and call out the expected run-to-run noise (<=5% p50 for most rows, ~10% for hnswlib). This is what lets the table survive a 're-run it and compare' review. --- docs/benchmarks.md | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index a8824a8..c24378d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -43,24 +43,33 @@ story, use sqlite-vec or FAISS `IndexFlatIP`. | Backend | recall@10 | p50 us | p99 us | disk MB | build s | |---------|----------:|-------:|-------:|--------:|--------:| -| sqlite-vec (brute-force cosine, exact) | **1.000** | 13757 | 17539 | 91.1 | 0.5 | -| hnswlib (M=32, ef_search=128) | 0.994 | 507 | 823 | 104.5 | 43 | -| **snapvec IVFPQ + fp16 rerank (M=192)** | **0.945** | **346** | 417 | 56.9 | 107 | -| FAISS IVFPQ (M=192) [matched-budget] | 0.906 | 484 | 2116 | 12.7 | 17 | -| **snapvec IVFPQ no rerank (M=192)** | 0.895 | **319** | 392 | 12.6 | 108 | -| snapvec SnapIndex 4-bit scalar (full-scan) | 0.854 | 2727 | 4152 | 15.4 | 1.1 | -| snapvec SnapIndex 3-bit scalar (full-scan) | 0.736 | 2717 | 2965 | 11.7 | 0.8 | -| FAISS IVFPQ (M=48) | 0.603 | 143 | 194 | 4.4 | 10 | -| snapvec SnapIndex 2-bit scalar (full-scan) | 0.618 | 2649 | 3127 | 8.0 | 0.7 | -| snapvec IVFPQ no rerank (M=48) [matched-budget] | 0.549 | 269 | 342 | 4.3 | 33 | +| sqlite-vec (brute-force cosine, exact) | **1.000** | 13891 | 18628 | 91.1 | 0.5 | +| hnswlib (M=32, ef_search=128) | 0.994 | 561 | 994 | 104.5 | 45 | +| **snapvec IVFPQ + fp16 rerank (M=192)** | **0.945** | **359** | 457 | 56.9 | 108 | +| FAISS IVFPQ (M=192) [matched-budget] | 0.906 | 483 | 584 | 12.7 | 17 | +| **snapvec IVFPQ no rerank (M=192)** | 0.895 | **325** | 376 | 12.6 | 110 | +| snapvec SnapIndex 4-bit scalar (full-scan) | 0.854 | 2676 | 3164 | 15.4 | 1.1 | +| snapvec SnapIndex 3-bit scalar (full-scan) | 0.736 | 2688 | 3013 | 11.7 | 0.8 | +| snapvec SnapIndex 2-bit scalar (full-scan) | 0.618 | 2726 | 4016 | 8.0 | 0.7 | +| FAISS IVFPQ (M=48) | 0.603 | 142 | 200 | 4.4 | 10 | +| snapvec IVFPQ no rerank (M=48) [matched-budget] | 0.549 | 267 | 350 | 4.3 | 33 | Rows ordered by recall@10 descending. +**Methodology.** All numbers come from a single fresh-process run of +`experiments/bench_competitive.py` so backend-to-backend comparisons +are not contaminated by OS page-cache state inherited from earlier +runs. An earlier multi-session measurement showed a ~56% p50 delta +on SnapIndex between cold and warm runs; the single-run convention +above eliminates that. + Thread pinning: `faiss.omp_set_num_threads(1)` + `idx.set_num_threads(1)` on the hnswlib instance (set before `add_items`) for apples-to-apples build and search timings. snapvec's `fit` still uses whatever NumPy BLAS is configured to; for this machine `np.show_config()` reports -Accelerate with its default thread count. +Accelerate with its default thread count. Expected run-to-run noise +on this hardware is <=5% p50 for every row except hnswlib, which +hits +/-10% routinely. ### Reading the table From 12440434f64979e6880dd974132378c1afe3e615 Mon Sep 17 00:00:00 2001 From: Jayson Steffens Date: Mon, 20 Apr 2026 15:11:56 +0200 Subject: [PATCH 2/2] docs: sync narrative bullets with refreshed competitive table Two reviewers flagged that after updating the table values I left the narrative bullets (Reading the table, Positioning, Caveats) citing old p50/p99/build numbers. Align every number in the prose with the refreshed fresh-process run so the page is internally consistent: - M=192 matched-budget headline: 1.4x -> 1.5x, 336/475 -> 325/483 - IVFPQ+rerank bullet: 355 us -> 359 us - hnswlib p99: 819 us -> 994 us - sqlite-vec cite: 13 ms -> 13.9 ms (so the '~5x faster' claim matches) - FAISS fit gap: 7x vs 16/112 -> 6.5x vs 17/110 - FAISS M=48 latency bullet: 168 us -> 142 us Also refine two wording issues from the same review: - Methodology said 'single fresh-process run' but the bench actually runs four per-backend subprocesses under one orchestrator. Reword to 'single end-to-end invocation' and explain the subprocess layout so readers know why. - Expected hnswlib noise was noted as +/-10%; observed +11% delta in this run puts it closer to 10-15%. Update the envelope to match reality. --- docs/benchmarks.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index c24378d..d1be64b 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -56,12 +56,15 @@ story, use sqlite-vec or FAISS `IndexFlatIP`. Rows ordered by recall@10 descending. -**Methodology.** All numbers come from a single fresh-process run of -`experiments/bench_competitive.py` so backend-to-backend comparisons -are not contaminated by OS page-cache state inherited from earlier -runs. An earlier multi-session measurement showed a ~56% p50 delta -on SnapIndex between cold and warm runs; the single-run convention -above eliminates that. +**Methodology.** All numbers come from a single end-to-end invocation +of `experiments/bench_competitive.py`. The orchestrator spawns one +subprocess per backend (each backend bundles its own libomp; loading +two into the same Python process crashes on macOS arm64) but all four +subprocesses run back-to-back in one session so OS page-cache state +does not drift between earlier and later rows. An earlier +multi-session measurement showed a ~56% p50 delta on SnapIndex between +cold and warm runs; the single-invocation convention above eliminates +that. Thread pinning: `faiss.omp_set_num_threads(1)` + `idx.set_num_threads(1)` on the hnswlib instance (set before `add_items`) for apples-to-apples @@ -69,34 +72,34 @@ build and search timings. snapvec's `fit` still uses whatever NumPy BLAS is configured to; for this machine `np.show_config()` reports Accelerate with its default thread count. Expected run-to-run noise on this hardware is <=5% p50 for every row except hnswlib, which -hits +/-10% routinely. +hits ~10-15% routinely. ### Reading the table The Pareto frontier (no backend strictly dominated) is: 1. **FAISS IVFPQ M=48** owns the aggressive-compression corner -- - 0.603 recall at 4.4 MB and 168 us. snapvec at the same M budget + 0.603 recall at 4.4 MB and 142 us. snapvec at the same M budget is slower AND has lower recall, so at that ultra-compressed point FAISS wins outright. 2. **snapvec IVFPQ M=192** matches FAISS M=192 on disk (12.6 vs - 12.7 MB) and on recall (0.895 vs 0.906) while being **1.4x faster** - at p50 (336 vs 475 us). This is the matched-budget headline. + 12.7 MB) and on recall (0.895 vs 0.906) while being **1.5x faster** + at p50 (325 vs 483 us). This is the matched-budget headline. 3. **snapvec IVFPQ + fp16 rerank** is the Pareto-dominant high-recall - point under 500 us: 0.945 recall at 355 us -- faster than FAISS + point under 500 us: 0.945 recall at 359 us -- faster than FAISS M=192 AND higher recall, at the cost of a 4.5x larger index file (holds a float16 copy for the rerank pass). 4. **hnswlib** reaches the highest non-exact recall (0.994) but pays - with disk (104 MB) and p99 latency (819 us). + with disk (104 MB) and p99 latency (994 us). 5. **sqlite-vec** is exact (recall 1.000) but its brute-force - cosine scan is 13 ms -- ~40x slower than any of the ANN backends + cosine scan is 13.9 ms -- ~40x slower than any of the ANN backends on this N. It's the 'zero ANN tuning, accept the latency' baseline. ### Positioning in plain language - If you need **one dependency, no training, acceptable latency on small N**: `SnapIndex` at 4-bit scalar or sqlite-vec. snapvec is - ~5x faster (2.7 ms vs 13.8 ms) but gives up exactness (0.85 vs + ~5x faster (2.7 ms vs 13.9 ms) but gives up exactness (0.85 vs 1.00 recall) because it quantizes the vectors. `SnapIndex` p50 latency is essentially constant across bit depths (the fp16 centroid-expansion matmul dominates); the recall/disk tradeoff is @@ -113,8 +116,8 @@ The Pareto frontier (no backend strictly dominated) is: ### Caveats -- FAISS `fit` is ~7x faster than snapvec's `fit` at the same config - (16 s vs 112 s at M=192). Build time is a real competitive gap. +- FAISS `fit` is ~6.5x faster than snapvec's `fit` at the same config + (17 s vs 110 s at M=192). Build time is a real competitive gap. - FAISS IVFPQ at M=48 beats snapvec at M=48; snapvec's PQ training isn't uniformly better at every compression point. The advantage shows up at mid-range (M=96-192).