Skip to content

bench: refresh competitive table from single fresh-process run #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stffns merged 2 commits into from
Apr 20, 2026
Merged

bench: refresh competitive table from single fresh-process run #55

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c01233b
bench: refresh competitive table from single fresh-process run
Apr 20, 2026
1244043
docs: sync narrative bullets with refreshed competitive table
Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 32 additions & 20 deletions docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,51 +43,63 @@ story, use sqlite-vec or FAISS `IndexFlatIP`.

| Backend | recall@10 | p50 us | p99 us | disk MB | build s |
|---------|----------:|-------:|-------:|--------:|--------:|
| sqlite-vec (brute-force cosine, exact) | **1.000** | 13757 | 17539 | 91.1 | 0.5 |
| hnswlib (M=32, ef_search=128) | 0.994 | 507 | 823 | 104.5 | 43 |
| **snapvec IVFPQ + fp16 rerank (M=192)** | **0.945** | **346** | 417 | 56.9 | 107 |
| FAISS IVFPQ (M=192) [matched-budget] | 0.906 | 484 | 2116 | 12.7 | 17 |
| **snapvec IVFPQ no rerank (M=192)** | 0.895 | **319** | 392 | 12.6 | 108 |
| snapvec SnapIndex 4-bit scalar (full-scan) | 0.854 | 2727 | 4152 | 15.4 | 1.1 |
| snapvec SnapIndex 3-bit scalar (full-scan) | 0.736 | 2717 | 2965 | 11.7 | 0.8 |
| FAISS IVFPQ (M=48) | 0.603 | 143 | 194 | 4.4 | 10 |
| snapvec SnapIndex 2-bit scalar (full-scan) | 0.618 | 2649 | 3127 | 8.0 | 0.7 |
| snapvec IVFPQ no rerank (M=48) [matched-budget] | 0.549 | 269 | 342 | 4.3 | 33 |
| sqlite-vec (brute-force cosine, exact) | **1.000** | 13891 | 18628 | 91.1 | 0.5 |
| hnswlib (M=32, ef_search=128) | 0.994 | 561 | 994 | 104.5 | 45 |
| **snapvec IVFPQ + fp16 rerank (M=192)** | **0.945** | **359** | 457 | 56.9 | 108 |
| FAISS IVFPQ (M=192) [matched-budget] | 0.906 | 483 | 584 | 12.7 | 17 |
| **snapvec IVFPQ no rerank (M=192)** | 0.895 | **325** | 376 | 12.6 | 110 |
| snapvec SnapIndex 4-bit scalar (full-scan) | 0.854 | 2676 | 3164 | 15.4 | 1.1 |
| snapvec SnapIndex 3-bit scalar (full-scan) | 0.736 | 2688 | 3013 | 11.7 | 0.8 |
| snapvec SnapIndex 2-bit scalar (full-scan) | 0.618 | 2726 | 4016 | 8.0 | 0.7 |
| FAISS IVFPQ (M=48) | 0.603 | 142 | 200 | 4.4 | 10 |
| snapvec IVFPQ no rerank (M=48) [matched-budget] | 0.549 | 267 | 350 | 4.3 | 33 |
Comment on lines +46 to +55

Rows ordered by recall@10 descending.

**Methodology.** All numbers come from a single end-to-end invocation
of `experiments/bench_competitive.py`. The orchestrator spawns one
subprocess per backend (each backend bundles its own libomp; loading
two into the same Python process crashes on macOS arm64) but all four
subprocesses run back-to-back in one session so OS page-cache state
does not drift between earlier and later rows. An earlier
multi-session measurement showed a ~56% p50 delta on SnapIndex between
cold and warm runs; the single-invocation convention above eliminates
that.

Thread pinning: `faiss.omp_set_num_threads(1)` + `idx.set_num_threads(1)`
on the hnswlib instance (set before `add_items`) for apples-to-apples
build and search timings. snapvec's `fit` still uses whatever NumPy
BLAS is configured to; for this machine `np.show_config()` reports
Accelerate with its default thread count.
Accelerate with its default thread count. Expected run-to-run noise
on this hardware is <=5% p50 for every row except hnswlib, which
hits ~10-15% routinely.

### Reading the table

@gemini-code-assist gemini-code-assist Bot Apr 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The benchmark table has been updated with fresh numbers, but the descriptive text in the following sections (### Reading the table, ### Positioning in plain language, and ### Caveats) still contains hardcoded values from the previous run.

For example:

  • Line 84: (336 vs 475 us) should be (325 vs 483 us).
  • Line 86: 355 us should be 359 us.
  • Line 90: 819 us should be 994 us.
  • Line 92: 13 ms should be 13.9 ms.
  • Line 117: (16 s vs 112 s) should be (17 s vs 110 s).

These should be updated to maintain consistency with the new table data.


The Pareto frontier (no backend strictly dominated) is:

1. **FAISS IVFPQ M=48** owns the aggressive-compression corner --
0.603 recall at 4.4 MB and 168 us. snapvec at the same M budget
0.603 recall at 4.4 MB and 142 us. snapvec at the same M budget
is slower AND has lower recall, so at that ultra-compressed point
FAISS wins outright.
2. **snapvec IVFPQ M=192** matches FAISS M=192 on disk (12.6 vs
12.7 MB) and on recall (0.895 vs 0.906) while being **1.4x faster**
at p50 (336 vs 475 us). This is the matched-budget headline.
12.7 MB) and on recall (0.895 vs 0.906) while being **1.5x faster**
at p50 (325 vs 483 us). This is the matched-budget headline.
3. **snapvec IVFPQ + fp16 rerank** is the Pareto-dominant high-recall
point under 500 us: 0.945 recall at 355 us -- faster than FAISS
point under 500 us: 0.945 recall at 359 us -- faster than FAISS
M=192 AND higher recall, at the cost of a 4.5x larger index file
(holds a float16 copy for the rerank pass).
4. **hnswlib** reaches the highest non-exact recall (0.994) but pays
with disk (104 MB) and p99 latency (819 us).
with disk (104 MB) and p99 latency (994 us).
5. **sqlite-vec** is exact (recall 1.000) but its brute-force
cosine scan is 13 ms -- ~40x slower than any of the ANN backends
cosine scan is 13.9 ms -- ~40x slower than any of the ANN backends
on this N. It's the 'zero ANN tuning, accept the latency' baseline.

### Positioning in plain language

- If you need **one dependency, no training, acceptable latency on
small N**: `SnapIndex` at 4-bit scalar or sqlite-vec. snapvec is
~5x faster (2.7 ms vs 13.8 ms) but gives up exactness (0.85 vs
~5x faster (2.7 ms vs 13.9 ms) but gives up exactness (0.85 vs
1.00 recall) because it quantizes the vectors. `SnapIndex` p50
latency is essentially constant across bit depths (the fp16
centroid-expansion matmul dominates); the recall/disk tradeoff is
Expand All @@ -104,8 +116,8 @@ The Pareto frontier (no backend strictly dominated) is:

### Caveats

- FAISS `fit` is ~7x faster than snapvec's `fit` at the same config
(16 s vs 112 s at M=192). Build time is a real competitive gap.
- FAISS `fit` is ~6.5x faster than snapvec's `fit` at the same config
(17 s vs 110 s at M=192). Build time is a real competitive gap.
- FAISS IVFPQ at M=48 beats snapvec at M=48; snapvec's PQ training
isn't uniformly better at every compression point. The advantage
shows up at mid-range (M=96-192).
Expand Down
Loading