bench: publish search_batch threading curve (FIQA, M4 Pro) - #51
Conversation
Completes a WIP benchmark that was never run (it called search() with num_threads, which only exists on search_batch). Rewritten against search_batch with an explicit close()/rebuild between num_threads settings to work around the executor's lazy one-shot init. Headline numbers on Apple M4 Pro, BEIR FIQA (N=57,638, dim=384), batch_size=128: - num_threads=4 is the sweet spot across every nprobe - scaling improves with nprobe: 1.7x at nprobe=4, 3.7x at nprobe=256 - sub-millisecond per query at 4 threads for nprobe<=64 (3.4k-20k QPS) - num_threads=8 regresses: over-subscribes efficiency cores vs BLAS Added the table to docs/benchmarks.md under a new 'Batched search threading curve' section, linked the bench in the Reproduction block.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new benchmarking script, experiments/bench_ivf_pq_threading.py, designed to measure the performance of IVFPQSnapIndex.search_batch across different thread counts. Additionally, docs/benchmarks.md has been updated to include a threading curve analysis and instructions for running the new benchmark. I have no feedback to provide.
There was a problem hiding this comment.
Pull request overview
Adds and documents a completed benchmark for measuring IVFPQSnapIndex.search_batch throughput scaling across different num_threads settings on the FIQA harness.
Changes:
- Introduce
experiments/bench_ivf_pq_threading.pyto benchmarksearch_batchacrossnprobeand thread-count settings. - Document the resulting threading curve and key takeaways in
docs/benchmarks.md. - Update the reproduction command list to include the new benchmark script.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| experiments/bench_ivf_pq_threading.py | New benchmark script to measure search_batch performance vs. num_threads across nprobe values. |
| docs/benchmarks.md | Adds a “Batched search threading curve” section and updates reproduction commands. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| batch_size: int = BATCH_SIZE, | ||
| ) -> tuple[float, float]: | ||
| """Return (ms_per_query, total_elapsed_s) averaged over all batches.""" | ||
| # Warm-up: first batch hits caches and spawns the executor. |
There was a problem hiding this comment.
The warm-up comment is slightly inaccurate: search_batch(..., num_threads=1) will not spawn the thread-pool executor (the executor is only created when num_threads > 1). Consider rewording to reflect that it always warms caches and only creates the executor when threading is enabled.
| # Warm-up: first batch hits caches and spawns the executor. | |
| # Warm-up: first batch warms caches; it only creates the executor when | |
| # threading is enabled (num_threads > 1). |
| row = [] | ||
| for t in THREAD_COUNTS: | ||
| # The executor is lazily created and locked to the first | ||
| # num_threads value. Close and rebuild between t settings. |
There was a problem hiding this comment.
The inline comment says “Close and rebuild between t settings”, but the code only calls idx.close() (which just releases the lazy executor) and does not rebuild the index. Suggest updating the comment to match the actual behavior (e.g., “call close() between thread-count settings to reset the executor”).
| # num_threads value. Close and rebuild between t settings. | |
| # num_threads value. Call close() between t settings to | |
| # reset the executor so it can be recreated for the next t. |
| That is 3,400 - 20,000 queries per second per process. | ||
|
|
||
| Reproduce with `python experiments/bench_ivf_pq_threading.py` after | ||
| caching the FIQA corpus. |
There was a problem hiding this comment.
This reproduction note mentions caching the FIQA corpus, but the benchmark also requires the cached FIQA queries file (experiments/.cache_fiqa_queries_bge_small.npy). Update the text to mention caching/downloading both corpus and queries.
| caching the FIQA corpus. | |
| caching/downloading both the FIQA corpus and the FIQA queries file | |
| (`experiments/.cache_fiqa_queries_bge_small.npy`). |
- bench_ivf_pq_threading.py warm-up comment: the executor is only created when num_threads > 1, not on every search_batch. - bench_ivf_pq_threading.py close() comment: close() releases the executor; it does not rebuild the index. - docs/benchmarks.md: the bench needs both the corpus and queries caches. Call out both files in the reproduction note.
Bumps the version to 0.10.0 and documents the landed work. No library behaviour changes beyond the surgical SnapIndex.search(k<1) validation shipped in PR #50; everything else is CI, docs, tests, and benchmarks. Adds: - CHANGELOG entry for 0.10.0 covering PRs #43, #49, #50, #51, #52, and #53 (CI matrix + wheels, MkDocs site, 40+ new tests, threading curve, forward-compat errors, competitive Pareto bench). - ROADMAP.md with scoped plans for v0.11 (streaming ingest, OPQ, strict mypy), v0.12 (file format v2, delta buffer), and v1.0 (API freeze + deprecation policy). Explicit non-goals so nobody opens a GPU-backend PR. - CITATION.cff so downstream papers can cite snapvec with a machine-readable metadata file. References TurboQuant (arXiv:2504.19874) and Jegou et al. product quantization as underlying algorithms. - README roadmap link so ROADMAP.md is discoverable from the project root.
Summary
Completes a WIP benchmark that was never run (it called `search()`
with `num_threads`, which only exists on `search_batch`). Rewritten
against `search_batch` with an explicit `close()`/rebuild between
`num_threads` settings to work around the executor's lazy one-shot
init.
Threading curve (Apple M4 Pro, FIQA, N=57,638)
Values in ms/query, batch_size=128.
Key takeaways documented in docs/benchmarks.md
(per-query work grows, so threading overhead amortises).
(3.4k - 20k QPS per process).
efficiency cores and starts fighting BLAS.
Test plan
it needs the cached corpus)
Notes
in either order.