docs: phase 2 -- MkDocs site, API reference, executable examples - #49
Conversation
Six stand-alone example scripts that exercise the public API end to end: quickstart, pq_index, ivf_pq (with fp16 rerank), filter_search, save_load, and streaming_ingest. Each example is tiny (synthetic data, a few seconds) and runs as part of the CI matrix on every OS + Python combination, so a broken public API shows up before release.
- mkdocs.yml: Material theme with dark/light toggle, mkdocstrings for auto-generated API reference, navigation tree with getting-started, user-guide, architecture, benchmarks, API, changelog. - docs/: 16 pages. User guide has one page per index type plus 'choosing an index' decision tree, save/load, and filtered search. Architecture explains RHT, Lloyd-Max, PQ, IVF, and fp16 rerank. Benchmarks page carries the FIQA and sqlite-vec comparison numbers. API reference is five short pages that mkdocstrings expands from the existing docstrings -- zero extra maintenance burden. - docs/changelog.md uses mkdocs-material's snippet include so the canonical CHANGELOG.md stays at the repo root. - docs/blog/ (pre-existing drafts) is excluded from the built site. - pyproject.toml: new [docs] optional extra pulls mkdocs-material and mkdocstrings. Regular dev install unaffected. - .github/workflows/docs.yml: strict build on every push and PR, deploy to GitHub Pages on main. PR builds validate without deploying. - .gitignore: exclude the site/ build output.
Reduce README from 673 lines to 91. Keep the hero, the decision-tree summary table, install + quickstart, and the headline benchmark. Push everything else (sizing, algorithm, full API reference, roadmap) to the MkDocs site where it lives under structured navigation. Link the mkdocs.io site from the badges row and from a dedicated Docs section so the README is primarily an entry point, not a reference manual. Preserves the Context / Contributing / License sections at the bottom.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive documentation site built with MkDocs, including a new architecture overview, user guides, and API references. It also adds several runnable examples and updates the README to be more concise. The review feedback identifies several inaccuracies in the new documentation: the handling of float64 inputs is described as an error when it is actually cast, certain required arguments are incorrectly labeled as having defaults, and the implementation details regarding filtered search performance and ID type preservation are misrepresented.
| `snapvec` expects `np.float32` inputs everywhere. Passing `np.float64` | ||
| (the NumPy default for `np.array([...])`) is an error because silently | ||
| downcasting would be a surprise on the hot path. |
There was a problem hiding this comment.
The documentation states that passing np.float64 is an error, but the implementation in SnapIndex.add_batch (and other index classes) uses np.asarray(vectors, dtype=np.float32), which performs a cast rather than raising an error. If the intention is to enforce float32 strictly to avoid silent performance penalties, the code should be updated to raise a TypeError or ValueError. Otherwise, this documentation should be corrected to reflect that a cast occurs.
| | `M` (PQ) | `dim // 4` | Higher M = higher recall, more disk. `M=16` is a common starting point for `dim=384` (BGE-small). | | ||
| | `K` (PQ) | 256 | Fixed at 256 (one byte per sub-index). | | ||
| | `nlist` (IVF) | `4 * sqrt(N)` | E.g. N=57k -> nlist=512, N=1M -> nlist=4096. | |
There was a problem hiding this comment.
The "Default" column in this table lists values for M and nlist, but these are actually required arguments in the constructors for PQSnapIndex and IVFPQSnapIndex respectively. Labeling them as having defaults is misleading. It would be better to mark them as required or move the suggested values to the "Guidance" column.
| @@ -0,0 +1,29 @@ | |||
| # Filtered search | |||
|
|
|||
| Every index accepts `filter_ids=<set>` to restrict results to a subset | |||
There was a problem hiding this comment.
This statement is currently inaccurate. While SnapIndex and IVFPQSnapIndex support filter_ids, the search methods for PQSnapIndex and ResidualSnapIndex do not yet accept this parameter. This appears to be a roadmap item that has been documented as already implemented.
| - **SnapIndex / PQSnapIndex / ResidualSnapIndex**: the filter is applied | ||
| after scoring, so it does not save scoring work. Useful when the | ||
| filter set is small relative to corpus. |
There was a problem hiding this comment.
The claim that SnapIndex does not save scoring work when using a filter is incorrect. The implementation in SnapIndex.search slices the indices based on the filter before performing the matrix multiplication (see snapvec/_index.py lines 434-454), which significantly reduces computation for sparse filters. The documentation should be updated to reflect this performance benefit.
| in the file and round-tripped through `load`. If you pass integers, | ||
| they come back as integers; strings come back as strings. |
There was a problem hiding this comment.
The statement "strings come back as strings" is not entirely accurate. The _decode_id helper used during load() (e.g., in snapvec/_index.py lines 638-644) attempts to parse string IDs as int or float. Consequently, a string ID that looks like a number (e.g., "123") will be loaded back as a numeric type, which may be unexpected for users relying on type preservation for IDs.
All 5 inline comments were valid. - installation.md: float64 is silently cast by np.asarray, not raised as an error. Warn about the hidden allocation instead. - choosing-an-index.md: rename 'Default' column to 'Required?'; clarify that M (PQ) and nlist (IVF) are required ctor args, and give recommended starting values in the guidance column. - filter-search.md: only SnapIndex and IVFPQSnapIndex expose filter_ids today; PQSnapIndex and ResidualSnapIndex do not. - filter-search.md: SnapIndex applies the filter before the matmul, so it *does* save scoring work on sparse filters; updated to describe the actual O(|filter_ids| * dim) cost. - save-load.md: _decode_id parses numeric-looking strings back to int/float. Document the exact behaviour and the 'prefix to keep it a string' workaround.
There was a problem hiding this comment.
Pull request overview
Adds a first-class documentation site for snapvec (MkDocs + Material + mkdocstrings), introduces runnable examples and CI smoke-testing for them, and trims the README to be an entry point that points users to the docs site.
Changes:
- Add MkDocs site configuration, user-guide pages, architecture/benchmarks pages, and mkdocstrings-based API reference.
- Add 6 executable example scripts and run them in CI as a smoke test.
- Update README to a shorter “hero + quickstart + docs links” format and add a docs workflow + Pages deploy.
Reviewed changes
Copilot reviewed 29 out of 30 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Adds a docs optional-dependency extra for MkDocs tooling. |
| mkdocs.yml | Introduces MkDocs Material configuration, plugins, and nav structure. |
| examples/quickstart.py | Adds a minimal SnapIndex runnable example script. |
| examples/pq_index.py | Adds a PQSnapIndex runnable example script. |
| examples/ivf_pq.py | Adds an IVFPQSnapIndex runnable example script (incl. rerank). |
| examples/filter_search.py | Adds a filtered-search runnable example script. |
| examples/save_load.py | Adds a save/load round-trip runnable example script. |
| examples/streaming_ingest.py | Adds a streaming-ingest runnable example script. |
| docs/index.md | Adds docs site landing page and a quickstart snippet. |
| docs/getting-started/installation.md | Adds installation guidance for PyPI/source + dtype notes. |
| docs/getting-started/quickstart.md | Adds end-to-end “five-minute tour” content. |
| docs/user-guide/choosing-an-index.md | Adds decision table + sizing heuristics across index types. |
| docs/user-guide/snap-index.md | Adds SnapIndex user-guide page. |
| docs/user-guide/pq.md | Adds PQSnapIndex user-guide page. |
| docs/user-guide/ivf-pq.md | Adds IVFPQSnapIndex user-guide page. |
| docs/user-guide/residual.md | Adds ResidualSnapIndex user-guide page. |
| docs/user-guide/save-load.md | Adds save/load guarantees and format/ID notes. |
| docs/user-guide/filter-search.md | Adds filtered-search behavior/perf/edge-case notes. |
| docs/architecture.md | Adds high-level explanation of the algorithms and pipeline. |
| docs/benchmarks.md | Adds benchmark results summary and reproduction pointers. |
| docs/api/snap-index.md | Adds mkdocstrings entry for SnapIndex API. |
| docs/api/pq.md | Adds mkdocstrings entry for PQSnapIndex API. |
| docs/api/ivf-pq.md | Adds mkdocstrings entry for IVFPQSnapIndex API. |
| docs/api/residual.md | Adds mkdocstrings entry for ResidualSnapIndex API. |
| docs/api/helpers.md | Adds mkdocstrings entries for helper functions (rht, padded_dim, get_codebook). |
| docs/changelog.md | Includes top-level CHANGELOG via snippets include. |
| README.md | Trims README and adds docs links + docs workflow badge. |
| .gitignore | Ignores MkDocs site/ output. |
| .github/workflows/docs.yml | Adds docs build + GitHub Pages deploy workflow. |
| .github/workflows/ci.yml | Runs the new examples as a CI smoke test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| corpus = rng.standard_normal((10_000, 384)).astype(np.float32) | ||
|
|
||
| #### Why IVF-PQ + rerank beats PQ full-scan | ||
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) |
There was a problem hiding this comment.
The quickstart snippet passes normalized=True while building corpus/query from standard_normal, which are not unit-length. In snapvec, normalized=True means the caller guarantees inputs are already unit-normalized; otherwise scores won’t correspond to cosine similarity. Either drop normalized=True (use the default) or explicitly L2-normalize corpus and query before adding/searching.
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) | |
| idx = SnapIndex(dim=384, bits=4, seed=0) |
| idx = PQSnapIndex(dim=dim, M=16, K=256, normalized=True, seed=0) | ||
| idx.fit(corpus[:1000]) # train codebooks on first half | ||
| idx.add_batch(list(range(n_corpus)), corpus) |
There was a problem hiding this comment.
This example passes normalized=True but uses random standard_normal vectors (not unit-length). For PQSnapIndex, normalized=True means inputs are already unit-length and no norms are stored; using it with unnormalized data changes the scoring behavior. Either remove normalized=True or normalize corpus/queries first.
| corpus = rng.standard_normal((10_000, 384)).astype(np.float32) | ||
|
|
||
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) | ||
| idx.add_batch(list(range(10_000)), corpus) | ||
| ``` |
There was a problem hiding this comment.
This snippet uses normalized=True with rng.standard_normal(...) inputs, which are not unit-length. Since normalized=True skips normalization in add_batch, the example won’t reflect cosine similarity behavior as described. Use the default normalized=False or normalize corpus/query first.
| `ids` can be any hashable Python value. They are serialized as strings | ||
| in the file; on `load()`, `snapvec` tries to decode each string as | ||
| `int`, then `float`, and falls back to the raw string. That means: |
There was a problem hiding this comment.
The “IDs” section implies any hashable id round-trips through load, but ids are written as UTF-8 strings and decoded back only for numeric-looking values (int/float); everything else loads as a string. Please adjust this wording to match the actual serialization behavior so users don’t assume arbitrary Python objects will round-trip.
| `ids` can be any hashable Python value. They are serialized as strings | |
| in the file; on `load()`, `snapvec` tries to decode each string as | |
| `int`, then `float`, and falls back to the raw string. That means: | |
| `ids` are serialized in the file as UTF-8 strings. On `load()`, | |
| integer-looking values come back as integers, float-looking values | |
| come back as floats, and all other values come back as strings. | |
| Do not rely on arbitrary Python objects preserving their original type | |
| across save/load. |
| corpus = np.random.randn(100_000, 384).astype(np.float32) | ||
|
|
||
| idx = IVFPQSnapIndex( | ||
| dim=384, | ||
| nlist=512, # 4 * sqrt(N) | ||
| M=16, | ||
| K=256, | ||
| normalized=True, | ||
| keep_full_precision=True, | ||
| seed=0, | ||
| ) | ||
| idx.fit(corpus[:20_000]) | ||
| idx.add_batch(list(range(100_000)), corpus) | ||
|
|
||
| hits = idx.search(query, k=10, nprobe=32, rerank_candidates=100) | ||
| ``` |
There was a problem hiding this comment.
This snippet passes normalized=True but uses np.random.randn(...) inputs (not unit-length), which contradicts the meaning of normalized=True in IVFPQSnapIndex. Also, query is referenced without being defined. Prefer normalized=False in the snippet (or normalize inputs explicitly) and define query for a copy/pasteable example.
| idx = SnapIndex(dim=dim, bits=4, normalized=True, seed=0) | ||
| idx.add_batch(ids, corpus) | ||
|
|
||
| query = rng.standard_normal(dim).astype(np.float32) | ||
|
|
||
| unrestricted = idx.search(query, k=5) | ||
| restricted_set = {f"doc-{i:04d}" for i in range(100)} | ||
| restricted = idx.search(query, k=5, filter_ids=restricted_set) |
There was a problem hiding this comment.
This example uses normalized=True but the corpus/query vectors are drawn from standard_normal (not unit-length). Since normalized=True skips normalization at ingest, the search scores won’t correspond to cosine similarity. Prefer the default normalized=False or explicitly normalize inputs before indexing/searching.
| corpus = np.random.randn(10_000, 384).astype(np.float32) | ||
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) | ||
| idx.add_batch(list(range(10_000)), corpus) | ||
|
|
||
| query = np.random.randn(384).astype(np.float32) |
There was a problem hiding this comment.
The code sample sets normalized=True but corpus is generated with np.random.randn(...) and is not unit-length. In snapvec, normalized=True means inputs are already unit-normalized; otherwise the example’s scores/behavior won’t match cosine similarity. Either remove normalized=True or normalize corpus and query explicitly.
| corpus = np.random.randn(10_000, 384).astype(np.float32) | |
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) | |
| idx.add_batch(list(range(10_000)), corpus) | |
| query = np.random.randn(384).astype(np.float32) | |
| corpus = np.random.randn(10_000, 384).astype(np.float32) | |
| corpus /= np.linalg.norm(corpus, axis=1, keepdims=True) | |
| idx = SnapIndex(dim=384, bits=4, normalized=True, seed=0) | |
| idx.add_batch(list(range(10_000)), corpus) | |
| query = np.random.randn(384).astype(np.float32) | |
| query /= np.linalg.norm(query) |
| rng = np.random.default_rng(0) | ||
| dim, n_corpus = 128, 500 | ||
|
|
||
| corpus = rng.standard_normal((n_corpus, dim)).astype(np.float32) |
There was a problem hiding this comment.
This example passes normalized=True but corpus is drawn from standard_normal and is not unit-length. Since normalized=True skips normalization in add_batch, the search scores won’t match cosine similarity as implied by most docs. Use the default normalized=False or explicitly L2-normalize corpus and query.
| corpus = rng.standard_normal((n_corpus, dim)).astype(np.float32) | |
| corpus = rng.standard_normal((n_corpus, dim)).astype(np.float32) | |
| norms = np.linalg.norm(corpus, axis=1, keepdims=True) | |
| norms = np.where(norms == 0, 1.0, norms) | |
| corpus = corpus / norms |
| """Minimal SnapIndex example. | ||
|
|
||
| Build a 4-bit scalar-quantized index over 1,000 random vectors and | ||
| recover the exact top-1 for a handful of queries without any training. |
There was a problem hiding this comment.
The module docstring says it will “recover the exact top-1 … without any training”, but SnapIndex is approximate and the script even computes recall. Consider rewording to avoid implying exactness/guarantees (e.g., “achieves high top-1 recall”).
| recover the exact top-1 for a handful of queries without any training. | |
| measure top-1 recall for a handful of queries without any training. |
| idx = SnapIndex(dim=384, bits=4, normalized=True) | ||
| idx.add_batch(list(range(10_000)), np.random.randn(10_000, 384).astype(np.float32)) | ||
|
|
||
| results = idx.search(np.random.randn(384).astype(np.float32), k=10) |
There was a problem hiding this comment.
This quickstart uses random (non-unit) vectors but passes normalized=True. In snapvec, normalized=True means inputs are already unit-length and skips normalization on ingest, so this snippet can mislead users about expected behavior. Prefer the default normalized=False here, or explicitly L2-normalize the inputs.
| idx = SnapIndex(dim=384, bits=4, normalized=True) | |
| idx.add_batch(list(range(10_000)), np.random.randn(10_000, 384).astype(np.float32)) | |
| results = idx.search(np.random.randn(384).astype(np.float32), k=10) | |
| vectors = np.random.randn(10_000, 384).astype(np.float32) | |
| vectors /= np.linalg.norm(vectors, axis=1, keepdims=True) | |
| query = np.random.randn(384).astype(np.float32) | |
| query /= np.linalg.norm(query) | |
| idx = SnapIndex(dim=384, bits=4, normalized=True) | |
| idx.add_batch(list(range(10_000)), vectors) | |
| results = idx.search(query, k=10) |
Thirteen of 16 comments were about the same mismatch: examples and docs passed normalized=True with raw random-normal vectors. normalized=True means 'caller guarantees unit-length inputs' and skips internal normalization, so the snippets were silently computing uncalibrated scores instead of cosine similarity. Drop normalized=True everywhere the surrounding code produces non-unit vectors; the default (False) already does the right thing. Additional fixes: - examples/quickstart.py docstring: SnapIndex is approximate. Reword from 'recover the exact top-1' to 'measure top-1 recall'. - docs/getting-started/quickstart.md: expand the 'ids can be any hashable' line to call out that only numeric-looking values round-trip to their original type; everything else comes back as str. - docs/user-guide/snap-index.md: add an admonition explaining when to use normalized=True so the flag does not get cargo-culted. - docs/user-guide/pq.md / ivf-pq.md / residual.md: define the query variable in each snippet so they are self-contained. - .github/workflows/docs.yml: scope pages:write / id-token:write to the deploy job only; workflow-level permissions applied to PR builds too (principle of least privilege).
Bumps the version to 0.10.0 and documents the landed work. No library behaviour changes beyond the surgical SnapIndex.search(k<1) validation shipped in PR #50; everything else is CI, docs, tests, and benchmarks. Adds: - CHANGELOG entry for 0.10.0 covering PRs #43, #49, #50, #51, #52, and #53 (CI matrix + wheels, MkDocs site, 40+ new tests, threading curve, forward-compat errors, competitive Pareto bench). - ROADMAP.md with scoped plans for v0.11 (streaming ingest, OPQ, strict mypy), v0.12 (file format v2, delta buffer), and v1.0 (API freeze + deprecation policy). Explicit non-goals so nobody opens a GPU-backend PR. - CITATION.cff so downstream papers can cite snapvec with a machine-readable metadata file. References TurboQuant (arXiv:2504.19874) and Jegou et al. product quantization as underlying algorithms. - README roadmap link so ROADMAP.md is discoverable from the project root.
Summary
Phase 2 of professionalizing snapvec. Adds a first-class documentation
site and executable examples, and trims the README to an entry point.
Commits
docs: add executable examples + CI smoke test-- 6 stand-alonescripts under `examples/` plus a CI step that runs them on every
OS + Python combination.
docs: add MkDocs site with Material theme + API reference--`mkdocs.yml` + 16 documentation pages + GitHub Pages deploy
workflow. API pages use mkdocstrings so they stay in sync with the
docstrings automatically.
docs: trim README to hero + quickstart + docs links-- 673 linesdown to 91. All technical content now lives on the docs site under
structured navigation.
Site structure
```
Home
Getting started/
Installation
Quickstart
User guide/
Choosing an index
SnapIndex
PQSnapIndex
IVFPQSnapIndex
ResidualSnapIndex
Save and load
Filtered search
Architecture
Benchmarks
API reference/
SnapIndex / PQSnapIndex / IVFPQSnapIndex / ResidualSnapIndex / Helpers
Changelog
```
Manual steps required before the site goes live
After merging, in repo Settings -> Pages:
The Docs workflow will then publish to `https://stffns.github.io/snapvec/\`.
Test plan
Out of scope