From 71513467c52e13d42f8e5beb675e23e658105ff5 Mon Sep 17 00:00:00 2001 From: Jayson Steffens Date: Mon, 20 Apr 2026 14:32:03 +0200 Subject: [PATCH 1/3] release: v0.10.0 -- professionalisation release Bumps the version to 0.10.0 and documents the landed work. No library behaviour changes beyond the surgical SnapIndex.search(k<1) validation shipped in PR #50; everything else is CI, docs, tests, and benchmarks. Adds: - CHANGELOG entry for 0.10.0 covering PRs #43, #49, #50, #51, #52, and #53 (CI matrix + wheels, MkDocs site, 40+ new tests, threading curve, forward-compat errors, competitive Pareto bench). - ROADMAP.md with scoped plans for v0.11 (streaming ingest, OPQ, strict mypy), v0.12 (file format v2, delta buffer), and v1.0 (API freeze + deprecation policy). Explicit non-goals so nobody opens a GPU-backend PR. - CITATION.cff so downstream papers can cite snapvec with a machine-readable metadata file. References TurboQuant (arXiv:2504.19874) and Jegou et al. product quantization as underlying algorithms. - README roadmap link so ROADMAP.md is discoverable from the project root. --- CHANGELOG.md | 86 +++++++++++++++++++++++++++++++++++++++++++++ CITATION.cff | 60 +++++++++++++++++++++++++++++++ README.md | 4 +++ ROADMAP.md | 83 +++++++++++++++++++++++++++++++++++++++++++ pyproject.toml | 2 +- snapvec/__init__.py | 2 +- 6 files changed, 235 insertions(+), 2 deletions(-) create mode 100644 CITATION.cff create mode 100644 ROADMAP.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce9625..6966a5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,92 @@ the project uses [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [0.10.0] -- 2026-04-20 + +Headline: **professionalisation release.** No library behaviour +changes; every PR landed between `v0.9.0` and this tag wires up the +infrastructure, tests, documentation, and benchmarks that make snapvec +ready for third-party adoption and publishable comparisons. + +### Added + +- **CI matrix** across Linux x86_64/aarch64, macOS-13 / macOS-14, and + Windows on CPython 3.10, 3.12, 3.13 (PR #43). Pre-compiled wheels + via `cibuildwheel` + PyPI trusted publishing on `v*` tags. +- **Documentation site** at built + from MkDocs Material + `mkdocstrings`. Eighteen pages: getting + started, one user-guide per index type, architecture, benchmarks, + concurrency contract, API reference, changelog (PR #49). +- **Executable examples** in `examples/` (quickstart, PQ, IVF-PQ with + rerank, filter search, save/load, streaming ingest) exercised by + the CI matrix on every commit (PR #49). +- **Community files**: `SECURITY.md`, `CODE_OF_CONDUCT.md`, + `CONTRIBUTING.md`, bug / feature / PR templates, dependabot, a + root `CLAUDE.md`, pre-commit config (PR #43). +- **Test suite grew from 151 to 190+**: + - Hypothesis property-based tests (order-invariance, save/load + round-trip, delete reduces len, search respects k, filter + subset) across SnapIndex, PQSnapIndex, IVFPQSnapIndex (PR #50). + - Determinism tests: same seed + same inputs produces byte-identical + index files for all four classes (PR #50). + - Adversarial tests: empty index, n=1, k > n, zero-norm queries, + degenerate distributions, bit-extreme configurations (PR #50). + - Forward-compat tests: patch a known-good file's version byte and + assert the loader raises an upgrade hint (PR #52). + - Recall regression smoke test on a clustered synthetic corpus: + asserts IVFPQ + rerank stays >= 0.90 recall@10 on every CI run + (PR #52). +- **`bench/threading`** curve on search_batch: p50 across num_threads + = 1, 2, 4, 8 and nprobe in (4..256). Published in docs (PR #51). +- **`bench/competitive`** head-to-head on BEIR FIQA: snapvec (three + bit-depths for flat + IVFPQ with and without rerank at M=48 and + M=192) vs FAISS IVFPQ (M=48 and M=192 matched-budget) vs hnswlib + vs sqlite-vec. Unified Pareto table in docs (PR #53). +- **MkDocs site now deploys automatically** to GitHub Pages on every + merge to `main` (PR #49). + +### Changed + +- **`SnapIndex.search(k)`** now raises `ValueError` for `k < 1`, matching + the validation that `PQSnapIndex`, `ResidualSnapIndex`, and + `IVFPQSnapIndex` already had (PR #50). Caught by the new adversarial + suite; prior behaviour returned all results on `k=0` and `n-1` + results on `k=-1` via an accidental slicing interaction. +- **Forward-compat errors** on `load()` now name the versions this + build supports and point at `pip install -U snapvec` (PR #52). The + old `"unsupported version N"` is gone. +- **README trimmed from 673 to 91 lines**; technical content moved to + the MkDocs site under structured navigation (PR #49). +- **macOS arm64 build** uses `-mcpu=native` instead of `-march=native` + (which Xcode 15 clang rejects as `unknown target CPU 'apple-m1'`). + Fixed the CI build on GitHub's `macos-14` runners (PR #43). + +### Fixed + +- Ten ruff lint errors in snapvec/ and tests/ (unused locals and + ambiguous `l` loop variable) (PR #43). + +### Documentation + +- **Concurrency contract** (`docs/user-guide/concurrency.md`) makes + the single-writer / multi-reader guarantee explicit. Calls out + that `freeze()` (or one warm-up search) is required before + fanning out to reader threads because `SnapIndex.search` + lazily materialises its fp16 centroid cache on the first query + (PR #50, PR #51). +- **Benchmarks page** covers FIQA recall, sqlite-vec scale, the + threading curve, and the competitive comparison (PR #53). +- **README badges**: PyPI version, Python versions, CI, Docs + deployment, license, monthly downloads (PR #43, PR #49). + +### Infrastructure + +- `hypothesis` pulled into `[dev]` optional deps (PR #50). +- `mkdocs`, `mkdocs-material`, `mkdocstrings[python]` pulled into a + new `[docs]` optional extra (PR #49). +- `faiss-cpu`, `hnswlib`, `sqlite-vec` used only by the competitive + bench; **not** added as snapvec runtime dependencies. + ## [0.9.0] -- 2026-04-16 Headline: **5.8x faster search via Cython compiled kernels, diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000..dd4bfe3 --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,60 @@ +cff-version: 1.2.0 +message: "If you use snapvec in your research, please cite it as below." +title: "snapvec: fast compressed approximate nearest-neighbor search" +authors: + - family-names: "Steffens" + given-names: "Jayson" + email: "stffens@gmail.com" +type: software +version: "0.10.0" +date-released: "2026-04-20" +license: MIT +repository-code: "https://github.com/stffns/snapvec" +url: "https://stffns.github.io/snapvec/" +keywords: + - "approximate nearest neighbor" + - "vector search" + - "quantization" + - "product quantization" + - "randomized Hadamard transform" + - "TurboQuant" + - "embeddings" + - "retrieval" +abstract: >- + snapvec is a Python library for fast, compressed approximate + nearest-neighbor search over embedding vectors. It implements + randomized Hadamard transform plus Lloyd-Max scalar quantization + (TurboQuant), product quantization, and IVF-PQ with optional + float16 rerank. Runtime dependencies are limited to NumPy; hot + paths are compiled via Cython and OpenMP. +references: + - type: article + authors: + - family-names: "Zandieh" + given-names: "Amir" + - family-names: "Daliri" + given-names: "Mahdi" + - family-names: "Hadian" + given-names: "Amirhossein" + - family-names: "Mirrokni" + given-names: "Vahab" + title: "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" + year: 2025 + identifiers: + - type: other + value: "arXiv:2504.19874" + - type: article + authors: + - family-names: "Jégou" + given-names: "Hervé" + - family-names: "Douze" + given-names: "Matthijs" + - family-names: "Schmid" + given-names: "Cordelia" + title: "Product Quantization for Nearest Neighbor Search" + journal: "IEEE Transactions on Pattern Analysis and Machine Intelligence" + year: 2011 + volume: 33 + number: 1 + start: 117 + end: 128 diff --git a/README.md b/README.md index 6bfe873..c2aea16 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,10 @@ quantization library, but the design constraints (NumPy-only base install, predictable latency, reproducible index files) come from vstash's local-first requirements. +## Roadmap + +See [ROADMAP.md](ROADMAP.md) for planned work and explicit non-goals. + ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for dev setup, the test matrix, diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..0475822 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,83 @@ +# snapvec roadmap + +Planned work and non-goals for upcoming releases. Items here are +intent, not commitments -- dates slip, priorities change. Track +open work in [issues](https://github.com/stffns/snapvec/issues). + +## v0.11 (target: June 2026) + +### Streaming ingest + +- **`add_batch` resort cost**: IVFPQSnapIndex currently re-sorts the + entire corpus by cluster id on every `add_batch` call (O(N log N) + per append). Acceptable for the documented bulk-ingest-then-search + pattern, painful for streaming. Plan: append-only layout with a + periodic compaction phase. + +- **`early_stop` for IVF probing**: short-circuit the probe loop once + the top-k boundary can no longer be beaten by remaining clusters + (using the coarse-centroid score as a cluster upper bound). + Benchmark already scaffolded in `experiments/bench_ivf_pq_early_stop.py`. + +### Recall + +- **OPQ rotation** for PQSnapIndex / IVFPQSnapIndex: learn a rotation + during `fit()` that minimises PQ reconstruction error (Ge et al., + 2013). Typically +1-2 pp recall at the same bytes/vec. + +### Types + +- **mypy strict in CI as a hard gate**. 17 errors remain, all small; + cleanup is tracked. + +## v0.12 (target: August 2026) + +### File format v2 + +- **Per-block CRC32** in addition to the existing trailer CRC. Catches + corruption earlier in the read path, without reading the whole file. +- **Explicit feature flags** in the magic header so old readers fail + cleanly on files that use a feature they don't support (right now + this surfaces as a generic version mismatch). + +### Incremental updates (delta buffer) + +- **WAL-style append buffer** for `add` / `delete` without rebuilding + the IVFPQ layout. Reads merge the buffer with the base index. + Periodic compaction flushes the buffer into the base. Targets the + pattern where snapvec backs a live system (for example vstash) and + rebuilding on every write is not feasible. + +## v1.0 (target: Q4 2026) + +- **API freeze** across the four index types. Semver kicks in -- + breaking changes require a major version and a documented upgrade + path. +- **Two-minor-version deprecation window** for any public symbol. +- **Reproducible `bench/` suite** that runs in CI with pinned + datasets (BEIR SciFact, FIQA, NFCorpus) and emits machine-readable + JSON results keyed by commit + hardware. +- **Published head-to-head** vs FAISS / ScaNN / hnswlib on a + Linux x86_64 cloud instance (the current bench is macOS-arm64; + ScaNN only ships Linux wheels). + +## Explicit non-goals + +- **GPU backend.** The design point is "no special hardware, laptop + latency". A GPU path would invert that. +- **Billion-scale indices.** `IVFPQSnapIndex` is tested to 1M and + designed for local-first RAG / memory systems, not web-scale search. + If you need billion-scale, use FAISS or Milvus. +- **Approximate hybrid retrieval.** snapvec is a pure vector-ANN + library. For hybrid text + vector retrieval with RRF fusion, use + [vstash](https://github.com/stffns/vstash). +- **Matching FAISS's BLAS-heavy `fit()` speed.** snapvec's `fit()` + is ~7x slower than FAISS IVFPQ at matched config because FAISS + hooks into heavily optimised BLAS. Closing this gap is not a + goal on its own; it's a sub-goal of the broader "depend on NumPy + only" design constraint. + +## Contributing + +See [CONTRIBUTING.md](CONTRIBUTING.md). If you want to work on an +item listed here, open an issue first so we can scope it together. diff --git a/pyproject.toml b/pyproject.toml index b0d855e..0f000a5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "snapvec" -version = "0.9.0" +version = "0.10.0" description = "Fast compressed ANN search via randomized Hadamard transform and optimal Gaussian scalar quantization. NumPy + Cython compiled kernels." readme = "README.md" license = { text = "MIT" } diff --git a/snapvec/__init__.py b/snapvec/__init__.py index 118d273..2923e3b 100644 --- a/snapvec/__init__.py +++ b/snapvec/__init__.py @@ -19,7 +19,7 @@ from ._residual import ResidualSnapIndex from ._rotation import padded_dim, rht -__version__ = "0.9.0" +__version__ = "0.10.0" __all__ = [ "SnapIndex", "PQSnapIndex", From 5babfbe79a9cf6feaf6271ecfc685b1141fcef55 Mon Sep 17 00:00:00 2001 From: Jayson Steffens Date: Mon, 20 Apr 2026 14:36:33 +0200 Subject: [PATCH 2/3] docs: fix contradictory 0.10.0 headline The headline claimed 'no library behaviour changes' but the Changed section right below lists one: SnapIndex.search now rejects k < 1 with ValueError instead of silently returning all results. Rewrite the headline to name the one behaviour change explicitly so the reader does not have to reconcile the contradiction themselves. --- CHANGELOG.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6966a5a..f188701 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,11 @@ the project uses [Semantic Versioning](https://semver.org/). ## [0.10.0] -- 2026-04-20 -Headline: **professionalisation release.** No library behaviour -changes; every PR landed between `v0.9.0` and this tag wires up the -infrastructure, tests, documentation, and benchmarks that make snapvec -ready for third-party adoption and publishable comparisons. +Headline: **professionalisation release.** The only library change +is a narrow fix in `SnapIndex.search` (reject `k < 1` with a clear +error, matching the other three index types); everything else wires +up the infrastructure, tests, documentation, and benchmarks that make +snapvec ready for third-party adoption and publishable comparisons. ### Added From 8cb2df1cd60bb2b5e1de6f1a2789031fd8eead02 Mon Sep 17 00:00:00 2001 From: Jayson Steffens Date: Mon, 20 Apr 2026 14:47:43 +0200 Subject: [PATCH 3/3] docs: clarify CI test matrix vs wheel build matrix in 0.10.0 notes The previous wording blurred two distinct matrices. CI tests run on 3.10, 3.12, 3.13. cibuildwheel emits wheels for 3.10 through 3.13 -- Python 3.11 wheels exist even though there is no 3.11 test job. Spell both out so the release notes match the PyPI artifacts. --- CHANGELOG.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f188701..8afeafc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,9 +16,11 @@ snapvec ready for third-party adoption and publishable comparisons. ### Added -- **CI matrix** across Linux x86_64/aarch64, macOS-13 / macOS-14, and - Windows on CPython 3.10, 3.12, 3.13 (PR #43). Pre-compiled wheels - via `cibuildwheel` + PyPI trusted publishing on `v*` tags. +- **CI test matrix** on Linux, macOS-14, and Windows across CPython + 3.10, 3.12, 3.13 (PR #43). **Pre-compiled wheels** produced by + `cibuildwheel` cover Linux x86_64/aarch64, macOS-13 x86_64, + macOS-14 arm64, and Windows AMD64 across CPython 3.10, 3.11, 3.12, + 3.13, published to PyPI on `v*` tags via trusted publishing. - **Documentation site** at built from MkDocs Material + `mkdocstrings`. Eighteen pages: getting started, one user-guide per index type, architecture, benchmarks,