Release 2.6.2: correctness, security, and robustness hardening

.gitignore

@@ -68,3 +68,7 @@ simplevecdb_plan.md
 AGENTS.md
 NEXT_UPDATES.md
 pro_pack/
+
+
+# Local notes
+IMPORTANT.md

CHANGELOG.md

@@ -5,6 +5,112 @@ All notable changes to SimpleVecDB will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.6.2] - 2026-06-06
+
+### Correctness and contract fixes
+
+Hardening of the index-rebuild, search, clustering, and integration layers
+surfaced by a code review. Two intentional behavior changes are noted under
+“Changed”.
+
+#### Fixed
+
+- **`rebuild_index` no longer bricks a collection on failure** — if building or
+  swapping the new HNSW index raises after the live index is closed, the
+  collection re-opens the intact on-disk index instead of holding a closed one.
+- **Catalog write lock released on connection error** — a raising
+  `connection.__enter__` no longer leaks the catalog lock (which could deadlock
+  the database).
+- **Max-Marginal-Relevance respects the distance metric** — MMR on `l2`
+  collections used a cosine-specific relevance formula that swamped the
+  diversity term; it now uses a bounded, metric-appropriate relevance.
+- **`similarity_search_batch` fills `k` under filters and accepts text queries**
+  — large filtered batches no longer silently under-deliver, and a text query in
+  a large batch behaves the same as in a small one.
+- **Clustering handles impossible `n_clusters`** — `ClusterEngine.cluster_vectors`
+  raises a clear error when `n_clusters` exceeds the number of vectors;
+  `Collection.cluster()` caps `n_clusters` to the number of vectors actually
+  clustered (the sample when `sample_size` is set, fixing a latent error when
+  `n_clusters > sample_size`).
+- **Metadata filter keys match literally** — a filter key containing a dot
+  (e.g. `{"a.b": x}`) now matches the literal top-level key `a.b` instead of the
+  nested JSON path `a → b`, consistent with the Python filter path. Keys
+  containing a double-quote are rejected.
+- **BIT-quantized vector retrieval unpacks correctly** — `UsearchIndex.get()`
+  (used by the MMR fallback) returned packed bytes for BIT indexes instead of
+  the unpacked ±1 float vectors; it now unpacks them.
+- **`rebuild_index` no longer blocks the database during the HNSW build** — the
+  expensive build runs without the shared lock (held only to snapshot and swap);
+  writes that land during the build are folded into the new index before the
+  swap.
+- **Embedding server caps request body size** — an ASGI middleware rejects
+  request bodies larger than the server's own accept limits before they are
+  buffered/parsed, closing an unauthenticated memory-exhaustion vector (only
+  relevant with the `[server]` extra exposed on a network). A missing
+  encryption salt sidecar now logs a warning instead of silently falling back
+  to the shared legacy salt.
+- **Robustness pass** — malformed FTS5 keyword queries raise `ValueError` instead
+  of a raw SQLite error; the cluster-state table is created eagerly so a
+  rolled-back first `save_cluster` cannot desync it; a non-integer
+  `EMBEDDING_BATCH_SIZE`/`EMBEDDING_SERVER_MAX_REQUEST_ITEMS` env value warns and
+  falls back instead of crashing import; `vacuum()` holds the DB lock; a failed
+  index add after the catalog commit is logged (divergence visibility); hybrid
+  search applies the Python metadata filter on the keyword side too (SQL/Python
+  parity); `logging.configure_logging` swaps handlers atomically.
+- **LangChain `asimilarity_search_with_score`** offloads to a thread instead of
+  blocking the event loop.
+
+#### Changed
+
+- **`AsyncVectorCollection.increment_metadata` now returns `int`** (1 if the row
+  existed and was updated, 0 otherwise), matching the synchronous API; it
+  previously discarded the value and returned `None`.
+- **LlamaIndex metadata filters fail loudly on unsupported shapes** — the
+  `SimpleVecDBLlamaStore` adapter now maps comparison operators
+  (`$gt/$gte/$lt/$lte/$ne/$in/$nin`) instead of silently treating them as
+  equality, and raises `NotImplementedError` for `OR`/`NOT` conditions and
+  unsupported operators rather than returning wrong results.
+- **LangChain relevance scoring now works** — `SimpleVecDBVectorStore` implements
+  `_select_relevance_score_fn`, so `similarity_search_with_relevance_scores` and
+  `as_retriever(search_type="similarity_score_threshold")` return metric-aware
+  `[0, 1]` relevance (higher = better). `similarity_search_with_score` still
+  returns the raw distance (FAISS/Chroma convention), now documented as such.
+
+### Clustering and hierarchy fixes
+
+Internal correctness and performance work on the clustering and hierarchy
+layers. No public API changes; existing databases are unaffected.
+
+#### Fixed
+
+- **`load_cluster` survives empty k-means clusters** — when k-means leaves a
+  requested cluster empty (common with duplicate vectors or `n_clusters` near
+  the number of distinct points), the stored `n_clusters` is smaller than the
+  number of centroid rows. The centroid reshape now derives its row count from
+  the stored buffer rather than `n_clusters`, which previously raised
+  `ValueError` on load.
+- **`assign_to_cluster` matches metadata keys literally** — a `metadata_key`
+  containing `.` or `[` is now matched as a literal top-level key (via
+  `json_each`) instead of being misread as a nested JSON path, which had caused
+  every already-assigned document to be re-assigned on each call.
+- **`cluster(algorithm="hdbscan", sample_size=…)` raises instead of silently
+  dropping documents** — HDBSCAN produces no centroids, so out-of-sample
+  documents cannot be assigned. The combination now raises a clear `ValueError`
+  rather than clustering only the sample.
+
+#### Performance
+
+- **BLAS-backed out-of-sample centroid assignment** — nearest-centroid
+  assignment uses the `‖c‖² − 2·x·c` expansion (a single matmul) instead of
+  materialising the dense `(n_vectors, n_centroids, dim)` broadcast temporary
+  that could exhaust memory on large collections.
+- **Unassigned-id lookup pushed into SQLite** — `assign_to_cluster(doc_ids=None)`
+  finds documents lacking the cluster key with one `json_each` query instead of
+  loading and JSON-parsing every row's text and metadata.
+- **Bounded ancestor-walk for cycle detection** — `set_parent` detects
+  parent/child cycles by walking the ancestor chain with a depth-bounded
+  recursive CTE instead of materialising the entire descendant subtree.
+
 ## [2.6.1] - 2026-05-10
 
 ### Storage, mutation, and eventing improvements

README.md

@@ -7,17 +7,17 @@
 
 <a href='https://ko-fi.com/U7U01WTJF9' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://storage.ko-fi.com/cdn/kofi3.png?v=6' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>
 
-**The dead-simple, local-first vector database.**
+**A local-first, embedded vector database backed by SQLite and usearch.**
 
-SimpleVecDB brings **Chroma-like simplicity** to a single **SQLite file**. Built on `usearch` HNSW indexing, it offers high-performance vector search, quantization, and zero infrastructure headaches. Perfect for local RAG, offline agents, and indie hackers who need production-grade vector search without the operational overhead.
+SimpleVecDB pairs **Chroma-like ergonomics** with a **file-based** store — a SQLite database for metadata and text alongside a `usearch` HNSW index per collection. It provides high-performance vector search, quantization, and hybrid retrieval with no separate services to run. It fits local RAG pipelines, offline agents, and any application that needs production-grade vector search without the operational overhead of a hosted database.
 
 ## Why SimpleVecDB?
 
-- **Zero Infrastructure** — Just a `.db` file. No Docker, no Redis, no cloud bills.
-- **Blazing Fast** — 10-100x faster search via usearch HNSW. Adaptive: brute-force for <10k vectors (perfect recall), HNSW for larger collections.
-- **Truly Portable** — Runs anywhere SQLite runs: Linux, macOS, Windows, even WASM.
-- **Async Ready** — Full async/await support with optional executor injection for thread-safe ONNX/usearch sharing.
-- **Batteries Included** — Optional FastAPI embeddings server + LangChain/LlamaIndex integrations via `[integrations]` extra.
+- **Zero Infrastructure** — Local files on disk: a SQLite database plus a `usearch` index. No Docker, no Redis, no external services.
+- **High Performance** — usearch HNSW indexing with adaptive search: brute-force under 10k vectors (perfect recall), HNSW above that.
+- **Portable** — Runs anywhere SQLite runs: Linux, macOS, Windows, and WASM.
+- **Async Support** — A complete async/await surface with optional executor injection for thread-safe ONNX/usearch sharing.
+- **Integrations Included** — Optional FastAPI embeddings server and LangChain/LlamaIndex adapters via the `[integrations]` extra.
 - **Production Ready** — Hybrid search (BM25 + vector), metadata filtering, multi-collection support, and automatic hardware acceleration.
 
 ### When to Choose SimpleVecDB
@@ -67,7 +67,7 @@ pip install "simplevecdb[server]"
 **Verify Installation:**
 
 ```bash
-python -c "from simplevecdb import VectorDB; print('SimpleVecDB installed successfully!')"
+python -c "import simplevecdb; print(simplevecdb.__version__)"
 ```
 
 ## Quickstart
@@ -157,8 +157,8 @@ in the [Setup Guide](ENV_SETUP.md).
 
 ### Option 3: LangChain or LlamaIndex
 
-Already wired into one of the big RAG frameworks? Drop SimpleVecDB in
-as the vector store:
+Already using one of the major RAG frameworks? Use SimpleVecDB as the
+vector store:
 
 ```bash
 pip install "simplevecdb[integrations]"
@@ -208,7 +208,7 @@ A few of the things SimpleVecDB does well — see
 - **Quantization** — `FLOAT32`, `FLOAT16`, `INT8`, `BIT` for 1×–32×
   compression.
 - **Multi-collection + cross-collection search** — isolated namespaces in
-  one `.db` file, with merged ranked search across them.
+  one database, with merged ranked search across them.
 - **Mongo-style filters** — `$eq $ne $gt $gte $lt $lte $in $nin $exists
   $between` on metadata, edges, and events.
 - **Memory primitives (v2.6.1)** — pending-vector buffer with atomic
@@ -300,7 +300,7 @@ ideas in [GitHub Discussions](https://github.com/coderdayton/simplevecdb/discuss
 
 ## Contributing
 
-Contributions are welcome! Whether you're fixing bugs, improving documentation, or proposing new features:
+Contributions are welcome — bug fixes, documentation improvements, and new feature proposals alike:
 
 1. Read [CONTRIBUTING.md](CONTRIBUTING.md) for development setup
 2. Check existing [Issues](https://github.com/coderdayton/simplevecdb/issues) and [Discussions](https://github.com/coderdayton/simplevecdb/discussions)
@@ -318,12 +318,12 @@ Contributions are welcome! Whether you're fixing bugs, improving documentation,
 - [GitHub Releases](https://github.com/coderdayton/simplevecdb/releases) — Changelog and updates
 - [Examples Gallery](https://coderdayton.github.io/SimpleVecDB/examples/) — Community-contributed notebooks
 
-## Other Ways to Support
+## Support the Project
 
-- ☕ **[Buy me a coffee](https://ko-fi.com/xbbvii)** - One-time donation
-- ⭐ **Star the repo** - Helps with visibility
-- 🐛 **Report bugs** - Improve the project for everyone
-- 📝 **Contribute** - See [CONTRIBUTING.md](CONTRIBUTING.md)
+- **[Sponsor on Ko-fi](https://ko-fi.com/xbbvii)** — one-time donations
+- **Star the repository** — helps with visibility
+- **[Report issues](https://github.com/coderdayton/simplevecdb/issues)** — bug reports and feedback
+- **[Contribute](CONTRIBUTING.md)** — development setup and guidelines
 
 ## License
 

docs/CHANGELOG.md

@@ -5,6 +5,112 @@ All notable changes to SimpleVecDB will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.6.2] - 2026-06-06
+
+### Correctness and contract fixes
+
+Hardening of the index-rebuild, search, clustering, and integration layers
+surfaced by a code review. Two intentional behavior changes are noted under
+“Changed”.
+
+#### Fixed
+
+- **`rebuild_index` no longer bricks a collection on failure** — if building or
+  swapping the new HNSW index raises after the live index is closed, the
+  collection re-opens the intact on-disk index instead of holding a closed one.
+- **Catalog write lock released on connection error** — a raising
+  `connection.__enter__` no longer leaks the catalog lock (which could deadlock
+  the database).
+- **Max-Marginal-Relevance respects the distance metric** — MMR on `l2`
+  collections used a cosine-specific relevance formula that swamped the
+  diversity term; it now uses a bounded, metric-appropriate relevance.
+- **`similarity_search_batch` fills `k` under filters and accepts text queries**
+  — large filtered batches no longer silently under-deliver, and a text query in
+  a large batch behaves the same as in a small one.
+- **Clustering handles impossible `n_clusters`** — `ClusterEngine.cluster_vectors`
+  raises a clear error when `n_clusters` exceeds the number of vectors;
+  `Collection.cluster()` caps `n_clusters` to the number of vectors actually
+  clustered (the sample when `sample_size` is set, fixing a latent error when
+  `n_clusters > sample_size`).
+- **Metadata filter keys match literally** — a filter key containing a dot
+  (e.g. `{"a.b": x}`) now matches the literal top-level key `a.b` instead of the
+  nested JSON path `a → b`, consistent with the Python filter path. Keys
+  containing a double-quote are rejected.
+- **BIT-quantized vector retrieval unpacks correctly** — `UsearchIndex.get()`
+  (used by the MMR fallback) returned packed bytes for BIT indexes instead of
+  the unpacked ±1 float vectors; it now unpacks them.
+- **`rebuild_index` no longer blocks the database during the HNSW build** — the
+  expensive build runs without the shared lock (held only to snapshot and swap);
+  writes that land during the build are folded into the new index before the
+  swap.
+- **Embedding server caps request body size** — an ASGI middleware rejects
+  request bodies larger than the server's own accept limits before they are
+  buffered/parsed, closing an unauthenticated memory-exhaustion vector (only
+  relevant with the `[server]` extra exposed on a network). A missing
+  encryption salt sidecar now logs a warning instead of silently falling back
+  to the shared legacy salt.
+- **Robustness pass** — malformed FTS5 keyword queries raise `ValueError` instead
+  of a raw SQLite error; the cluster-state table is created eagerly so a
+  rolled-back first `save_cluster` cannot desync it; a non-integer
+  `EMBEDDING_BATCH_SIZE`/`EMBEDDING_SERVER_MAX_REQUEST_ITEMS` env value warns and
+  falls back instead of crashing import; `vacuum()` holds the DB lock; a failed
+  index add after the catalog commit is logged (divergence visibility); hybrid
+  search applies the Python metadata filter on the keyword side too (SQL/Python
+  parity); `logging.configure_logging` swaps handlers atomically.
+- **LangChain `asimilarity_search_with_score`** offloads to a thread instead of
+  blocking the event loop.
+
+#### Changed
+
+- **`AsyncVectorCollection.increment_metadata` now returns `int`** (1 if the row
+  existed and was updated, 0 otherwise), matching the synchronous API; it
+  previously discarded the value and returned `None`.
+- **LlamaIndex metadata filters fail loudly on unsupported shapes** — the
+  `SimpleVecDBLlamaStore` adapter now maps comparison operators
+  (`$gt/$gte/$lt/$lte/$ne/$in/$nin`) instead of silently treating them as
+  equality, and raises `NotImplementedError` for `OR`/`NOT` conditions and
+  unsupported operators rather than returning wrong results.
+- **LangChain relevance scoring now works** — `SimpleVecDBVectorStore` implements
+  `_select_relevance_score_fn`, so `similarity_search_with_relevance_scores` and
+  `as_retriever(search_type="similarity_score_threshold")` return metric-aware
+  `[0, 1]` relevance (higher = better). `similarity_search_with_score` still
+  returns the raw distance (FAISS/Chroma convention), now documented as such.
+
+### Clustering and hierarchy fixes
+
+Internal correctness and performance work on the clustering and hierarchy
+layers. No public API changes; existing databases are unaffected.
+
+#### Fixed
+
+- **`load_cluster` survives empty k-means clusters** — when k-means leaves a
+  requested cluster empty (common with duplicate vectors or `n_clusters` near
+  the number of distinct points), the stored `n_clusters` is smaller than the
+  number of centroid rows. The centroid reshape now derives its row count from
+  the stored buffer rather than `n_clusters`, which previously raised
+  `ValueError` on load.
+- **`assign_to_cluster` matches metadata keys literally** — a `metadata_key`
+  containing `.` or `[` is now matched as a literal top-level key (via
+  `json_each`) instead of being misread as a nested JSON path, which had caused
+  every already-assigned document to be re-assigned on each call.
+- **`cluster(algorithm="hdbscan", sample_size=…)` raises instead of silently
+  dropping documents** — HDBSCAN produces no centroids, so out-of-sample
+  documents cannot be assigned. The combination now raises a clear `ValueError`
+  rather than clustering only the sample.
+
+#### Performance
+
+- **BLAS-backed out-of-sample centroid assignment** — nearest-centroid
+  assignment uses the `‖c‖² − 2·x·c` expansion (a single matmul) instead of
+  materialising the dense `(n_vectors, n_centroids, dim)` broadcast temporary
+  that could exhaust memory on large collections.
+- **Unassigned-id lookup pushed into SQLite** — `assign_to_cluster(doc_ids=None)`
+  finds documents lacking the cluster key with one `json_each` query instead of
+  loading and JSON-parsing every row's text and metadata.
+- **Bounded ancestor-walk for cycle detection** — `set_parent` detects
+  parent/child cycles by walking the ancestor chain with a depth-bounded
+  recursive CTE instead of materialising the entire descendant subtree.
+
 ## [2.6.1] - 2026-05-10
 
 ### Storage, mutation, and eventing improvements

docs/ENV_SETUP.md

@@ -53,6 +53,7 @@ Configuration for `simplevecdb-server`.
 | `SERVER_HOST`                        | Host to bind the server to.                                                    | `0.0.0.0`                                 |
 | `SERVER_PORT`                        | Port to bind the server to.                                                    | `53287` (Code default) / `8000` (Example) |
 | `EMBEDDING_SERVER_MAX_REQUEST_ITEMS` | Max number of prompts allowed per `/v1/embeddings` request (protects latency). | `max(32, EMBEDDING_BATCH_SIZE)`           |
+| `EMBEDDING_SERVER_MAX_BODY_BYTES`    | Max raw request body size in bytes; larger bodies are rejected (413) before being buffered/parsed, preventing memory exhaustion. | _Derived from the request-item and text-length limits (min 1 MiB)_ |
 | `EMBEDDING_SERVER_API_KEYS`          | Comma-separated API keys to require `Authorization: Bearer`/`X-API-Key`.       | _Disabled (unauthenticated)_              |
 
 When `EMBEDDING_SERVER_API_KEYS` is set, SimpleVecDB also tracks request counts and token usage per key. Call `GET /v1/usage` with the same key to retrieve your stats.

docs/Features.md

@@ -6,8 +6,9 @@ release-by-release detail, see the [Changelog](CHANGELOG.md).
 
 ## Storage & schema
 
-- **Single-file SQLite** — one `.db` file (or `:memory:`) holds everything:
-  documents, vectors, FTS5 index, edges, events, TTL, clusters.
+- **File-based storage** — a `.db` file (or `:memory:`) holds documents, the
+  FTS5 index, edges, events, TTL, and clusters; vectors live in a per-collection
+  `.usearch` HNSW index file alongside it.
 - **Multi-collection** — isolated namespaces per database via
   `db.collection("name")`. Each collection has its own quantization,
   distance metric, and (optional) embedding storage.