Release 2.6.2: correctness, security, and robustness hardening - #18
Merged
Conversation
- fix find_ids_without_metadata_key to match literal top-level keys via json_each (dotted keys were misread as JSON paths by json_extract) - fix empty k-means clusters on load_cluster; guard hdbscan + sample_size - perf: BLAS nearest-centroid assignment, SQLite unassigned-id lookup, bounded ancestor-walk cycle detection - docs/README: correct storage layout to SQLite .db + per-collection .usearch sidecar (no longer a single file) - bump version to 2.6.2 and update CHANGELOG
…ering, filters, integrations - rebuild_index: re-open the intact on-disk index when a rebuild fails after the live index is closed; run the HNSW build without the shared DB lock and fold in writes that land during the build before the atomic swap - catalog: release the write lock if connection.__enter__ raises (was leaked) - search: metric-aware MMR relevance for l2 so large distances no longer swamp the diversity term - search: similarity_search_batch routes filtered/text batches to the per-query path so it fills k and accepts text queries - clustering: cluster_vectors raises a clear error when n_clusters exceeds the vector count; cluster() caps to the sampled count when sample_size is set - catalog/utils: metadata filter keys match literal top-level keys via a quoted JSON path; reject keys containing a double-quote - usearch: get() unpacks BIT-quantized bytes to ±1 floats instead of returning packed bytes of the wrong shape - async: increment_metadata returns int (1/0), matching the sync API - langchain: add metric-aware _select_relevance_score_fn so relevance-score retrievers work; document similarity_search_with_score as raw distance - llamaindex: map filter comparison operators and raise on OR/NOT/unsupported conditions instead of silently degrading to equality/AND
… salt fallback - embeddings server: add an ASGI middleware that rejects request bodies larger than the server's own accept limits before they are buffered/parsed, closing an unauthenticated memory-exhaustion vector ([server] extra, network exposure); cap derived from EMBEDDING_SERVER_MAX_REQUEST_ITEMS, override via EMBEDDING_SERVER_MAX_BODY_BYTES - encryption: log a warning when a resource has no salt sidecar and falls back to the legacy shared salt (was silent); document that the modern raw-key path derives at-rest strength from 600k-iter PBKDF2, not SQLCipher's internal KDF
- catalog: malformed FTS5 keyword queries raise ValueError instead of a raw sqlite OperationalError; create the cluster-state table eagerly so a rolled-back first save_cluster cannot leave it desynced - config: tolerate a non-integer EMBEDDING_BATCH_SIZE / MAX_REQUEST_ITEMS env value (warn + fallback) instead of crashing every import - core: vacuum() runs under the DB lock; a failed index add after the catalog commit is logged (catalog/index divergence visibility) - search: hybrid search applies the Python metadata filter on the keyword side too, for SQL/Python parity - encryption: key the PBKDF2 cache by a salted hash rather than retaining raw passphrase bytes; warn (do not silently downgrade) on legacy-salt fallback; document that the raw-key path derives strength from 600k-iter PBKDF2 - logging: swap handlers in one assignment (no zero-handler window) - integrations: add LangChain asimilarity_search_with_score (offloaded); LlamaIndex delete warns instead of silently no-op'ing on a miss - usearch: fix remove() docstring to reference VectorCollection.rebuild_index()
…nt, FTS5 catch) - embeddings server: guard EMBEDDING_SERVER_MAX_BODY_BYTES env parsing (no longer crashes import on a bad value) and floor the cap at 1 MiB so a 0/low override can't reject every request - core: rebuild delta catch-up counts only docs actually indexed (add_pairs), and warns when concurrent adds during the build lack stored embeddings and are skipped (was silently dropping them and inflating the count) - catalog: FTS5 error translation no longer matches a bare 'syntax error' (only fts5/unterminated/malformed), so an unrelated SQL error isn't mislabeled as an invalid user query - tests: add a bad/zero EMBEDDING_SERVER_MAX_BODY_BYTES import-guard test and an end-to-end LangChain relevance-score test (validates [0,1] from real search)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…CodeQL alert The #14a change hashed the passphrase with SHA256 for the cache lookup key, which CodeQL flags as weak password hashing. CodeQL's suggested PBKDF2 fix is a trap: it would run a 600k-iteration KDF on every cache lookup, defeating the cache (whose only purpose is to AVOID re-running that KDF). The real key derivation already uses 600k-iter PBKDF2; the cache key is just a process-local, LRU-bounded lookup index. Revert to the original tuple key (no hashing, no alert) and document why we don't hash here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships SimpleVecDB 2.6.2 — the clustering/hierarchy correctness + performance work, plus a review-driven hardening pass across correctness, security, integrations, and robustness. No schema changes; existing databases are unaffected.
Highlights
Correctness
rebuild_indexno longer bricks a collection on failure, and runs the HNSW build without holding the shared DB lock (concurrent writes during the build are folded into the new index before the atomic swap)l2;similarity_search_batchfillskunder filters and accepts text queriesn_clusters > vectors; metadata filter keys match literal top-level keys (not nested JSON paths)get()unpacks correctly;async increment_metadatanow returnsintSecurity (
[server]/[encryption]extras)Integrations
_select_relevance_score_fnso relevance-score /score_thresholdretrievers work; asyncsimilarity_search_with_scoreoffloadOR/NOTconditions raise instead of silently degrading to equality/ANDRobustness
ValueError; config tolerates malformed int env values;vacuum()under lock; eager cluster-state table (rollback-safe); atomic logging handler swapTesting
tests/unit/test_tier{1,2,3,4}_fixes.py)See
CHANGELOG.md[2.6.2] for the full list.