Release 2.6.0: durability, security, and integration hardening - #16
Merged
Conversation
Top-priority correctness and durability fixes from the 2.5.0 review: - UsearchIndex.save: write to .tmp, fsync, os.replace, fsync dir; move dirty check inside _write_lock so concurrent add can't have its dirty flag silently cleared. - rebuild_index: build at sibling .rebuild path, then atomic os.replace onto the live path; old index remains canonical until swap succeeds. - Encrypted save: encrypt_file/decrypt_file write atomically via tmp + fsync + os.replace + chmod 0o600; encrypt_index_file only unlinks plaintext after the encrypted output is durably on disk. - VectorDB-level RLock shared with every CatalogManager: serializes the shared sqlite3.Connection's Python-level transaction context (WAL alone is not enough — `with self.conn:` two-thread interleave); also closes the collection() check-then-insert TOCTOU. - AsyncVectorDB.close: shutdown(wait=True) drains in-flight pool tasks before closing the SQLite connection. - set_parent: cycle check + UPDATE in one critical section. - Cluster persistence: replace bare conn.commit() with `with self._lock, self.conn:` so failed executes are properly rolled back. - add_documents: replace last_insert_rowid arithmetic (wrong for batches mixing explicit and None IDs because UPSERT doesn't advance the auto-increment) with INSERT ... RETURNING id for auto-ID rows. - delete_collection: close cached UsearchIndex objects before unlinking the file so stale mmap views can't race. - FTS shadow methods renamed to _upsert_fts_rows / _delete_fts_rows; must run inside the same transaction as the main table. - file_lock: nested try/finally so fd.close() always runs even if LOCK_UN raises; .lock siblings unlinked on release so they don't accumulate. - In-memory DB: ephemeral usearch tempfiles tracked and cleaned up on close. - API key auth: hmac.compare_digest constant-time comparison. - LangChain VectorStore.embeddings property override added. - python-dotenv declared as a dependency (was used at config import).
Second batch of 2.5.0 review findings: Correctness - hybrid_search RRF dedupes by document id instead of page_content; two documents with identical text are no longer silently merged. - add_texts / add_texts_streaming reject NaN/Inf input vectors before feeding them to HNSW. - normalize_l2 uses < 1e-12 instead of == 0 so subnormal floats don't blow up on division. - silhouette_score caps the evaluation sample at SILHOUETTE_MAX_SAMPLE (10_000) so large collections don't OOM on the O(n²) computation. - LlamaIndex node IDs persisted into metadata under _simplevecdb_node_id so delete() works after a process restart and so query results carry stable IDs (no more randomized Python hash()). - AsyncVectorDB.collection accepts store_embeddings so async callers can enable embedding storage and rebuild_index(). - _parse_bool_env treats empty string as unset (default) instead of truthy. Perf - MMR maintains the selected-vectors matrix incrementally with vstack instead of np.stack-ing the growing list every iteration. Security - API key comparison uses hmac.compare_digest (constant-time). - SQLCipher PRAGMA key always goes through _normalize_key + x'hex' form; no user-supplied passphrase characters are interpolated into the PRAGMA's quoted argument. - is_database_encrypted rejects zero-byte files. LangChain integration - aadd_texts / asimilarity_search / amax_marginal_relevance_search use asyncio.to_thread so they don't block the event loop. Tooling - ruff target-version and mypy python_version aligned with requires-python>=3.10. Cleaned three F401 unused-import warnings exposed by the change.
Round 3-5 of 2.5.0 review findings:
API surface
- Export ClusterResult and ClusterTagCallback from simplevecdb (they
were return/arg types of public methods with no public import path).
- AsyncVectorDB.collection accepts store_embeddings — already added in
the previous batch, confirmed parity with VectorDB.collection.
Library hygiene
- Attach NullHandler to the simplevecdb root logger at import time per
the Python logging HOWTO. Idempotent.
- SimpleVecDBLlamaStore.delete_nodes(filters=...) raises
NotImplementedError instead of silently dropping the filter.
- Recursive CTE depth in get_descendants / get_ancestors bound as a
parameter instead of f-string interpolation.
- Config.from_env() docstring clarifies it returns the import-time
frozen instance; setting env vars after import does not refresh.
Server hardening
- ModelRegistry(allow_unlisted=...) defaults to False so programmatic
instances match the secure default of the configured server.
- /v1/usage returns an aggregated total when auth is disabled, instead
of leaking the per-IP usage buckets to any anonymous caller.
- Server enforces EMBEDDING_SERVER_MAX_REQUEST_ITEMS <= _MAX_ENCODE_BATCH
at startup so an out-of-range env var fails fast.
Encryption
- Encrypted file format gains a 3-byte header ('SV' + version byte).
decrypt_file detects the header and falls back to the legacy
format, so existing encrypted indexes still load without
re-encryption.
Packaging / tooling
- pyproject.toml: add [project.urls], classifiers, and keywords.
- .bandit: document the B104 skip and the conditions under which it
must be removed.
Round 6 (hygiene + perf) - Remove dead log_error() (and its test). - VectorDB.__repr__ no longer hits SQL on every call; debuggers and exception formatters that auto-stringify objects don't trigger I/O. - INT8 quantization rejects out-of-range inputs instead of silently clipping and destroying magnitude information. - AsyncVectorCollection.cluster runtime-validates the algorithm string so we can drop the # type: ignore[arg-type] and produce a clear ValueError. - catalog: index on text column so find_ids_by_texts/remove_texts no longer full-scan. - _normalize_key caches its PBKDF2 result keyed by (passphrase, salt) so repeated encrypt/decrypt round-trips don't pay the 480k-iteration cost more than once per process. Round 7 (security defaults) - CORS is opt-in: default no CORS unless EMBEDDING_SERVER_CORS_ORIGINS is set. Explicit origins enable credentials; wildcard drops them so the spec-violating "*" + allow_credentials combo cannot be produced. - HF model loader rejects repo_ids that don't match the canonical "namespace/name" format (blocks path traversal / local-path inputs) and forces trust_remote_code=False so a model card cannot trigger arbitrary downloaded Python on load. - scripts/ removed from .gitignore. The previously-untracked bump_version.py, check_version_sync.py, and track_metrics.py are now on the branch with the version-sync fix from earlier.
Round 8 — C3 per-DB random salt - _resolve_salt() helper looks for a <resource>.salt sidecar; if absent, generates a random 16-byte salt and writes it atomically with mode 0o600. If the sidecar can't be read (legacy resource), falls back to the previous fixed salt so pre-2.6.0 encrypted databases keep opening unchanged. - create_encrypted_connection: salt sidecar generated for new DBs, read for existing ones; legacy DBs (no sidecar) keep using the fixed salt. SQLCipher key derived per-DB. - encrypt_index_file / decrypt_index_file: salt sidecar generated on first encryption, read for re-encrypt and decrypt; legacy files fall back to the fixed salt. - _normalize_key now caches its PBKDF2 result keyed by the salt too, so the per-DB salt does not regress the perf gain from Round 6. - delete_collection unlinks the .salt sidecar alongside the .usearch and .usearch.enc files. - New tests/unit/test_encryption_salt.py covers create/read/legacy fallback paths plus an end-to-end SQLCipher reopen test. Round 9 — LlamaIndex migration + integration tests - SimpleVecDBLlamaStore.migrate_node_id_metadata() walks the underlying collection and stamps _simplevecdb_node_id into metadata for documents inserted before 2.6.0 (where node_id was never persisted). Idempotent. - Integration test for /v1/usage updated to assert the new aggregate _total bucket (the M-4 fix from Round 4 changed the contract). Verification: 549 unit + 23 non-ollama integration tests passing, ruff clean, 7 new salt tests added.
Adds dedicated test coverage for every new behavior introduced on the release/2.6.0 branch. These complement existing unit/integration tests that only happened to exercise some of the changes incidentally. New test files (95 tests total): - tests/unit/test_encryption_v1_format.py (17): v0 backwards-compat decrypt, v1 header roundtrip, _atomic_write_bytes mode 0o600 + parent-creation + no leftover .tmp, _normalize_key per-salt cache keys, encrypt_index_file/decrypt_index_file roundtrip with sidecar. - tests/unit/embeddings/test_repo_id_validation.py (31): _REPO_ID_RE accepts canonical HF IDs, rejects path traversal / absolute paths / control chars / URL forms; load_model forces trust_remote_code=False and validates before snapshot_download is called. - tests/unit/integrations/test_llamaindex_v26.py (8): node_id stamped into _simplevecdb_node_id, delete() falls back to metadata after _id_map is cold, migrate_node_id_metadata() backfills + idempotent + preserves stamped rows, delete_nodes(filters=...) raises. - tests/unit/core/test_v26_safety.py (12): NaN/Inf vectors rejected by add_texts, __repr__ runs no SQL (works after close), _lock is re-entrant, ephemeral usearch files cleaned up on close. - tests/unit/engine/test_v26_quantization_clustering.py (11): INT8 range guard (subnormal pass-through, |x|>1+1e-5 rejected), normalize_l2 < 1e-12 returned unchanged, SILHOUETTE_MAX_SAMPLE constant. - tests/unit/test_async_v26.py (9): AsyncVectorDB.collection accepts store_embeddings (cache key includes it), cluster() validates algorithm at runtime, close() drains owned executor and leaves externally-supplied executors alive. - tests/unit/test_v26_misc.py (10): hybrid_search dedupes by doc id (two distinct docs with identical text remain separate), utils.file_lock cleans up the .lock sibling on release / on exception / under contention, simplevecdb.logging attaches exactly one NullHandler at import (idempotent across reloads). Test count: 561 → 656 unit, 23 → 24 integration. Full suite runs in under 20 seconds with all green.
Round 10 — review findings on top of 2.6.0: Critical - add_texts / add_texts_streaming validate NaN/Inf BEFORE the catalog insert. Previously the SQLite row committed first and a non-finite vector then raised, leaving the row in the catalog with no HNSW entry — invisible to search but visible via document fetches. Important - VectorCollection.__repr__ no longer calls count(); SQL in repr fails with ProgrammingError after close() and trips debuggers / exception formatters that auto-stringify objects. - embeddings/server.py validates EMBEDDING_SERVER_MAX_REQUEST_ITEMS <= _MAX_ENCODE_BATCH at module import, so the guard fires under any ASGI deployment (gunicorn, programmatic uvicorn) — not just the run_server() CLI path. - SimpleVecDBLlamaStore.add now generates a uuid for nodes that arrive with no node_id and stamps it into metadata BEFORE the row insert, so the metadata commit is atomic with the row. Previous code ran a separate UPDATE after add_texts; a crash in the gap left rows un-stampable, and cross-restart delete() silently no-op'd. - catalog.load_cluster_state / list_cluster_states acquire self._lock before SELECT — sqlite3.Connection is not safe for concurrent stmt execution from multiple threads even under WAL. - All bare conn.execute reads in CatalogManager (get_documents_by_ids, get_embeddings_by_ids, get_documents_and_embeddings_by_ids, find_ids_by_*, keyword_search, count, get_all_docs_with_text, check_legacy_sqlite_vec, get_legacy_vectors, get_children, get_parent, get_descendants, get_ancestors) and VectorDB.list_collections now run under self._lock for the same reason. Suggestions - rebuild_index serializes the entire fetch + build + swap on self._lock so concurrent add/delete cannot mutate the catalog mid-rebuild and produce a stale snapshot. - _ensure_cluster_table re-checks _cluster_table_ready inside the lock and sets it inside the with-block; concurrent first-callers no longer both run the DDL. - utils.file_lock opens via os.open(O_CREAT|O_RDWR, 0o600) — no truncation of stale lock files, restricted permissions. Tests - test_v26_safety: NaN/Inf rejection asserts count() == 0 after the raise (single-item, multi-item, streaming variants). - test_v25_correctness: VectorCollection.__repr__ no longer expected to contain size. - test_llamaindex_v26: empty-node_id path stamps a uuid that round-trips through cold-restart delete(). Verification: 659 unit + 24 non-ollama integration tests passing, ruff clean.
- Move the 2.6.0 section from "unreleased" to dated 2026-05-06. - Add a "Fixed (review pass 2)" subsection covering 4ee438b: NaN/Inf orphan-row, repr SQL, ASGI startup guard, LlamaIndex atomic node_id, catalog read locking, serialized rebuild_index, _ensure_cluster_table re-check, and file_lock no-truncate.
Review pass 3 closed nine critical and five important issues across durability, encryption, integrations, and search: - usearch_index: _dirty=False inside file_lock; data fsync uses O_RDWR. - encryption: PBKDF2 600k iters; v1 header bound into AES-GCM AAD; bounded LRU normalize-key cache; salt sidecar created with O_EXCL (concurrent openers converge); v0->v1 re-encryption now creates a fresh sidecar. - core: _rebuild_index_locked routes through CatalogManager.list_all_ids instead of bare conn.execute; delete_collection existence check moved inside the shared RLock to close the TOCTOU. - catalog: defensive RuntimeError if any -1 sentinel rowid leaks into _upsert_fts_rows. - search: hybrid-search RRF rank now symmetric between vector and keyword candidates under metadata filter (uses original HNSW position). - llamaindex: delete() no longer swallows sqlite3.DatabaseError; __init__ emits a one-shot DeprecationWarning for v2.5-shaped collections; migration helper documents the rowid-as-node-id limit. - quantization: INT8 out-of-range softened from raise to one-shot DeprecationWarning + clip, preserving prior callers. - scripts/check_version_sync.py validates CHANGELOG.md matches pyproject.toml so a release cannot ship with a stale changelog. Tests: 36 new regression tests pin the new invariants (parent-dir fsync, .tmp cleanup on failure, shared RLock identity, adversarial table names, RRF symmetry, nonce uniqueness, wrong-key no-output, header AAD tamper detection, llamaindex query round-trip, migrate-then-delete, legacy-collection warning, DatabaseError propagation). 695 passed, 2 skipped.
Replaces .pre-commit-config.yaml with lefthook.yml. Splits the prior
single-stage hook chain by cost:
- pre-commit (fast feedback): version-sync, ruff --fix with auto-restage.
- pre-push (gating): version-sync, mypy, bandit, full pytest+coverage.
Run `lefthook install` after pulling to wire up .git/hooks/{pre-commit,pre-push}.
mypy: - core.py: assert non-None at the one call site that always has a fallback (UsearchIndex index_path). - core.py: streaming-batch placeholder type widened to list[Sequence[float] | None] across the chain; _process_streaming_batch narrows back before persistence. - embeddings/server.py: targeted type:ignore on the unreachable list[int] branch in _normalize_input. Lint hygiene (ruff): - Removed unused locals (original_search, collection, sklearn, original_save, key) flagged by F841. - Added missing `from typing import Any` (F821) to tests/unit/embeddings/test_v25_enhancements.py. - Added `# noqa: E402` after pytest.importorskip()-gated imports in the new review-pass-3 test files and in pre-existing places. - Auto-fixes from `ruff --fix` applied across legacy test modules (28 issues, mostly stale imports and trailing whitespace). Docs: - docs/CHANGELOG.md synced from root CHANGELOG.md. - docs/api/encryption.md updated for 2.6.0 review-pass-3 reality: 600k PBKDF2 iters (OWASP 2024), AAD-bound v1 header, per-DB random salt sidecar with O_EXCL guard, atomic save semantics. Tests: - test_rag_with_ollama: switched model llama3 -> qwen3.5:0.8b (the llama3 model wasn't installed locally so the test always failed), and added `skipif(CI)` so the test never runs in GitHub Actions (no Ollama daemon in CI).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 13eb189f2f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…k files
Two P1 review-pass-4 findings (codex):
1. Legacy passphrase-mode SQLCipher databases (pre-2.6, no .salt
sidecar) were unopenable under 2.6 because every key was being
normalized through PBKDF2 and sent as PRAGMA key = "x'...'", which
produces a different SQLCipher internal key than the
passphrase-derived one used at original creation.
create_encrypted_connection now branches on sidecar presence:
- new DB or sidecar present → 600k-iter PBKDF2 → x'hex' raw-key path.
- existing DB without sidecar → legacy passphrase PRAGMA so
SQLCipher's internal KDF still matches.
2. utils.file_lock was unlinking the .lock file at context exit. flock
and msvcrt.locking are inode-bound, so removing the path while
another process is still queued on the old inode lets a third
process create a fresh inode and acquire a different lock
concurrently — defeating cross-process mutual exclusion for index
save/rebuild paths. The unlink is removed; a surviving zero-byte
.lock sidecar is far cheaper than a torn save.
Tests:
- tests/unit/test_v26_review_pass_4.py covers both invariants:
* legacy passphrase DB round-trips under 2.6 without creating a sidecar
* fresh DB still gets the new raw-key path + sidecar
* lock file persists across context exits and reuses the same inode
* concurrent acquisitions serialize
- tests/unit/test_v26_misc.py::TestFileLockCleanup updated to assert
the lock file persists (the new correct behavior) rather than the
prior unlink-on-exit invariant.
Plus: .gitignore now ignores .claude/ and .codex (per-developer
agentic-CLI scratch dirs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ships SimpleVecDB 2.6.0, a release focused on closing
correctness, durability, and security holes uncovered by three rounds
of code review against the 2.5.0 baseline. The version is in sync
across
pyproject.tomlandCHANGELOG.md(validated by the newpre-commit hook).
55 files changed, +4131/-749 across 11 commits.
Highlights
Concurrency & durability
UsearchIndex.save(sibling.tmp→ fsync →os.replace→ parent-dir fsync);_dirty=Falseclear is now inside the file lock so a concurrentadd()can never be silently overwritten.rebuild_indexswap via sibling.rebuildpath.encrypt_file/decrypt_file) with mode0o600and plaintext unlinked after the encrypted blob is durably on disk.RLockshared with everyCatalogManager, eliminatingwith self.conn:interleaving across collections.AsyncVectorDB.closedrains in-flight pool tasks (wait=True) before closing the connection.Security
<db>.salt, mode0o600), created withO_CREAT | O_EXCLso concurrent openers converge on the winner's salt and an existing sidecar is never clobbered.magic('SV') + version(1)bound into AES-GCMassociated_data, defeating header tampering and downgrade attacks. v0 blobs continue to decrypt for backwards compatibility.Search & catalog
^[a-zA-Z_][a-zA-Z0-9_]*$) at__init__time, blocking SQL injection viarepr(table_name)in legacy paths.RETURNING idfor auto-ID recovery (eliminates thelast_insert_rowid()arithmetic race)._upsert_fts_rowsraises if any-1sentinel rowid leaks in.delete_collectionexistence check moved inside the lock (closes TOCTOU).normalize_l2subnormal guard at< 1e-12.Integrations
add()stamps_simplevecdb_node_idinto metadata in the same transaction as the row insert; round-tripadd → querynow preserves the original LlamaIndex node id.migrate_node_id_metadata()helper backfills v2.5-shaped collections; idempotent.DeprecationWarningat__init__when v2.5-shaped rows are detected.delete()no longer swallowssqlite3.DatabaseError(silent-data-loss path closed).asyncio.to_thread.Quantization
DeprecationWarningand clip (preserving prior callers; future versions may raise).Tooling
scripts/check_version_sync.pyvalidates thatpyproject.tomland the latestCHANGELOG.mdheading are in lockstep.scripts/bump_version.py.scripts/track_metrics.py.pre-committo lefthook, split by cost:pre-commitruns version-sync + ruff (auto-stage fixes);pre-pushruns version-sync + mypy + bandit + pytest+coverage.Test coverage
725 unit tests pass, 2 skipped (the Ollama RAG test now uses
qwen3.5:0.8band is gated on$CI).Three new regression test modules pin invariants the prior suite
missed:
tests/unit/test_v26_review_pass_3.py— fsync-on-dir,.tmpcleanup on save failure,db._lock is catalog._lockshared-RLock identity, adversarial_validate_table_name, hybrid-search RRF symmetry, same-text-different-id dedup,list_all_idsround-trip.tests/unit/test_v26_encryption_review_pass_3.py— nonce uniqueness across saves, wrong-key-no-output, AAD-bound header tamper detection, salt sidecar O_EXCL preservation, v0→v1 migration round-trip.tests/unit/integrations/test_llamaindex_review_pass_3.py— round-tripadd → querypreserves node_id, end-to-end migrate-then-delete on v2.5 data, legacy-collectionDeprecationWarning,DatabaseErrorpropagation.Test plan
uv run pytest tests/unit -q --ignore=tests/integration→ 695 passed, 2 skippeduv run pytest tests/ -vv --cov=src/simplevecdb(full pre-push gate) → 725 passed, 2 skippeduv run mypy .→ no issues found in 87 source filesuv run bandit -r src/ -ll -c .bandit→ no issues identifieduv run python scripts/check_version_sync.py→ versions in sync (pyproject=2.6.0, CHANGELOG=2.6.0)mkdocs build→ built cleanly; deployed to gh-pagesSee the CHANGELOG for the full
file-by-file breakdown.