Release 2.5.0

CHANGELOG.md

@@ -5,6 +5,55 @@ All notable changes to SimpleVecDB will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.5.0] - 2026-04-07
+
+### Added
+
+- **`delete_collection(name)`** — drop a collection's SQLite tables, FTS index, and usearch file in one call. Available on both `VectorDB` and `AsyncVectorDB`.
+- **`store_embeddings` parameter** on `collection()` — opt into storing embedding BLOBs in SQLite (default `False`). Saves ~2x storage; MMR transparently fetches vectors from the usearch index when BLOBs are absent.
+- **`async_retry_on_lock` decorator** — async variant of `retry_on_lock` using `asyncio.sleep` instead of `time.sleep`, avoiding executor thread blocking.
+- **`file_lock` context manager** — advisory cross-process file locking (`fcntl`/`msvcrt`) for usearch index files. Prevents corruption from concurrent processes.
+- **`__repr__`** on `VectorDB`, `VectorCollection`, `AsyncVectorDB`, `AsyncVectorCollection` for debuggable string representations.
+- **FLOAT16 quantization** fully implemented in `serialize()`/`deserialize()` — was previously defined in the enum but raised `ValueError` at runtime.
+- **Pagination** on `get_documents(limit=, offset=)` and catalog methods (`find_ids_by_filter`, `find_ids_by_texts`) — previously returned unbounded result sets.
+- **Embeddings server enhancements:**
+  - Graceful shutdown with SIGTERM/SIGINT draining (10s timeout)
+  - CORS middleware with configurable origins for browser-based clients
+  - Model warm-up on startup (skip with `--no-warmup`)
+  - Input validation: rejects empty strings (422) and texts exceeding 100k chars (413)
+  - Proper `argparse` CLI with `--host`, `--port`, `--no-warmup`, `--help`
+  - Startup banner logging config summary (host, port, model, auth, rate limits)
+  - Nested token array normalization (`list[list[int]]` input format)
+  - Async executor offload for `embed_texts` (non-blocking event loop)
+  - OpenAPI version synced from package metadata
+  - Module `__init__.py` exports (`embed_texts`, `get_embedder`, `load_model`, `app`, `run_server`)
+
+### Fixed
+
+- **`delete_by_ids` ordering** — SQLite deletion now happens first (transactional, can rollback), then usearch. Previously usearch removed first, leaving orphaned catalog entries on SQLite failure.
+- **`_matches_filter` string semantics** — now uses exact equality, consistent with SQL `build_filter_clause`. Was using substring match (`value in str(meta_value)`).
+- **`list_collections`** — scans `sqlite_master` for persisted collection tables instead of returning only session-cached names. Works across reopened databases.
+- **WAL mode for encrypted databases** — `PRAGMA journal_mode=WAL` and `PRAGMA synchronous=NORMAL` now set for SQLCipher connections (was only set for unencrypted).
+- **`collection()` cache key** — includes `distance_strategy` and `quantization` in cache key (sync version). Previously cached by name only, silently ignoring differing params on cache hit.
+- **`_ensure_fts_table`** — retries up to 3 times on transient "database is locked" errors instead of permanently disabling FTS on first failure.
+- **Connection health check** — `SELECT 1` probe after connection creation; raises `RuntimeError` immediately on corrupt databases.
+
+### Improved
+
+- **Usearch batch operations** — `add()`, `remove()`, and `get()` now use batch usearch APIs instead of per-key loops. Significant speedup for large operations.
+- **Filtered search iterative deepening** — replaces fixed `k*3` overfetch with adaptive doubling (up to `k*30`). Highly selective filters now reliably return `k` results.
+- **Memory-map heuristic** — uses file size threshold (50MB) instead of inaccurate `file_size // 100` vector count estimate for mmap vs load decision.
+- **Apple chip detection** — uses `platform.processor()` instead of spawning a `sysctl` subprocess.
+
+### Removed
+
+- **Duplicate `_dim` property** — removed in favor of the public `dim` property.
+
+### Breaking Changes
+
+- String metadata filters now use exact equality (was substring match).
+- `store_embeddings` defaults to `False` — `rebuild_index()` requires `store_embeddings=True` or re-adding documents.
+
 ## [2.4.0] - 2026-03-22
 
 ### Added

README.md

@@ -140,10 +140,13 @@ hybrid = collection.hybrid_search("powerhouse cell", k=2)
 **Optional: Run embeddings server (OpenAI-compatible)**
 
 ```bash
-simplevecdb-server --port 8000
+simplevecdb-server --port 8000                # Default model, auto warm-up
+simplevecdb-server --host 0.0.0.0 --port 9000 # Bind to all interfaces
+simplevecdb-server --no-warmup                # Skip model preload on startup
+simplevecdb-server --help                     # Show all options
 ```
 
-See [Setup Guide](ENV_SETUP.md) for configuration: model registry, rate limits, API keys, CUDA optimization.
+See [Setup Guide](ENV_SETUP.md) for configuration: model registry, rate limits, API keys, CORS, CUDA optimization.
 
 ### Option 3: With LangChain or LlamaIndex
 
@@ -302,6 +305,10 @@ docs = collection.get_documents(filter_dict={"category": "tech"})
 for doc_id, text, metadata in docs:
     print(f"[{doc_id}] {text[:50]}...")
 
+# Paginated access (v2.5+)
+page1 = collection.get_documents(limit=100)
+page2 = collection.get_documents(limit=100, offset=100)
+
 # Fetch stored embeddings
 embeddings = collection.get_embeddings_by_ids([1, 2, 3])
 
@@ -313,6 +320,9 @@ collection.update_metadata([
 
 # Quick stats
 print(f"Collection has {collection.count()} documents, dim={collection.dim}")
+
+# Delete an entire collection (v2.5+)
+db.delete_collection("old_data")
 ```
 
 ### Vector Clustering (v2.2+)
@@ -355,6 +365,10 @@ Supports K-means, MiniBatch K-means, and HDBSCAN. See [Clustering Guide](https:/
 | **Cluster Persistence**   | ✅     | Save/load cluster centroids for fast assignment (v2.2+)      |
 | **Public Catalog API**    | ✅     | `get_documents`, `get_embeddings_by_ids`, `update_metadata` (v2.4+) |
 | **Executor Injection**    | ✅     | Share thread pool across async instances for ONNX safety (v2.4+) |
+| **Collection Management** | ✅     | `delete_collection()`, paginated `get_documents(limit=, offset=)` (v2.5+) |
+| **Cross-Process Safety**  | ✅     | Advisory file locking on usearch index files (v2.5+) |
+| **FLOAT16 Quantization**  | ✅     | Half-precision storage with 2x compression (v2.5+) |
+| **Embeddings Server**     | ✅     | CORS, graceful shutdown, input validation, model warm-up (v2.5+) |
 
 ## Performance Benchmarks
 
@@ -427,6 +441,9 @@ pip install torch --index-url https://download.pytorch.org/whl/cu118
 - [x] Vector clustering and auto-tagging (v2.2)
 - [x] Public catalog API for document management (v2.4)
 - [x] Async executor injection for thread-safe sharing (v2.4)
+- [x] Collection management: `delete_collection()`, pagination (v2.5)
+- [x] Cross-process file locking and connection health checks (v2.5)
+- [x] Embeddings server hardening: CORS, graceful shutdown, input validation (v2.5)
 - [ ] Incremental clustering (online learning)
 - [ ] Cluster visualization exports
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "simplevecdb"
-version = "2.4.0"
+version = "2.5.0"
 description = "Dead-simple local vector database powered by usearch HNSW."
 authors = [{ name = "Dayton Dunbar", email = "coderdayton14@gmail.com" }]
 license = { text = "MIT" }

src/simplevecdb/__init__.py

@@ -16,7 +16,13 @@
 except ImportError:
     pass
 from .logging import get_logger, configure_logging, log_operation
-from .utils import DatabaseLockedError, retry_on_lock, validate_filter
+from .utils import (
+    DatabaseLockedError,
+    async_retry_on_lock,
+    file_lock,
+    retry_on_lock,
+    validate_filter,
+)
 from .encryption import EncryptionError, EncryptionUnavailableError
 
 from importlib.metadata import version as _pkg_version
@@ -49,6 +55,8 @@
     "MigrationRequiredError",
     "EncryptionError",
     "EncryptionUnavailableError",
+    "async_retry_on_lock",
+    "file_lock",
     "retry_on_lock",
     "validate_filter",
 ]

src/simplevecdb/async_core.py

@@ -60,6 +60,9 @@ def name(self) -> str:
         """Collection name."""
         return self._collection.name
 
+    def __repr__(self) -> str:
+        return f"AsyncVectorCollection(name={self._collection.name!r})"
+
     async def add_texts(
         self,
         texts: Sequence[str],
@@ -210,15 +213,20 @@ async def delete_by_ids(self, ids: Sequence[int]) -> None:
     async def get_documents(
         self,
         filter_dict: dict[str, Any] | None = None,
+        *,
+        limit: int | None = None,
+        offset: int | None = None,
     ) -> list[tuple[int, str, dict[str, Any]]]:
-        """Get all documents with text content and metadata.
+        """Get documents with text content and metadata.
 
         See VectorCollection.get_documents for full documentation.
         """
         loop = asyncio.get_running_loop()
         return await loop.run_in_executor(
             self._executor,
-            lambda: self._collection.get_documents(filter_dict=filter_dict),
+            lambda: self._collection.get_documents(
+                filter_dict=filter_dict, limit=limit, offset=offset
+            ),
         )
 
     async def get_embeddings_by_ids(self, ids: Sequence[int]) -> dict[int, Any]:
@@ -599,9 +607,21 @@ def collection(
             return self._collections[cache_key]
 
     def list_collections(self) -> list[str]:
-        """Return names of all initialized collections."""
+        """Return names of all persisted collections in the database."""
         return self._db.list_collections()
 
+    async def delete_collection(self, name: str) -> None:
+        """Delete a collection and all its data."""
+        loop = asyncio.get_running_loop()
+        await loop.run_in_executor(
+            self._executor, lambda: self._db.delete_collection(name)
+        )
+        # Evict from async-level cache too
+        with self._collections_lock:
+            keys_to_remove = [k for k in self._collections if k[0] == name]
+            for k in keys_to_remove:
+                del self._collections[k]
+
     async def search_collections(
         self,
         query: Sequence[float],
@@ -644,6 +664,9 @@ async def vacuum(self, checkpoint_wal: bool = True) -> None:
             self._executor, lambda: self._db.vacuum(checkpoint_wal)
         )
 
+    def __repr__(self) -> str:
+        return f"AsyncVectorDB(path={self._db.path!r})"
+
     async def close(self) -> None:
         """Close the database connection and shutdown executor."""
         try:

src/simplevecdb/constants.py

@@ -74,7 +74,8 @@
 # - Instant startup (no full load into RAM)
 # - Lower memory footprint (OS manages page cache)
 # - Slight latency increase for cold pages (acceptable trade-off)
-USEARCH_MMAP_THRESHOLD = 100000
+# Threshold in bytes — 50MB covers ~30k 384-dim f32 vectors.
+USEARCH_MMAP_THRESHOLD = 50 * 1024 * 1024  # 50 MB
 
 # Batch search threshold: auto-batch queries when > this count
 # usearch batch search provides ~10x throughput for multi-query workloads