Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified .bandit
100644 → 100755
Empty file.
Empty file modified .env.example
100644 → 100755
Empty file.
Empty file modified .github/FUNDING.yml
100644 → 100755
Empty file.
Empty file modified .github/ISSUE_TEMPLATE/bug_report.yml
100644 → 100755
Empty file.
Empty file modified .github/ISSUE_TEMPLATE/config.yml
100644 → 100755
Empty file.
Empty file modified .github/ISSUE_TEMPLATE/feature_request.yml
100644 → 100755
Empty file.
Empty file modified .github/dependabot.yml
100644 → 100755
Empty file.
Empty file modified .github/workflows/ci.yml
100644 → 100755
Empty file.
Empty file modified .github/workflows/publish.yml
100644 → 100755
Empty file.
Empty file modified .github/workflows/security.yml
100644 → 100755
Empty file.
Empty file modified .github/workflows/update-sponsors.yml
100644 → 100755
Empty file.
Empty file modified .gitignore
100644 → 100755
Empty file.
7 changes: 7 additions & 0 deletions .pre-commit-config.yaml
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
repos:
- repo: local
hooks:
- id: version-sync
name: version sync check
entry: python3 scripts/check_version_sync.py
language: system
files: ^(pyproject\.toml|src/simplevecdb/__init__\.py)$
pass_filenames: false

- id: ruff
name: ruff
entry: uv run ruff check . --fix
Expand Down
2 changes: 1 addition & 1 deletion .python-version
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.10
3.12
57 changes: 57 additions & 0 deletions CHANGELOG.md
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,62 @@ All notable changes to SimpleVecDB will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.3.0] - 2026-03-08

### Breaking Changes

- **Integration dependencies are now optional.** LangChain and LlamaIndex packages are no longer installed by default. Install with `pip install simplevecdb[integrations]` to use them. Existing users upgrading from v2.2.x will see a clear ImportError with migration instructions.

### Added

- **`[integrations]` optional extra** — Install LangChain and LlamaIndex dependencies only when needed, reducing default install footprint
- **Runtime import guards** in integration modules with v2.3.0 migration messaging
- **Lazy `__getattr__` loading** in `integrations/__init__.py` — integration classes are only imported when accessed
- **Input validation guards** on search methods:
- `similarity_search`, `similarity_search_batch`, `keyword_search`, `hybrid_search` now reject `k <= 0`
- `add_texts` validates length consistency of `metadatas`, `embeddings`, `ids`, and `parent_ids` against `texts`
- **NaN/Inf validation** for float values in metadata filters (`utils.validate_filter`)
- **Empty list rejection** for list filter values
- **Double-close protection** on `VectorDB` with `_closed` flag
- **Context manager protocol** (`__enter__`/`__exit__`) on `VectorDB`
- **Table name validation** in `check_migration` (defense-in-depth against SQL injection)
- **Graceful per-future error handling** in `search_collections`
- **Adaptive batch search threshold** — queries below `USEARCH_BATCH_THRESHOLD` (10) use sequential search to avoid batch overhead

### Changed

- **Python dev target changed to 3.12** (`.python-version`), `requires-python` remains `>= "3.10"`
- **Version bumped to 2.3.0**
- **Performance: MMR search vectorized** — pre-normalize embeddings once, use `sel_matrix @ emb` matrix-vector multiply instead of Python inner loop, O(1) `list.pop` replaces O(n) `list.remove`, hoist `1 - lambda_mult` loop invariant
- **Performance: merged SQL round-trips in MMR** — new `get_documents_and_embeddings_by_ids` fetches text, metadata, and embeddings in a single query (previously two separate SELECTs)
- **Performance: `get_parent` collapsed** from 2 sequential SELECTs to 1 self-JOIN
- **Performance: `add_documents` ID recovery** — skip redundant `SELECT ORDER BY DESC` when explicit IDs are provided; removed unnecessary `list(texts)` copy
- **Performance: FLOAT serialization** — `np.asarray().tobytes()` replaces `struct.pack` with per-element Python loop (single C memcpy)
- **Performance: `np.array` → `np.asarray`** on every search and insert path to avoid unnecessary copies
- **Performance: SQL placeholder strings** — `",".join(["?"] * len(ids))` replaces generator expression across all 9 call sites
- **Performance: batched numpy conversion** in `add_texts` — single `np.asarray` call instead of per-item conversion
- **Performance: compact JSON separators** in catalog serialization
- **Performance: deduplicated `.tolist()` calls** in search engine
- **Performance: `np.unique(ravel())`** for batch key collection in `similarity_search_batch`
- **Performance: usearch upsert** — skip contains-check loop on empty index, cache `int(key)` once per iteration
- **Performance: cluster table DDL** — `_cluster_table_ready` flag skips `CREATE TABLE IF NOT EXISTS` on repeated calls; cached `_cluster_table_name`
- **`_normalize_key`** now delegates to `_derive_key` instead of duplicating PBKDF2 logic
- **HNSW defaults** in `usearch_index.py` now sourced from `constants.py` (removed local duplicates)
- **Collection name regex** uses `constants.COLLECTION_NAME_PATTERN` instead of hardcoded pattern
- **`VectorDB` defaults** for `distance_strategy` and `quantization` sourced from `constants.DEFAULT_DISTANCE_STRATEGY` / `constants.DEFAULT_QUANTIZATION`
- **`_batched` utility** moved from `core.py` to `utils.py` for reuse; now used in `catalog.py` batch updates
- **`auto_tag`** uses `defaultdict(list)` instead of manual if-not-in pattern
- **`import random`** hoisted to module level in `utils.py` (was inside retry loop)
- **Streaming placeholder bug fixed** — `_process_streaming_batch` now correctly detects `None` placeholders (previously used empty list `[]`, preventing auto-embedding replacement)
- **README updated** to document `pip install simplevecdb[integrations]` installation

### Removed

- LangChain and LlamaIndex packages from core `[project.dependencies]` (moved to `[project.optional-dependencies] integrations`)
- Duplicated HNSW default constants from `usearch_index.py` (now single source in `constants.py`)
- Unused `struct` import from `quantization.py`
- Unused `itertools` import from `core.py`

## [2.2.1] - 2026-01-27

### Changed
Expand Down Expand Up @@ -429,6 +485,7 @@ Benchmarks on i9-13900K & RTX 4090 with 10k vectors (384-dim):
- **Documentation**: https://coderdayton.github.io/simplevecdb/
- **License**: MIT

[2.3.0]: https://github.com/coderdayton/simplevecdb/releases/tag/v2.3.0
[2.2.1]: https://github.com/coderdayton/simplevecdb/releases/tag/v2.2.1
[2.2.0]: https://github.com/coderdayton/simplevecdb/releases/tag/v2.2.0
[2.1.0]: https://github.com/coderdayton/simplevecdb/releases/tag/v2.1.0
Expand Down
Empty file modified CODE_OF_CONDUCT.md
100644 → 100755
Empty file.
Empty file modified CONTRIBUTING.md
100644 → 100755
Empty file.
Empty file modified LICENSE
100644 → 100755
Empty file.
12 changes: 9 additions & 3 deletions README.md
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ SimpleVecDB brings **Chroma-like simplicity** to a single **SQLite file**. Built
- **Blazing Fast** — 10-100x faster search via usearch HNSW. Adaptive: brute-force for <10k vectors (perfect recall), HNSW for larger collections.
- **Truly Portable** — Runs anywhere SQLite runs: Linux, macOS, Windows, even WASM.
- **Async Ready** — Full async/await support for web servers and concurrent workloads.
- **Batteries Included** — Optional FastAPI embeddings server + LangChain/LlamaIndex integrations.
- **Batteries Included** — Optional FastAPI embeddings server + LangChain/LlamaIndex integrations via `[integrations]` extra.
- **Production Ready** — Hybrid search (BM25 + vector), metadata filtering, multi-collection support, and automatic hardware acceleration.

### When to Choose SimpleVecDB
Expand Down Expand Up @@ -49,6 +49,9 @@ SimpleVecDB brings **Chroma-like simplicity** to a single **SQLite file**. Built
# Standard installation (includes clustering, encryption)
pip install simplevecdb

# With LangChain & LlamaIndex integrations
pip install "simplevecdb[integrations]"

# With local embeddings server (adds 500MB+ models)
pip install "simplevecdb[server]"
```
Expand All @@ -58,7 +61,6 @@ pip install "simplevecdb[server]"
- Clustering (K-means, MiniBatch K-means, HDBSCAN)
- Encryption (SQLCipher AES-256)
- Async support
- LangChain & LlamaIndex integrations

**Verify Installation:**

Expand Down Expand Up @@ -147,6 +149,10 @@ See [Setup Guide](ENV_SETUP.md) for configuration: model registry, rate limits,

Best for: Existing RAG pipelines, framework-based workflows.

```bash
pip install "simplevecdb[integrations]"
```

```python
from simplevecdb.integrations.langchain import SimpleVecDBVectorStore
from langchain_openai import OpenAIEmbeddings
Expand Down Expand Up @@ -316,7 +322,7 @@ Supports K-means, MiniBatch K-means, and HDBSCAN. See [Clustering Guide](https:/
| **Quantization** | ✅ | FLOAT32, FLOAT16, INT8, BIT for 2-32x compression |
| **Parallel Operations** | ✅ | `threads` parameter for add/search |
| **Metadata Filtering** | ✅ | SQL `WHERE` clause support |
| **Framework Integration** | ✅ | LangChain \& LlamaIndex adapters |
| **Framework Integration** | ✅ | LangChain \& LlamaIndex adapters via `[integrations]` extra |
| **Hardware Acceleration** | ✅ | Auto-detects CUDA/MPS/CPU + SIMD via usearch |
| **Local Embeddings** | ✅ | HuggingFace models via `[server]` extras |
| **Built-in Encryption** | ✅ | SQLCipher AES-256 at-rest encryption via `[encryption]` extras |
Expand Down
Empty file modified SECURITY.md
100644 → 100755
Empty file.
Empty file modified docs/CHANGELOG.md
100644 → 100755
Empty file.
Empty file modified docs/CONTRIBUTING.md
100644 → 100755
Empty file.
Empty file modified docs/ENV_SETUP.md
100644 → 100755
Empty file.
1 change: 0 additions & 1 deletion docs/LICENSE

This file was deleted.

Empty file added docs/LICENSE
Empty file.
Empty file modified docs/api/async.md
100644 → 100755
Empty file.
Empty file modified docs/api/config.md
100644 → 100755
Empty file.
Empty file modified docs/api/core.md
100644 → 100755
Empty file.
Empty file modified docs/api/embeddings.md
100644 → 100755
Empty file.
Empty file modified docs/api/encryption.md
100644 → 100755
Empty file.
Empty file modified docs/api/engine/catalog.md
100644 → 100755
Empty file.
Empty file modified docs/api/engine/quantization.md
100644 → 100755
Empty file.
Empty file modified docs/api/engine/search.md
100644 → 100755
Empty file.
Empty file modified docs/api/integrations.md
100644 → 100755
Empty file.
Empty file modified docs/api/types.md
100644 → 100755
Empty file.
Empty file modified docs/benchmarks.md
100644 → 100755
Empty file.
Empty file modified docs/examples.md
100644 → 100755
Empty file.
Empty file modified docs/guides/clustering.md
100644 → 100755
Empty file.
Empty file modified docs/index.md
100644 → 100755
Empty file.
Empty file modified examples/auto_embed.py
100644 → 100755
Empty file.
Empty file modified examples/backend_benchmark.py
100644 → 100755
Empty file.
Empty file modified examples/embeddings/perf_benchmark.py
100644 → 100755
Empty file.
Empty file modified examples/quant_benchmark.py
100644 → 100755
Empty file.
Empty file modified examples/rag/langchain_rag.ipynb
100644 → 100755
Empty file.
Empty file modified examples/rag/llama_rag.ipynb
100644 → 100755
Empty file.
Empty file modified examples/rag/ollama_rag.ipynb
100644 → 100755
Empty file.
Empty file modified examples/smoke_test.py
100644 → 100755
Empty file.
Empty file modified mkdocs.yml
100644 → 100755
Empty file.
18 changes: 10 additions & 8 deletions pyproject.toml
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "simplevecdb"
version = "2.2.1"
version = "2.3.0"
description = "Dead-simple local vector database powered by usearch HNSW."
authors = [{ name = "Dayton Dunbar", email = "coderdayton14@gmail.com" }]
license = { text = "MIT" }
Expand All @@ -15,14 +15,16 @@ dependencies = [
"hdbscan>=0.8.33", # Density-based clustering
"sqlcipher3-binary>=0.5.0", # Encryption support
"cryptography>=41.0", # Encryption utilities
"langchain-core>=1.0.7", # LangChain integration
"langchain-openai>=1.0.3", # LangChain OpenAI support
"llama-index>=0.14.8", # LlamaIndex integration
"llama-index-llms-ollama>=0.9.0", # LlamaIndex Ollama support
"llama-index-llms-openai-like>=0.5.3", # LlamaIndex OpenAI-like support
]

[project.optional-dependencies]
integrations = [
"langchain-core>=1.0.7",
"langchain-openai>=1.0.3",
"llama-index>=0.14.8",
"llama-index-llms-ollama>=0.9.0",
"llama-index-llms-openai-like>=0.5.3",
]
server = [
"fastapi>=0.115",
"uvicorn[standard]>=0.30",
Expand Down Expand Up @@ -66,9 +68,9 @@ markers = [
]

[tool.ruff]
target-version = "py310"
target-version = "py312"
exclude = ["exploration", "docs", "htmlcov", "site"]

[tool.mypy]
python_version = "3.10"
python_version = "3.12"
exclude = ["exploration", "docs", "htmlcov", "site"]
8 changes: 5 additions & 3 deletions src/simplevecdb/__init__.py
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@
from .core import VectorDB, VectorCollection, get_optimal_batch_size
from .async_core import AsyncVectorDB, AsyncVectorCollection
from .config import config
from .integrations.langchain import SimpleVecDBVectorStore
from .integrations.llamaindex import SimpleVecDBLlamaStore
try:
from .integrations import SimpleVecDBVectorStore, SimpleVecDBLlamaStore
except ImportError:
pass
from .logging import get_logger, configure_logging, log_operation
from .utils import DatabaseLockedError, retry_on_lock, validate_filter
from .encryption import EncryptionError, EncryptionUnavailableError

__version__ = "2.2.1"
__version__ = "2.3.0"
__all__ = [
# Core classes
"VectorDB",
Expand Down
Empty file modified src/simplevecdb/async_core.py
100644 → 100755
Empty file.
Empty file modified src/simplevecdb/config.py
100644 → 100755
Empty file.
Empty file modified src/simplevecdb/constants.py
100644 → 100755
Empty file.
Loading
Loading