Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bandit
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ exclude_dirs:

skips:

- B104
- B104 # 0.0.0.0 binding: SERVER_HOST defaults to 127.0.0.1; bandit can't see runtime defaults, so the warning is a false positive on this codebase. Keep it skipped only because the default is safe — if anyone introduces a hardcoded "0.0.0.0", remove this skip.
- B608 # SQL injection false positive: table names are validated via _validate_table_name()
6 changes: 4 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,18 @@ build/
*.db
*.sqlite

# OpenCode
# Agentic CLI tools (per-developer state)
.opencode/
opencode.json
.claude/
.codex

# Project specific
simplevecdb_plan.md
AGENTS.md
htmlcov/
site/
scripts
htmlcov/
.coverage
NEXT_UPDATES.md
pro_pack/
37 changes: 0 additions & 37 deletions .pre-commit-config.yaml

This file was deleted.

111 changes: 111 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

160 changes: 160 additions & 0 deletions docs/CHANGELOG.md

Large diffs are not rendered by default.

59 changes: 53 additions & 6 deletions docs/api/encryption.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,57 @@ db = VectorDB("secure.db", encryption_key=encryption_key)
With encryption enabled, files are stored as:

```
mydb.db # SQLCipher encrypted SQLite database
mydb.db.default.usearch.enc # AES-256-GCM encrypted usearch index
mydb.db # SQLCipher encrypted SQLite database
mydb.db.salt # 16-byte random salt sidecar (mode 0o600)
mydb.db.default.usearch.enc # AES-256-GCM encrypted usearch index (v1)
mydb.db.default.usearch.enc.salt # 16-byte salt sidecar for the index
```

When opened, the index is decrypted to memory (or a temp file). On `save()` or `close()`, the index is re-encrypted.

### Per-DB random salt (2.6.0+)

Each encrypted database and each encrypted index file gets its own
random 16-byte salt, written to a sibling `.salt` file with mode
`0o600`. The salt is the second input to PBKDF2-HMAC-SHA256, so two
databases that share the same passphrase derive **different** keys.

The sidecar is created with `O_CREAT | O_EXCL` so two processes opening
the same fresh database concurrently cannot race to write conflicting
salts; the loser reads the winner's salt and proceeds. An existing
sidecar is never overwritten — clobbering it would render the database
permanently unreadable with the original passphrase.

Pre-2.6.0 databases continue to open with a fixed legacy salt when no
sidecar is present, so existing on-disk data keeps working unchanged.

### v1 index file format (2.6.0+)

Index files written by 2.6.0+ start with a 3-byte version header:

```
magic = b"SV" (2 bytes)
version = 0x01 (1 byte)
nonce = 12 bytes
ciphertext + GCM tag
```

The header bytes are bound into the AES-GCM **associated_data**, so
any tampering with the magic or version (including a downgrade attempt
that strips them) fails authentication on decrypt. Pre-2.6.0 (v0) blobs
have no header and continue to decrypt successfully — `decrypt_file`
detects the format automatically.

### Atomic durability

`encrypt_file` and `decrypt_file` write to a sibling `.tmp` file,
`fsync()` the data, set mode `0o600`, then `os.replace()` onto the
target. The parent directory is also fsynced so the rename itself is
durable on POSIX. A crash mid-write leaves only the orphan temp file —
the live target is never torn. `encrypt_index_file` only unlinks the
plaintext after the encrypted output is durably on disk, so an
interrupted re-encryption never destroys data.

## Performance

### Search Operations
Expand Down Expand Up @@ -135,10 +180,12 @@ except EncryptionUnavailableError:

## Security Notes

- **SQLCipher** uses AES-256-CBC with HMAC-SHA512 for authentication
- **Index encryption** uses AES-256-GCM with random 96-bit nonces
- **Key derivation** uses PBKDF2-SHA256 with 480,000 iterations (OWASP 2023 recommendation)
- **The encryption key is held in memory** during database usage
- **SQLCipher** uses AES-256-CBC with HMAC-SHA512 for authentication.
- **Index encryption** uses AES-256-GCM with random 96-bit nonces (`secrets.token_bytes`); each save generates a fresh nonce.
- **Key derivation** uses PBKDF2-HMAC-SHA256 with **600,000 iterations** (OWASP 2024 recommendation) and a per-DB random salt.
- **v1 file format** binds the magic+version header bytes into AES-GCM `associated_data`, defeating header tampering and downgrade attacks.
- **Derived keys** are cached in a bounded LRU (max 64 entries, serialized by a thread lock) so repeat opens within a process avoid the 600k-iter cost without leaking key material in long-running multi-tenant processes.
- **The encryption key is held in memory** during database usage.

## API Reference

Expand Down
39 changes: 39 additions & 0 deletions lefthook.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Lefthook config — replaces .pre-commit-config.yaml.
#
# Split:
# pre-commit → fast feedback only (version sync + ruff --fix).
# pre-push → heavyweight gates (mypy, bandit, full pytest+coverage).
#
# Hooks within a stage run sequentially because ruff --fix can rewrite
# files mid-run and ``stage_fixed: true`` re-stages them, so the user
# does not have to re-add manually.

pre-commit:
jobs:
- name: version-sync
glob:
- "pyproject.toml"
- "src/simplevecdb/__init__.py"
run: uv run python3 scripts/check_version_sync.py

- name: ruff
glob: "*.py"
stage_fixed: true
run: uv run ruff check . --fix

pre-push:
jobs:
- name: version-sync
run: uv run python3 scripts/check_version_sync.py

- name: mypy
glob: "*.py"
run: uv run mypy .

- name: bandit
glob: "*.py"
run: uv run bandit -r src/ -ll -c .bandit

- name: pytest-cov
glob: "*.py"
run: uv run pytest tests/ -vv --cov=src/simplevecdb
42 changes: 39 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,11 +1,38 @@
[project]
name = "simplevecdb"
version = "2.5.0"
version = "2.6.0"
description = "Dead-simple local vector database powered by usearch HNSW."
authors = [{ name = "Dayton Dunbar", email = "coderdayton14@gmail.com" }]
license = { text = "MIT" }
readme = "README.md"
requires-python = ">=3.10"
keywords = [
"vector-database",
"vectordb",
"usearch",
"hnsw",
"sqlite",
"embeddings",
"rag",
"similarity-search",
"langchain",
"llamaindex",
]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Database",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Information Analysis",
"Typing :: Typed",
]

dependencies = [
"numpy>=1.24",
Expand All @@ -15,8 +42,15 @@ dependencies = [
"hdbscan>=0.8.33", # Density-based clustering
"sqlcipher3-binary>=0.5.0", # Encryption support
"cryptography>=41.0", # Encryption utilities
"python-dotenv>=1.0", # Loaded by simplevecdb.config at import time
]

[project.urls]
Homepage = "https://github.com/CoderDayton/simplevecdb"
Repository = "https://github.com/CoderDayton/simplevecdb"
Issues = "https://github.com/CoderDayton/simplevecdb/issues"
Changelog = "https://github.com/CoderDayton/simplevecdb/blob/main/CHANGELOG.md"

[project.optional-dependencies]
integrations = [
"langchain-core>=1.0.7",
Expand Down Expand Up @@ -68,9 +102,11 @@ markers = [
]

[tool.ruff]
target-version = "py312"
# Aligned with the declared floor in [project] requires-python so the
# linter actually flags 3.10/3.11-incompatible code.
target-version = "py310"
exclude = ["exploration", "docs", "htmlcov", "site"]

[tool.mypy]
python_version = "3.12"
python_version = "3.10"
exclude = ["exploration", "docs", "htmlcov", "site"]
88 changes: 88 additions & 0 deletions scripts/bump_version.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
import argparse
import re
import sys
from pathlib import Path

# Files to update. simplevecdb.__init__ derives __version__ dynamically via
# importlib.metadata, so no version literal lives there.
FILES = [
Path("pyproject.toml"),
]


def get_current_version():
"""Read version from pyproject.toml"""
content = FILES[0].read_text()
match = re.search(r'version = "(\d+\.\d+\.\d+)"', content)
if not match:
raise ValueError("Could not find version in pyproject.toml")
return match.group(1)


def bump_semver(current: str, part: str) -> str:
"""Bump major, minor, or patch version."""
major, minor, patch = map(int, current.split("."))
if part == "major":
return f"{major + 1}.0.0"
elif part == "minor":
return f"{major}.{minor + 1}.0"
elif part == "patch":
return f"{major}.{minor}.{patch + 1}"
return part # Assume it's a specific version string


def update_file(path: Path, old_ver: str, new_ver: str):
"""Update version in file content. Uses anchored regex to avoid replacing
incidental occurrences of the version string elsewhere in the file."""
content = path.read_text()

if path.name == "pyproject.toml":
pattern = re.compile(
r'^(version\s*=\s*)"' + re.escape(old_ver) + r'"',
flags=re.MULTILINE,
)
replacement = r'\g<1>"' + new_ver + r'"'
else:
return

new_content, count = pattern.subn(replacement, content)
if count == 0:
print(f"Warning: Could not find anchored version {old_ver!r} in {path}")
return

path.write_text(new_content)
print(f"Updated {path}")


def main():
parser = argparse.ArgumentParser(description="Bump version of SimpleVecDB")
parser.add_argument(
"version", help="New version (x.y.z) or part to bump (major, minor, patch)"
)
args = parser.parse_args()

try:
current_ver = get_current_version()
new_ver = bump_semver(current_ver, args.version)

print(f"Bumping version: {current_ver} -> {new_ver}")

for file_path in FILES:
if file_path.exists():
update_file(file_path, current_ver, new_ver)
else:
print(f"Warning: File not found: {file_path}")

print("\nDone! Don't forget to:")
print(f" git add {' '.join(str(f) for f in FILES)}")
print(f' git commit -m "Bump version to {new_ver}"')
print(f" git tag v{new_ver}")

except Exception as e:
print(f"Error: {e}")
sys.exit(1)


if __name__ == "__main__":
main()
Loading
Loading