feature/v2.2 features

README.md

@@ -46,16 +46,20 @@ SimpleVecDB brings **Chroma-like simplicity** to a single **SQLite file**. Built
 ## Installation
 
 ```bash
-# Core library only (lightweight, 50MB)
+# Standard installation (includes clustering, encryption)
 pip install simplevecdb
 
-# With local embeddings server + HuggingFace models (500MB+)
+# With local embeddings server (adds 500MB+ models)
 pip install "simplevecdb[server]"
-
-# With encryption support (SQLCipher)
-pip install "simplevecdb[encryption]"
 ```
 
+**What's included by default:**
+- Vector search with HNSW indexing
+- Clustering (K-means, MiniBatch K-means, HDBSCAN)
+- Encryption (SQLCipher AES-256)
+- Async support
+- LangChain & LlamaIndex integrations
+
 **Verify Installation:**
 
 ```bash
@@ -282,6 +286,23 @@ parent = collection.get_parent(child_ids[0])
 descendants = collection.get_descendants(parent_ids[0])
 ```
 
+### Vector Clustering (v2.2+)
+
+Discover natural groupings in your embeddings:
+
+```python
+# Cluster documents and auto-generate tags
+result = collection.cluster(n_clusters=5)
+tags = collection.auto_tag(result, method="tfidf")
+collection.assign_cluster_metadata(result, tags)
+
+# Save for fast assignment of new documents
+collection.save_cluster("categories", result)
+collection.assign_to_cluster("categories", new_doc_ids)
+```
+
+Supports K-means, MiniBatch K-means, and HDBSCAN. See [Clustering Guide](https://coderdayton.github.io/SimpleVecDB/guides/clustering) for details.
+
 ## Feature Matrix
 
 | Feature                   | Status | Description                                                  |
@@ -301,6 +322,8 @@ descendants = collection.get_descendants(parent_ids[0])
 | **Built-in Encryption**   | ✅     | SQLCipher AES-256 at-rest encryption via `[encryption]` extras |
 | **Streaming Insert**      | ✅     | Memory-efficient large-scale ingestion with progress callbacks |
 | **Document Hierarchies**  | ✅     | Parent/child relationships for chunked docs                  |
+| **Vector Clustering**     | ✅     | K-means, MiniBatch K-means, HDBSCAN with auto-tagging (v2.2+) |
+| **Cluster Persistence**   | ✅     | Save/load cluster centroids for fast assignment (v2.2+)      |
 
 ## Performance Benchmarks
 
@@ -369,8 +392,10 @@ pip install torch --index-url https://download.pytorch.org/whl/cu118
 - [x] SQLCipher encryption (at-rest data protection)
 - [x] Streaming insert API for large-scale ingestion
 - [x] Hierarchical document relationships (parent/child)
-- [ ] Cross-collection search
-- [ ] Vector clustering and auto-tagging
+- [x] Cross-collection search
+- [x] Vector clustering and auto-tagging (v2.2)
+- [ ] Incremental clustering (online learning)
+- [ ] Cluster visualization exports
 
 Vote on features or propose new ones in [GitHub Discussions](https://github.com/coderdayton/simplevecdb/discussions).
 

docs/CHANGELOG.md

@@ -5,6 +5,107 @@ All notable changes to SimpleVecDB will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2026-01-17
+
+### Added
+
+- **Vector Clustering & Auto-Tagging** - Discover natural groupings in embeddings
+  - `VectorCollection.cluster()` - Cluster documents by semantic similarity
+    - **K-means**: Classic centroid-based clustering for balanced clusters
+    - **MiniBatch K-means**: Scalable variant for large datasets (default)
+    - **HDBSCAN**: Density-based clustering that auto-discovers cluster count
+  - `VectorCollection.auto_tag()` - Generate descriptive tags for clusters
+    - TF-IDF method (default): Extract keywords with highest TF-IDF scores
+    - Frequency method: Extract most common words per cluster
+    - Custom callback: Implement custom tagging logic (e.g., LLM-based)
+  - `VectorCollection.assign_cluster_metadata()` - Persist cluster IDs to document metadata
+  - `VectorCollection.get_cluster_members()` - Retrieve all documents in a cluster
+
+- **Cluster Quality Metrics** - Evaluate clustering results
+  - `ClusterResult.inertia` - Sum of squared distances to centroids (K-means only, lower is better)
+  - `ClusterResult.silhouette_score` - Cluster separation metric (-1 to 1, higher is better)
+  - `ClusterResult.metrics()` - Get all metrics as dictionary
+
+- **Cluster Persistence** - Save and reuse cluster configurations
+  - `VectorCollection.save_cluster()` - Save cluster centroids and metadata to database
+  - `VectorCollection.load_cluster()` - Load saved cluster configuration
+  - `VectorCollection.list_clusters()` - List all saved cluster configurations
+  - `VectorCollection.delete_cluster()` - Delete a saved cluster configuration
+  - `VectorCollection.assign_to_cluster()` - Assign new documents to saved clusters without re-clustering
+
+- **Async Clustering Support** - Full async/await parity for all clustering operations
+  - `AsyncVectorCollection.cluster()`, `auto_tag()`, `assign_cluster_metadata()`, `get_cluster_members()`
+  - `AsyncVectorCollection.save_cluster()`, `load_cluster()`, `list_clusters()`, `delete_cluster()`, `assign_to_cluster()`
+
+- **New Dependencies** - Now included in standard installation
+  - `scikit-learn>=1.3.0` - K-means, MiniBatch K-means, silhouette score
+  - `hdbscan>=0.8.33` - Density-based clustering
+  - `sqlcipher3-binary>=0.5.0` - Encryption support (previously optional)
+  - `cryptography>=41.0` - Encryption utilities (previously optional)
+
+- **Documentation**
+  - New comprehensive clustering guide: `docs/guides/clustering.md`
+    - Algorithm comparison and selection guide
+    - Quality metrics interpretation
+    - Cluster persistence workflows
+    - Use cases: product categorization, topic discovery, customer segmentation, duplicate detection
+    - Best practices and troubleshooting
+  - New types reference: `docs/api/types.md`
+    - Complete `ClusterResult` API documentation
+    - `Document`, `DistanceStrategy`, `Quantization`, `ClusterAlgorithm` reference
+  - Updated README.md and docs/index.md with clustering sections
+  - Enhanced `docs/api/core.md` with clustering examples
+
+### Changed
+
+- **pyproject.toml**: Updated `scikit-learn` minimum version from `1.0` to `1.3.0` for improved clustering stability
+
+### Testing
+
+- Added 26 clustering tests in `tests/unit/test_clustering.py`:
+  - 16 core clustering tests (algorithms, auto-tagging, metadata persistence, edge cases)
+  - 4 cluster metrics tests (inertia, silhouette, metrics method)
+  - 6 cluster persistence tests (save/load/list/delete/assign)
+- Added 3 async clustering tests in `tests/unit/test_async.py`
+- Total test count: 305 (up from 292)
+
+### Installation
+
+Clustering and encryption are now included by default:
+
+```bash
+pip install simplevecdb
+```
+
+No extra installation steps required!
+
+### Example
+
+```python
+from simplevecdb import VectorDB
+
+db = VectorDB("products.db")
+collection = db.collection("items")
+
+# Cluster documents
+result = collection.cluster(n_clusters=5, algorithm="minibatch_kmeans")
+
+# Generate tags and persist
+tags = collection.auto_tag(result, method="tfidf", n_keywords=3)
+collection.assign_cluster_metadata(result, tags)
+
+# Save for fast assignment of new documents
+collection.save_cluster("categories", result, metadata={"tags": tags})
+
+# Later: assign new documents without re-clustering
+new_ids = collection.add_texts(new_texts, embeddings=new_embeddings)
+collection.assign_to_cluster("categories", new_ids)
+
+# Evaluate quality
+print(f"Silhouette Score: {result.silhouette_score:.2f}")  # 0.62
+print(f"Inertia: {result.inertia:.2f}")  # 1523.45
+```
+
 ## [2.0.0] - 2025-12-23
 
 ### Breaking Changes

docs/api/core.md

@@ -8,6 +8,8 @@ The main database class for managing vector collections.
     options:
       members:
         - collection
+        - list_collections
+        - search_collections
         - vacuum
         - close
         - check_migration
@@ -34,6 +36,15 @@ A named collection of vectors within a database.
         - get_descendants
         - get_ancestors
         - set_parent
+        - cluster
+        - auto_tag
+        - assign_cluster_metadata
+        - get_cluster_members
+        - save_cluster
+        - load_cluster
+        - list_clusters
+        - delete_cluster
+        - assign_to_cluster
 
 ## Quick Reference
 
@@ -145,3 +156,173 @@ results = collection.similarity_search(
 | `get_descendants(doc_id, max_depth)` | All nested children recursively |
 | `get_ancestors(doc_id)` | Path from document to root |
 | `set_parent(doc_id, parent_id)` | Move document to new parent (or None to orphan) |
+
+### Cross-Collection Search
+
+Search across multiple collections with unified, ranked results:
+
+```python
+from simplevecdb import VectorDB
+
+db = VectorDB("app.db")
+
+# Initialize collections
+users = db.collection("users")
+products = db.collection("products")
+docs = db.collection("docs")
+
+# Add data to each collection
+users.add_texts(["Alice likes hiking"], embeddings=[[0.1]*384])
+products.add_texts(["Hiking boots", "Trail map"], embeddings=[[0.2]*384, [0.15]*384])
+docs.add_texts(["Mountain hiking guide"], embeddings=[[0.12]*384])
+
+# List initialized collections
+print(db.list_collections())  # ['users', 'products', 'docs']
+
+# Search across ALL collections
+results = db.search_collections([0.1]*384, k=5)
+for doc, score, collection_name in results:
+    print(f"[{collection_name}] {doc.page_content} (score: {score:.3f})")
+
+# Search specific collections only
+results = db.search_collections(
+    [0.1]*384,
+    collections=["users", "products"],  # Exclude 'docs'
+    k=3
+)
+
+# With metadata filtering (applies to all collections)
+results = db.search_collections(
+    [0.1]*384,
+    k=10,
+    filter={"category": "outdoor"}
+)
+
+# Disable score normalization (returns inverted distances)
+results = db.search_collections([0.1]*384, normalize_scores=False)
+
+# Sequential search (disable parallelism)
+results = db.search_collections([0.1]*384, parallel=False)
+```
+
+| Method | Description |
+|--------|-------------|
+| `list_collections()` | Names of all initialized collections |
+| `search_collections(query, collections, k, filter, normalize_scores, parallel)` | Search across multiple collections with merged results |
+
+<a id="clustering-auto-tagging"></a>
+
+### Clustering & Auto-Tagging
+
+Group similar documents and generate descriptive tags:
+
+```python
+from simplevecdb import VectorDB
+
+db = VectorDB("app.db")
+collection = db.collection("docs")
+
+# Add documents with embeddings
+collection.add_texts(texts, embeddings=embeddings)
+
+# Cluster documents into groups
+result = collection.cluster(
+    n_clusters=5,
+    algorithm="minibatch_kmeans",  # or "kmeans", "hdbscan"
+    random_state=42
+)
+print(result.summary())  # {0: 42, 1: 38, 2: 15, 3: 3, 4: 2}
+
+# Generate keyword tags for each cluster
+tags = collection.auto_tag(result, n_keywords=5)
+# {0: 'machine learning, neural network, deep', 1: 'database, sql, query', ...}
+
+# Persist cluster assignments to metadata
+collection.assign_cluster_metadata(result, tags)
+
+# Query documents by cluster
+ml_docs = collection.get_cluster_members(0)
+db_docs = collection.similarity_search(query, filter={"cluster": 1})
+
+# Custom tagging callback
+def summarize_cluster(texts: list[str]) -> str:
+    return f"Group of {len(texts)} docs about {texts[0][:20]}..."
+
+custom_tags = collection.auto_tag(result, method="custom", custom_callback=summarize_cluster)
+```
+
+| Method | Description |
+|--------|-------------|
+| `cluster(n_clusters, algorithm, filter, sample_size)` | Cluster documents by embedding similarity |
+| `auto_tag(result, method, n_keywords, custom_callback)` | Generate descriptive tags for clusters |
+| `assign_cluster_metadata(result, tags, metadata_key)` | Persist cluster IDs to document metadata |
+| `get_cluster_members(cluster_id, metadata_key)` | Retrieve all documents in a cluster |
+| `save_cluster(name, result, metadata)` | Save cluster centroids for later assignment |
+| `load_cluster(name)` | Load saved cluster configuration |
+| `list_clusters()` | List all saved cluster configurations |
+| `delete_cluster(name)` | Delete a saved cluster configuration |
+| `assign_to_cluster(name, doc_ids, metadata_key)` | Assign documents to saved clusters |
+
+**Algorithms:**
+
+| Algorithm | Best For | Requires n_clusters |
+|-----------|----------|-------------------|
+| `minibatch_kmeans` | Large datasets (default) | Yes |
+| `kmeans` | Small datasets, precise centroids | Yes |
+| `hdbscan` | Unknown cluster count, density-based | No |
+
+Clustering is included in the standard installation (no extras needed).
+
+### Cluster Metrics
+
+Access clustering quality metrics to evaluate results:
+
+```python
+result = collection.cluster(n_clusters=5, random_state=42)
+
+# Inertia (K-means only): sum of squared distances to centroids
+# Lower is better; indicates tighter clusters
+print(f"Inertia: {result.inertia}")
+
+# Silhouette score: measure of cluster separation (-1 to 1)
+# Higher is better; >0.5 indicates good clustering
+print(f"Silhouette: {result.silhouette_score}")
+
+# Get all metrics as dict
+metrics = result.metrics()
+# {'inertia': 1523.45, 'silhouette_score': 0.62}
+```
+
+### Cluster Persistence
+
+Save cluster configurations for fast assignment of new documents:
+
+```python
+# 1. Cluster your documents
+result = collection.cluster(n_clusters=5, random_state=42)
+tags = collection.auto_tag(result)
+
+# 2. Save cluster state (centroids + metadata)
+collection.save_cluster(
+    "product_categories",
+    result,
+    metadata={"tags": tags, "version": 1}
+)
+
+# 3. Later: assign new documents without re-clustering
+new_ids = collection.add_texts(new_texts, embeddings=new_embeddings)
+collection.assign_to_cluster("product_categories", new_ids)
+
+# List saved clusters
+clusters = collection.list_clusters()
+# [{'name': 'product_categories', 'n_clusters': 5, 'algorithm': 'minibatch_kmeans', ...}]
+
+# Load cluster for inspection
+saved = collection.load_cluster("product_categories")
+if saved:
+    result, meta = saved
+    print(f"Loaded {result.n_clusters} clusters, tags: {meta['tags']}")
+
+# Delete when no longer needed
+collection.delete_cluster("product_categories")
+```