Skip to content

vectordbpipe/vectorDBpipe

BranchesTags

Repository files navigation

Typing SVG

vectorDBpipe

The All-in-One Enterprise RAG Engine with Omni-RAG Architecture

PyPI version Python 3.8+ License: MIT CI Version 0.2.4 Tests 39 passed PyPI npm TUI

The only Python SDK you need for RAG — Ingest once, query with 4 intelligent engines.

📋 Table of Contents

🧠 What is vectorDBpipe?

vectorDBpipe is a production-ready, open-source Python SDK that unifies the entire modern RAG (Retrieval-Augmented Generation) stack into a single, intelligent pipeline.

Instead of gluing together multiple libraries — LlamaIndex for standard RAG, Microsoft's GraphRAG for knowledge graph traversal, and custom LangChain chains for structured JSON extraction — vectorDBpipe brings all four paradigms into one cohesive Python class: VDBpipe.

With a single pip install, you get:

  • ✅ 4 AI Engines that automatically route based on query type
  • ✅ Tri-Processing Ingestion that builds vectors, page indexes, and knowledge graphs simultaneously
  • ✅ 15+ Native Data Loaders for PDFs, S3, Notion, Slack, GitHub, and more
  • ✅ Zero configuration required — works out of the box with sensible defaults
from vectorDBpipe import VDBpipe

pipeline = VDBpipe()
pipeline.ingest("data/contracts/")
answer = pipeline.query("What is the penalty for late payment?")

That's genuinely all it takes.

🌟 Why vectorDBpipe?

The Real Problem

The modern RAG landscape is deeply fragmented. A production AI application needs:

  • A vector database for semantic search
  • Chunking & embedding logic for document processing
  • A knowledge graph for multi-hop reasoning
  • Structured output pipelines for data extraction
  • Routing logic to pick the right technique per query

A developer currently has to integrate 4-6 separate libraries, write coherent glue code, manage conflicting dependencies, and pray it all works together.

Our Solution: One SDK, Four Engines, Zero Fragmentation

┌─────────────────────────────────────────────────────────────────┐
│                          VDBpipe SDK                            │
│  ╔═══════════════════════════════════════════════════════════╗  │
│  ║               OmniRouter (Intelligent Dispatch)           ║  │
│  ╚═════════════╦══════════════╦═══════════╦══════════════════╝  │
│                ▼              ▼           ▼           ▼         │
│  ┌─────────────┐  ┌──────────┐  ┌───────────┐  ┌──────────────┐│
│  │  Engine 1   │  │ Engine 2 │  │ Engine 3  │  │   Engine 4   ││
│  │ Vector RAG  │  │Vectorless│  │ GraphRAG  │  │LangChain     ││
│  │  (Fast)     │  │  (Deep)  │  │(Detective)│  │Extract (JSON)││
│  └─────────────┘  └──────────┘  └───────────┘  └──────────────┘│
└─────────────────────────────────────────────────────────────────┘

🆕 What's New in v0.2.4

This is a production-readiness release with 11 major improvements, zero breaking changes.

✨ New Features

Feature Description
🧠 Semantic OmniRouter Replaced keyword matching with cosine-similarity embedding routing. Queries are embedded and scored against pre-computed intent prototypes per engine (threshold = 0.35). Falls back to keywords when no embedder is configured.
💾 Graph + PageIndex Persistence _persist_state() auto-saves the knowledge graph and page index as JSON after every ingest(). _load_state() restores them on startup — data now survives restarts.
🌊 Streaming LLM Responses BaseLLMProvider.stream_response() added with a default wrapper for all providers. OpenAILLMProvider implements real SSE token streaming. VDBpipe.stream_query() is a new generator method. New POST /pipelines/chat/stream SSE endpoint.
📄 PPTX Loader .pptx files now supported via python-pptx. Text extracted slide-by-slide and ingested like any other document.
✂️ Sentence-Boundary Chunking chunk_text_sentences() in utils/common.py — groups sentences into chunks respecting boundaries, with configurable overlap. Eliminates mid-sentence splits from fixed-size chunking.
🏗️ VDBpipe Pure Composition VDBpipe no longer inherits from TextPipeline. It is a standalone class with providers as plain instance attributes. The _safe_reinit() hack is deleted.
🖥️ TUI Improvements System Doctor runs 6 real runtime checks. Setup Wizard error screen fixed (was silently ignored). API key validation added before saving config.yaml.
🧪 39 Unit Tests Test suite expanded from 4 → 39 tests across 12 classes. All mocked — no API keys or GPU needed.

🐛 Bug Fixes

  • File upload isolation: uploads now go to data/<user_id>/<uuid>_filename (no collisions)
  • Backend pipeline cache evicted on config update
  • finishSetup() write error now shows error screen instead of silent fail

🔥 The 4 Omni-RAG Engines

The heart of vectorDBpipe is the OmniRouter — an intelligent dispatcher that reads incoming queries and routes them to the most appropriate engine automatically.

Engine 1 — Vector RAG (Fast Factual Lookup) ⚡

Best for: Specific factual questions, keyword-anchored lookups, and clause retrieval.

Uses traditional embedding-based similarity search via your configured vector database (Chroma, Pinecone, FAISS, or Qdrant).

# Automatically triggers Engine 1
result = pipeline.query("What is the termination clause in section 14?")

Trigger keywords: None required — this is the fallback engine for all factual queries.

Engine 2 — Vectorless RAG / PageIndex (Deep Reading) 📖

Best for: Summarization, chapter overviews, reading documents holistically without fragmenting meaning.

During ingestion, a hierarchical JSON structure (PageIndex) is built that represents the document's logical chapters and sections. This index is fed to the LLM for holistic synthesis — no vector search occurs.

# Automatically triggers Engine 2 when summarization is detected
result = pipeline.query("Summarize the overall structure and key themes of this document.")
result = pipeline.query("Give me an overview of chapter 3.")

Trigger keywords: summarize, overall, chapter, overview, holistic

Engine 3 — GraphRAG (Multi-hop Reasoning Detective) 🕸️

Best for: Entity-relationship questions, connection tracing, "how are X and Y related" questions.

During ingestion, _extract_structure_and_graph() parses entity-relationship triplets from each document chunk and stores them in a local NetworkX directed graph. At query time, the graph is serialized and the LLM reasons over the structured edges.

# Automatically triggers Engine 3 when relationship reasoning is detected
result = pipeline.query("How is the CEO connected to the board of directors?")
result = pipeline.query("What is the relationship between Clause 5 and Clause 12?")

Trigger keywords: connected, relationship, how is, between, linked

Engine 4 — LangChain Structured Extract (JSON Output) 🧩

Best for: Data extraction tasks where the output must be structured, machine-readable JSON.

Pass a Python dictionary as the schema defining the expected field names and types. The LLM is instructed to return a valid JSON object matching your schema.

# Always call .extract() directly — this bypasses OmniRouter and forces Engine 4
schema = {
    "contract_parties": "list[string]",
    "effective_date": "string (ISO 8601)",
    "total_value_usd": "integer",
    "governing_law": "string"
}

data = pipeline.extract(
    query="Extract all key metadata from these contracts.",
    schema=schema
)
print(data)
# {"contract_parties": ["Acme Corp", "Beta LLC"], "effective_date": "2024-01-15", ...}

⚙️ Tri-Processing Ingestion Pipeline

When you call pipeline.ingest(path), three parallel processes are launched via concurrent.futures.ThreadPoolExecutor:

                    ┌─────────────────────┐
                    │  pipeline.ingest()  │
                    └──────────┬──────────┘
                               │
               ┌───────────────┼───────────────┐
               │               │               │
               ▼               ▼               ▼
    ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
    │  PHASE 1     │  │  PHASE 2     │  │  PHASE 3     │
    │ Vector Chunk │  │ Structural   │  │ Graph Entity │
    │ + Embed      │  │ PageIndex    │  │ Extraction   │
    │              │  │ JSON Build   │  │ NetworkX Edge│
    └──────┬───────┘  └──────┬───────┘  └──────┬───────┘
           │                 │                 │
           ▼                 ▼                 ▼
    ┌──────────┐      ┌──────────┐      ┌──────────┐
    │ VectorDB │      │page_index│      │  graph   │
    │(Chroma/  │      │  dict    │      │(NetworkX)│
    │ Pinecone)│      │          │      │          │
    └──────────┘      └──────────┘      └──────────┘

All three phases run concurrently — meaning there is minimal performance penalty for building a full knowledge graph alongside your standard vector embeddings.

🔌 15+ Native Data Integrations

DataLoader supports reading from virtually any source. Simply pass a path or URI:

Local File Loaders

File Type Extension Library
Plain Text .txt Built-in
PDF Documents .pdf PyMuPDF (fitz)
Word Documents .docx docx2txt
PowerPoint .pptx python-pptx ✨ New in v0.2.4
CSV Spreadsheets .csv Built-in
JSON Files .json Built-in
HTML Pages .html, .htm BeautifulSoup4
Markdown .md markdown + BeautifulSoup4
XML Documents .xml BeautifulSoup4

Cloud & Web Loaders

Source URI Format Library
Web URL https://example.com requests + BeautifulSoup4
AWS S3 Bucket s3://bucket/file.pdf boto3
Google Drive gdrive://file_id google-api-python-client

SaaS Connectors

Platform URI Format Status
Notion notion://page_id ✅ Connector Ready
Confluence confluence://space_key ✅ Connector Ready
Slack slack://channel_id ✅ Connector Ready
GitHub github://owner/repo ✅ Connector Ready
Jira jira://project_key ✅ Connector Ready 
# Examples of different data sources
pipeline.ingest("data/report.pdf")          # Local PDF
pipeline.ingest("data/wiki/")              # Entire directory of files
pipeline.ingest("https://example.com")     # Live Web Page
pipeline.ingest("s3://my-bucket/data/")    # S3 Bucket
pipeline.ingest("notion://abc123page")     # Notion Page
pipeline.ingest("github://openai/gpt-4")  # GitHub Repository

🏛️ Architecture Overview

vectorDBpipe/
│
├── vectorDBpipe/                    # 📦 Core Python SDK Package
│   │
│   ├── __init__.py                  # Entry point (VDBpipe)
│   │
│   ├── pipeline/
│   │   ├── vdbpipe.py               # ⭐ VDBpipe: pure composition, 4 engines, Semantic OmniRouter, persistence
│   │   └── text_pipeline.py         # TextPipeline: legacy (kept for compatibility)
│   │
│   ├── data/
│   │   └── loader.py                # DataLoader: 15+ sources incl. PPTX (v0.2.4)
│   │
│   ├── embeddings/                  # Embedding provider wrappers
│   │   ├── sentence_transformers.py
│   │   ├── openai_embeddings.py
│   │   └── cohere_embeddings.py
│   │
│   ├── llms/                        # LLM provider wrappers
│   │   ├── base.py                  # BaseLLMProvider + stream_response() (v0.2.4)
│   │   ├── openai_client.py         # OpenAI — real SSE streaming (v0.2.4)
│   │   ├── sarvam_client.py
│   │   ├── anthropic_client.py
│   │   └── groq_client.py
│   │
│   ├── vectordb/                    # Vector database connectors
│   │   ├── chroma_db.py
│   │   ├── pinecone_db.py
│   │   ├── faiss_db.py
│   │   └── qdrant_db.py
│   │
│   ├── config/
│   │   └── config_manager.py        # YAML + ENV configuration loader
│   │
│   ├── utils/
│   │   └── common.py                # clean_text, chunk_text, chunk_text_sentences (v0.2.4)
│   │
│   └── logger/
│       └── logging.py               # Structured logging setup
│
├── tests/
│   └── test_vdbpipe.py              # 🧪 39-test PyTest suite (v0.2.4), all mocked
│
├── .github/
│   └── workflows/
│       ├── ci.yml                   # CI: pytest on every push/PR to main
│       └── publish-to-pypi.yml      # CD: publish to PyPI on GitHub Release
│
├── CHANGELOG.md
├── config.yaml
├── requirements.txt
├── setup.py
└── MANIFEST.in

🖥 TUI — Terminal Interface

The vectorDBpipe TUI is published separately on npm and auto-installs the Python package.

# Install globally
npm install -g vectordbpipe-tui

# Launch
vdb

The TUI provides:

  • Setup Wizard — configure embedder, vector DB, and LLM with API key validation
  • System Doctor — 6 live runtime checks (Node.js, Python, pip package, config, internet, VectorDB)
  • Ingest — point at a folder or file, watch Tri-Processing run live
  • Chat — query with the Semantic OmniRouter; see which engine answered
  • Graph View — explore the extracted knowledge graph
  • PageIndex — browse the structural document index
  • Structured Extract — run Engine 4 with a custom JSON schema

The TUI communicates directly with the installed vectordbpipe Python package via vdb_runner.py — no backend server required.

📦 Installation

From PyPI (Recommended)

pip install vectordbpipe

From Source (Latest Development)

# Clone the repository
git clone https://github.com/yashdesai023/vectorDBpipe.git
cd vectorDBpipe

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate           # Linux / macOS
venv\Scripts\activate              # Windows

# Install in editable mode with all dependencies
pip install -e .

GPU-Accelerated Install (CUDA)

If you have an NVIDIA GPU with CUDA support:

pip install vectordbpipe
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

CPU-Only Install (Older Hardware / Laptops)

pip install vectordbpipe
pip install torch==2.3.1+cpu torchvision==0.18.1+cpu --index-url https://download.pytorch.org/whl/cpu

⚙️ Configuration

All pipeline settings are controlled via config.yaml. Copy the file from the repository and fill in your own API keys.

# config.yaml

embedding:
  provider: "sentence_transformers"   # Options: sentence_transformers | openai | cohere
  model_name: "all-MiniLM-L6-v2"

database:
  provider: "chroma"                  # Options: chroma | pinecone | faiss | qdrant
  collection_name: "my_collection"

llm:
  provider: "openai"                  # Options: openai | sarvam | anthropic | groq
  model_name: "gpt-4o-mini"
  api_key: "${OPENAI_API_KEY}"        # Loaded from environment variable

# Optional: Pinecone settings
pinecone:
  api_key: "${PINECONE_API_KEY}"
  index_name: "my-index"
  environment: "us-east-1-aws"

Set environment variables in .env or your shell:

export OPENAI_API_KEY="sk-..."
export PINECONE_API_KEY="pcsk-..."

Or load from .env automatically:

from dotenv import load_dotenv
load_dotenv()

from vectorDBpipe import VDBpipe
pipeline = VDBpipe()

🚀 Quickstart Guide

Step 1 — Initialize the pipeline

from vectorDBpipe import VDBpipe

# Uses config.yaml + environment variables automatically
pipeline = VDBpipe()

# Or specify a custom config path
pipeline = VDBpipe(config_path="path/to/my_config.yaml")

Step 2 — Ingest your data

# Ingest a single file
pipeline.ingest("data/contract.pdf")

# Ingest an entire directory (recursive)
pipeline.ingest("data/documents/")

# Ingest from a web URL
pipeline.ingest("https://en.wikipedia.org/wiki/Artificial_intelligence")

# Ingest from an S3 bucket
pipeline.ingest("s3://my-company-bucket/legal/contracts/")

# Ingest from Notion
pipeline.ingest("notion://my-workspace-page-id")

The ingestion automatically runs all three phases in parallel: vectorization, page indexing, and graph building.

Step 3 — Query using intelligent routing

# OmniRouter automatically picks the best engine
response = pipeline.query("What is the total contract value?")
print(response)

# Force Engine 2 (deep holistic reading)
response = pipeline.query("Summarize the entire document in bullet points.")

# Force Engine 3 (graph-based multi-hop reasoning)
response = pipeline.query("How is the revenue connected to the acquisition in Q4?")

Step 4 — Extract structured data (Engine 4)

schema = {
    "company_names": "list of strings",
    "effective_date": "ISO 8601 date string",
    "total_value_usd": "integer",
    "jurisdiction": "string",
    "penalty_clauses": "list of strings"
}

contract_data = pipeline.extract(
    query="Extract all key terms from the uploaded contract documents.",
    schema=schema
)

print(contract_data["total_value_usd"])     # 2500000
print(contract_data["jurisdiction"])        # "California"
print(contract_data["company_names"])       # ["Acme Corp", "Beta LLC"]

🔬 Advanced Usage

Using with Google Colab

vectorDBpipe runs perfectly on Google Colab. Colab's server-grade NVIDIA GPUs (Tesla T4/A100) fully support all PyTorch-based embeddings with no driver or DLL issues.

# Cell 1 — Install
!pip install vectordbpipe

# Cell 2 — Run
from vectorDBpipe import VDBpipe

pipeline = VDBpipe()
pipeline.ingest("https://your-document-url.com/report.pdf")
print(pipeline.query("What are the key risk factors?"))

Multi-Source Ingestion

sources = [
    "data/Q1_report.pdf",
    "data/Q2_report.pdf",
    "https://company.com/annual-report",
    "s3://data-lake/contracts/",
]

for source in sources:
    pipeline.ingest(source)

# Now query across all ingested sources
result = pipeline.query("Compare Q1 and Q2 revenues.")

Override Configuration at Runtime

pipeline = VDBpipe(config_override={
    "llm": {
        "provider": "groq",
        "model_name": "llama3-8b-8192",
        "api_key": "your-groq-key"
    }
})

Accessing the Knowledge Graph Directly

# Access the underlying NetworkX graph
graph = pipeline.graph

# List all entities extracted
nodes = list(graph.nodes())
print(f"Entities found: {nodes}")

# View all relationships
for u, v, data in graph.edges(data=True):
    print(f"  {u}  →[{data['relation']}]→  {v}")

Accessing the PageIndex Directly

# Access the structural document index
page_index = pipeline.page_index

for source, structure in page_index.items():
    print(f"Document: {source}")
    print(f"  Chapters: {structure.get('chapters')}")
    print(f"  Summary: {structure.get('summary')[:150]}...")

📚 API Reference

VDBpipe(config_path, config_override)

The main orchestrator class (pure composition — does not inherit TextPipeline as of v0.2.4).

Parameter Type Default Description
config_path str "config.yaml" Path to the YAML config file
config_override dict None Override any config key at runtime

Attributes:

Attribute Type Description
pipeline.graph nx.DiGraph The local NetworkX knowledge graph
pipeline.page_index dict The hierarchical document structure index
pipeline.embedder EmbeddingProvider The active embedding provider
pipeline.vector_store VectorDBProvider The active vector database
pipeline.llm LLMProvider The active language model

pipeline.ingest(data_path, batch_size=100)

Tri-processing ingestion of any supported data source.

Parameter Type Default Description
data_path str Required File path, directory, URL, S3 URI, or SaaS URI
batch_size int 100 Number of chunks per embedding batch

Returns: int — Total number of chunks embedded.

pipeline.query(user_query)

Intelligent query routing via the OmniRouter. Automatically selects the correct engine.

Parameter Type Description
user_query str Your natural language question

Returns: str — The LLM-generated answer.

pipeline.extract(query, schema)

Forces structured output using Engine 4 (LangChain Extract).

Parameter Type Description
query str What information to extract
schema dict[str, str] Field names mapped to type descriptions

Returns: dict — JSON-parsed structured output.

pipeline.stream_query(user_query) ✨ New in v0.2.4

Streaming generator — yields LLM response tokens one at a time.

for token in pipeline.stream_query("What are the key risks?"):
    print(token, end="", flush=True)

pipeline._route_query(query) (internal)

Returns the engine code for a given query string using cosine-similarity semantic routing (v0.2.4+).

Return Value Engine
"ENGINE_1" Vector RAG
"ENGINE_2" Vectorless / PageIndex RAG
"ENGINE_3" GraphRAG

chunk_text_sentences(text, max_tokens, overlap_sentences) ✨ New in v0.2.4

Sentence-boundary sliding-window chunker. Avoids splitting sentences mid-context.

from vectorDBpipe.utils.common import chunk_text_sentences

chunks = chunk_text_sentences(text, max_tokens=400, overlap_sentences=1)

⚡ Performance Benchmarks

All tests performed on Python 3.10 | Ubuntu 22.04 | 8-core CPU | 16GB RAM using:

  • LLM: gpt-4o-mini
  • Embeddings: all-MiniLM-L6-v2
  • Vector DB: ChromaDB (local)
Metric Value
Standard Ingestion (1M tokens) ~1.8 mins
Ingestion throughput improvement (v0.2.0 vs v0.1.x) +40% faster (ThreadPoolExecutor)
Engine 1 — Vector RAG latency ~45 ms
Engine 2 — Vectorless RAG latency ~200 ms
Engine 3 — GraphRAG latency ~350 ms
Engine 4 — LangChain Extract latency ~500 ms
PyTest suite execution time ~21 s
Memory usage (local ChromaDB, 10K docs) ~1.4 GB

🧪 Running Tests

The test suite uses pytest with unittest.mock to isolate the pipeline from LLM/API calls. No API keys are needed to run the tests.

Install test dependencies

pip install pytest pytest-cov
# or
pip install -r requirements_dev.txt

Run all tests

python -m pytest tests/test_vdbpipe.py -v

Expected output (v0.2.4):

tests/test_vdbpipe.py::TestInitialization::test_has_required_attributes        PASSED [  2%]
tests/test_vdbpipe.py::TestInitialization::test_is_not_text_pipeline_subclass  PASSED [  5%]
tests/test_vdbpipe.py::TestIngestion::test_ingest_sets_loader_path             PASSED [ 10%]
tests/test_vdbpipe.py::TestOmniRouter::test_summarize_routes_to_engine_2       PASSED [ 17%]
tests/test_vdbpipe.py::TestOmniRouter::test_relationship_routes_to_engine_3    PASSED [ 23%]
tests/test_vdbpipe.py::TestEngine1VectorRAG::test_returns_llm_answer           PASSED [ 33%]
tests/test_vdbpipe.py::TestEngine2VectorlessRAG::test_returns_llm_answer       PASSED [ 41%]
tests/test_vdbpipe.py::TestEngine3GraphRAG::test_populated_graph_calls_llm     PASSED [ 51%]
tests/test_vdbpipe.py::TestEngine4StructuredExtract::test_returns_parsed_json  PASSED [ 56%]
tests/test_vdbpipe.py::TestSentenceChunking::test_no_mid_sentence_splits       PASSED [ 76%]
tests/test_vdbpipe.py::TestPPTXLoader::test_pptx_load_extracts_slide_text      PASSED [ 82%]
tests/test_vdbpipe.py::TestPersistence::test_persist_state_creates_files       PASSED [ 84%]
tests/test_vdbpipe.py::TestStreaming::test_stream_query_yields_tokens          PASSED [ 94%]
... (39 total)

=================== 39 passed in 206.59s (0:03:26) ===================

Run with coverage report

python -m pytest tests/ --cov=vectorDBpipe --cov-report=html -v

Test descriptions

Test Description
test_vdbpipe_initialization Verifies the constructor sets up the NetworkX graph, PageIndex, and all required attributes.
test_vdbpipe_ingest_tri_processing Mocks DataLoader.load_data() and verifies all three ingestion phases run correctly.
test_omnirouter_classification Tests the _route_query() logic for all three engine routing paths.
test_vector_rag_engine Verifies _engine_1_vector_rag() chains correctly to query_with_llm().

🤝 Contributing

Contributions are warmly welcomed! Please follow these steps:

  1. Fork the repository on GitHub
  2. Create a feature branch: git checkout -b feature/your-feature-name
  3. Make your changes with clear, descriptive commits
  4. Run the tests: python -m pytest tests/ -v
  5. Push your branch: git push origin feature/your-feature-name
  6. Open a Pull Request targeting the main branch

Contribution Areas

  • Production OAuth wiring for SaaS connectors (Notion, Slack, GitHub)
  • Async ingestion support via asyncio
  • Qdrant and Weaviate vector database integrations
  • More embedding providers (Cohere, Google Vertex)
  • Graph persistence to Neo4j / TigerGraph
  • Streaming support for Anthropic and Groq providers

Code Style

  • Follow PEP 8
  • All new public methods must have docstrings
  • All new features must have corresponding test cases in tests/

📜 Changelog

v0.2.4 — Production Readiness Release (March 2026) ⭐ Latest

11 improvements: Semantic OmniRouter, Persistence, Streaming, PPTX, Sentence Chunking, VDBpipe refactor, Backend upgrade, 39 tests, TUI real diagnostics.

New:

  • Semantic OmniRouter — embedding cosine-similarity routing replaces keyword matching
  • Graph + PageIndex Persistence — auto-save/load on every ingest (JSON)
  • Streaming — stream_response() on all LLMs; OpenAI real SSE; /chat/stream SSE endpoint
  • PPTX Loader — .pptx files via python-pptx
  • Sentence-boundary chunking — chunk_text_sentences() in utils/common.py
  • VDBpipe pure composition — no more TextPipeline inheritance, _safe_reinit deleted
  • 39-test suite (was 4) — all mocked, no API keys required

Fixed:

  • File upload isolation (per-user UUID prefix)
  • Setup Wizard now shows error screen on write failure
  • API key validated before saving config

v0.2.3 — Hotfix (February 2026)

Hotfix — Missing llms/__init__.py caused ImportError on all LLM providers after PyPI install.

Fixed:

  • Added missing __init__.py to vectorDBpipe/llms/
  • Pinned chromadb>=0.5.0 for PersistentClient API compatibility

v0.2.2 — Critical Hotfix (March 2026)

Resolves critical pipeline initialization and engine routing bugs.

Fixed:

  • Embedder 'NoneType' object has no attribute 'tokenize' — TextPipeline was using the legacy model.name config key instead of the new embedding.model_name. This caused SentenceTransformer(None) to be created, crashing all ingestion and queries. _safe_reinit now completely bypasses legacy keys and reinitializes all providers from embedding, database, and llm config directly.
  • LLM not initialized with config_override — Added missing sarvam, google, and cohere LLM provider support to _safe_reinit. Sarvam users were silently getting self.llm = None even with a valid API key configured.
  • Graph always empty (0 nodes) after ingestion — Graph extraction was 100% LLM-gated with no fallback. Added _regex_graph_extract() that uses regex pattern matching to extract entity relationships (X is Y, X has Y, etc.) when no LLM is configured.
  • Corrupted PDF crash (FzErrorFormat) — _load_pdf now loads pages by index with per-page try/except, skipping broken pages gracefully instead of crashing the entire ingestion.
  • Engine 2/3/4 returning "LLM not configured" — All three engines now return useful, readable fallback content without an LLM. Engine 2 returns formatted PageIndex structure; Engine 3 returns filtered graph edges plus vector search; Engine 4 returns a helpful config snippet.
  • Engine 3 returning irrelevant graph output — GraphRAG now filters edges by query keywords, shows a clear "No direct match" note, and transparently supplements with vector search when the graph has no matching entities.
  • generate_response() signature mismatch — All engine calls now correctly pass the retrieved_context argument to the LLM provider interface.

v0.2.0 — Omni-RAG Architecture (February 2026)

Major Release — Complete architectural overhaul introducing the 4-engine Omni-RAG stack.

New:

  • VDBpipe orchestrator class with OmniRouter, 4 AI engines, and Tri-Processing ingestion
  • GraphRAG engine backed by NetworkX (Engine 3)
  • Vectorless RAG engine backed by hierarchical PageIndex (Engine 2)
  • LangChain Extract engine for Pydantic JSON output (Engine 4)
  • DataLoader rewritten with 15+ source integrations
  • ThreadPoolExecutor parallel ingestion pipeline
  • PyTest test suite with 4 core unit tests (all mocked, zero API key requirement)

Changed:

  • Migrated from langchain_core.pydantic_v1 → standard pydantic
  • TextPipeline attributes renamed: embedding → embedder, db → vector_store
  • setup.py bumped to version 0.2.0

Fixed:

  • OmniRouter misclassification on ambiguous queries
  • load_data() method API correctly takes no arguments (path set as attribute)
  • Package exclusion rules for TUI, Frontend, and Backend directories

v0.1.0 — Initial Release

  • Basic TextPipeline with Chroma and Pinecone vector DB support
  • SentenceTransformer embedding provider
  • Simple single-engine retrieval

📄 License

MIT License

Copyright (c) 2026 Yash Desai

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

Built with ❤️ by Yash Desai for the AI Development Community.

⭐ Star the repo • 🐛 Report a Bug • 💬 Discussions

If this project saves you hours of glue code, consider giving it a ⭐ on GitHub!

About

A modular text embedding and vector database pipeline for local and cloud vector stores. Designed to streamline text preprocessing, embedding generation, and semantic search with multiple backends such as FAISS, Chroma, and Pinecone.

Topics

python pypi-package vector-database chromadb pineco

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 16

v0.2.4 — Semantic OmniRouter + Persistence + Streaming + PPTX + 39 Tests Latest
Mar 3, 2026
+ 15 releases

Packages

 
 
 

Contributors

Languages