Enterprise Search Assistant

A Next-Generation Multi-Document Retrieval-Augmented Generation (RAG) Platform

---

Overview

The Enterprise Search Assistant is a state-of-the-art intelligent document processing and search engine. It synthesizes large repositories of unstructured data into a highly cohesive knowledge graph and provides semantic search capabilities powered by leading Large Language Models.

Designed for high performance, reliability, and precision, this application employs a hybrid retrieval strategy, blending sparse BM25 keyword matching with dense vector embeddings to ensure pinpoint accuracy in context retrieval.

🎥 Watch the Demo Video Presentation on Google Drive to see the system in action!

---

1. Architecture Overview

This platform is structured as an enterprise-grade monorepo containing three core packages:

• client: A modern React 19 visual interface built using Vite and Tailwind CSS.
• server: A robust, asynchronous Express backend serving streaming search, ingestion, and analytics endpoints.
• shared: Shared type definitions used to establish a unified protocol between frontend and backend.

Technical Data Flow & Pipelines

flowchart TD
    %% Styling
    classDef pipeline fill:#1a1b26,stroke:#7aa2f7,stroke-width:2px,color:#c0caf5;
    classDef database fill:#1e1e2e,stroke:#f5c2e7,stroke-width:2px,color:#cdd6f4;
    classDef engine fill:#11111b,stroke:#a6e3a1,stroke-width:2px,color:#cdd6f4;

    %% Ingestion Pipeline
    subgraph Ingestion["1. INGESTION PIPELINE"]
        Upload["📂 Multi-Format File Upload (PDF, DOCX, TXT, MD)"] --> Parse["⚙️ Parser Factory (pdf-parse / mammoth)"]
        Parse --> Chunk["✂️ Semantic Chunker (400-char window, 50-char overlap)"]
        
        Chunk --> DBStore["💾 SQL Transactions (SQLite - WAL Mode)"]
        Chunk --> LexStore["🔍 Lexical indexing (In-Memory BM25)"]
        Chunk --> DenseStore["⚡ Dense Vector Storage (Google Embedding API)"]
    end

    %% Search and RAG Pipeline
    subgraph Search["2. 3-STAGE HYBRID RETRIEVAL PIPELINE"]
        Query["💬 User Query"] --> Stage1A["🔍 Stage 1A: Sparse Search (BM25 Engine)"]
        Query --> Stage1B["⚡ Stage 1B: Dense Vector Search (ChromaDB Cosine Sim)"]
        
        Stage1A --> Stage2["⚖️ Stage 2: Reciprocal Rank Fusion (RRF, k=60)"]
        Stage1B --> Stage2
        
        Stage2 --> Stage3["📈 Stage 3: LLM Cross-Encoder Reranking (Groq API)"]
        Stage3 --> Context["📥 High-Relevance Content Contextualizer"]
    end

    %% Generation Pipeline
    subgraph Generation["3. GENERATION & FAILOVER ENGINE"]
        Context --> Route{"🔀 Failover Router"}
        Route -->|Primary| GroqLLM["⚡ Groq Inference (Llama-3.3-70b)"]
        Route -->|Failover / Rate Limit| GeminiLLM["🧠 Gemini 2.0 Flash"]
        
        GroqLLM --> Stream["🌊 Server-Sent Events (SSE) Stream"]
        GeminiLLM --> Stream
        Stream --> ClientUI["💻 Sleek Glassmorphism Client Dashboard"]
    end

    %% Apply Classes
    class Upload,Parse,Chunk,Stage1A,Stage1B,Stage2,Stage3,Context pipeline;
    class DBStore,LexStore,DenseStore database;
    class Route,GroqLLM,GeminiLLM,Stream,ClientUI engine;

Loading

---

2. Interactive Dashboard Showcase

The frontend features a stunning glassmorphism design system utilizing a dark-mode theme, sleek micro-animations, and dynamic data-binding.

Feature	Visual Demonstration	Description
Interactive Landing Portal		Modern visual entry portal introducing user capabilities, integrated mock query showcases, and dynamic fast-access entry points.
Q&A Assistant & Search Hub		Full-featured Q&A assistant demonstrating live SSE query streaming, instant citation verification, and standard preset quick-access questions.
Dynamic Knowledge Graph		Asynchronously extracted from documents and visualized dynamically using a D3-Force directed graph. Interact with nodes (entities) and links (relationships).
Comprehensive Analytics Dashboard		Monitors indexing loads, total relational entity extractions, document types composition (donut chart), and named entity frequency counts (bar chart).
Knowledge Base Document Hub		Clean multi-format document card library indicating metadata parameters, processing completion statuses, and tag taxonomies.

---

3. Core Design Decisions

• SQLite for Relational & Lexical Persistence

  • Decision: Employs better-sqlite3 operating in Write-Ahead Logging (WAL) mode as the primary transactional datastore.
  • Rationale: Eliminates the overhead of managing a separate database cluster in development environments while providing lightning-fast reads/writes, transactional integrity, and direct cascade deletes.

• Custom Dual-Engine Search Strategy (BM25 + ChromaDB Fallback)

  • Decision: Developed a customized BM25 keyword matching engine in TypeScript alongside ChromaDB dense vector lookups.
  • Rationale: Render's free tier and serverless environments frequently experience cold starts, or may lack full support for multi-container vector database hosting. If ChromaDB connection times out, the search orchestrator falls back silently and instantly to BM25 search. This guarantees 100% search uptime and resilient offline testing capabilities.

• Isolated Frontend Typings for Cloud Builds

  • Decision: Isolated and duplicated core shared interfaces directly into client/src/types/index.ts.
  • Rationale: Serverless deployments like Vercel build monorepo packages in isolated build contexts, often preventing access to sibling directories. Self-containing client types ensures Vercel compiles frontend assets successfully without needing backend references.

• Server-Sent Events (SSE) for Real-Time Streaming

  • Decision: Implemented SSE (text/event-stream) over WebSockets for real-time generative streaming.
  • Rationale: SSE is native to HTTP/1.1 and HTTP/2, extremely lightweight, operates cleanly through standard reverse proxies, and handles reconnection automatically without WebSocket connection management overhead.

• Dual LLM Adaptive Failover Routing

  • Decision: Created an abstraction layer routing primary requests through the Groq SDK (llama-3.3-70b-versatile), and automatically falling back to Gemini (gemini-2.0-flash) on error or rate-limiting.
  • Rationale: Combines Groq's sub-second token generation times and native rerank capabilities with Gemini's massive context window and stability, resulting in optimal speed and 100% system availability.

---

4. Practical Engineering Assumptions

During the ideation, design, and implementation phases of this platform, several critical engineering assumptions and trade-offs were deliberately made:

1. Hybrid Retrieval Dominance over Dense-Only RAG

   • Assumption: Dense vector embeddings excel at capturing high-level semantic themes, but fail dramatically on exact-match identifier queries, numeric serial parameters, or specific software tags.
   • Impact: We assumed that a production-grade enterprise RAG engine cannot rely on vector search alone. By pairing dense semantic retrieval with custom sparse BM25 keyword indexes and applying Reciprocal Rank Fusion (RRF), we guarantee high-precision retrieval across both structural identifiers and natural phrasing.

2. Infrastructure Resilience and Vector Fallback

   • Assumption: High-capacity vector databases (like ChromaDB or Pinecone) hosted in standard cloud environments are highly prone to connection timeouts, cold-start delays, or rate limits. We assumed the system must be completely resilient to external DB failures.
   • Impact: Rather than allowing database connection failures to crash the backend or disable search, the platform assumes ChromaDB is an optional enhancement. If ChromaDB times out or is unreachable, the system silently and instantly defaults to our custom in-memory BM25 lexical engine, ensuring 100% search uptime.

3. Monorepo Compile Boundaries in Serverless Builds

   • Assumption: Modern deployment platforms (e.g. Vercel) compile frontend codebases in strictly sandboxed, isolated build workspaces. If the client code tries to import common types from a shared project root folder (../../../shared), compilation will fail.
   • Impact: We assumed that the frontend client codebase must remain 100% self-contained for Vercel compilation. The React app references its own isolated typescript types (client/src/types/index.ts), avoiding cross-folder dependencies while maintaining structural contract symmetry with the Express server.

4. Large-Context LLMs as Dynamic Entity Extractors

   • Assumption: Traditional rule-based NLP extraction (like regex libraries) is highly brittle and fails to capture structural contextual connections across multi-domain enterprise docs.
   • Impact: We assumed that leveraging modern LLMs with high-concurrency JSON schemas is the most reliable, precise, and scale-ready way to parse unstructured text into clean nodes and relationships for the D3 Knowledge Graph.

5. Lightweight Streaming with Server-Sent Events (SSE)

   • Assumption: Streaming real-time LLM answers token-by-token using heavy bidirectional WebSockets introduces substantial state management, connection handling overhead, and friction with standard reverse proxies (like Nginx or Cloudflare).
   • Impact: We assumed unidirectional SSE (text/event-stream) over standard HTTP/1.1 or HTTP/2 is the most stable and performant strategy for streaming response tokens, simplifying proxy routing and handling automatic reconnections out-of-the-box.

6. Decoupled Ephemeral Storage for Serverless Environments

   • Assumption: Free-tier cloud runtime nodes (like Render) spin down on inactivity and use ephemeral filesystems, meaning any local files (uploaded documents, SQLite database) are lost on restarts.
   • Impact: We designed the backend to be completely decoupled. Local folders and SQLite transactions are used for zero-config local bootup, but the ingestion and relational database schemas are modularly structured to swap immediately to persistent stores (like PostgreSQL, Chroma Cloud, and AWS S3) via standard environment variables.

7. Monolingual Corpus Boundaries

   • Assumption: The target enterprise documents are monolingual (primarily English).
   • Impact: This assumption allowed us to build highly optimized tokenizers for BM25 and chunking boundaries without introducing heavy multi-language tokenizing systems, keeping the application fast and lightweight.

---

5. Comprehensive API Reference

Document Operations

1. Upload Documents

• Endpoint: POST /api/documents/upload
• Content-Type: multipart/form-data
• Request Schema:
  • files: File list (Up to 10 files; allowed extensions: .pdf, .docx, .txt, .md).
• Response Payload (201 Created):

  {
    "success": true,
    "data": [
      {
        "id": "c7a7a72d-8bde-4d7a-8f55-bfa30f4ee336",
        "filename": "Q1_Financial_Report.pdf",
        "type": "pdf",
        "size": 1048576,
        "status": "parsing"
      }
    ]
  }

2. Get Document List

• Endpoint: GET /api/documents
• Response Payload (200 OK):

  {
    "success": true,
    "data": [
      {
        "id": "c7a7a72d-8bde-4d7a-8f55-bfa30f4ee336",
        "filename": "c7a7a72d-8bde-4d7a-8f55-bfa30f4ee336.pdf",
        "originalName": "Q1_Financial_Report.pdf",
        "type": "pdf",
        "size": 1048576,
        "uploadedAt": "2026-05-27T12:00:00.000Z",
        "status": "complete",
        "analytics": {
          "summary": "Executive summary of quarterly growth and operation metrics.",
          "topics": ["Revenue Growth", "Infrastructure Spend", "Risk Factors"],
          "tags": ["Finance", "Q1", "2026"],
          "entities": [],
          "wordCount": 1420,
          "pageCount": 4,
          "readingTimeMinutes": 6
        }
      }
    ]
  }

3. Delete Document

• Endpoint: DELETE /api/documents/:id
• Response Payload (200 OK):

  {
    "success": true,
    "message": "Document deleted"
  }

---

Search & Retrieval Operations

1. Block Semantic Search

• Endpoint: POST /api/search
• Content-Type: application/json
• Request Schema:

  {
    "query": "What is our target revenue growth for Q2?",
    "model": "auto",
    "topK": 5,
    "documentIds": ["c7a7a72d-8bde-4d7a-8f55-bfa30f4ee336"],
    "sessionId": "s8d8e8b3-219e-4e3a-b855-bfad0f5ee891"
  }

• Response Payload (200 OK):

  {
    "success": true,
    "data": {
      "answer": {
        "text": "Based on the Q1 financial report, the target revenue growth for Q2 is 15%.",
        "modelUsed": "groq"
      },
      "sources": [
        {
          "chunkId": "chunk-102",
          "documentId": "c7a7a72d-8bde-4d7a-8f55-bfa30f4ee336",
          "documentName": "Q1_Financial_Report.pdf",
          "content": "Our Q2 projection marks a target revenue expansion of 15% quarter-over-quarter...",
          "pageNumber": 2,
          "score": 0.94
        }
      ],
      "sessionId": "s8d8e8b3-219e-4e3a-b855-bfad0f5ee891",
      "queryId": "q8c9a3d2-310b-47e1-8f55-cfa90f7ee123",
      "processingTimeMs": 285
    }
  }

2. Streaming Search (SSE)

• Endpoint: GET /api/search/stream?query=your+query&model=auto
• Headers Returned:
  • Content-Type: text/event-stream
  • Cache-Control: no-cache
• Event Messages Formats:
  • Sources event: data: {"type":"sources","data":[...]}
  • Token event: data: {"type":"token","data":"Hello"}
  • Done event: data: {"type":"done","data":""}

3. Cross-Document Comparison

• Endpoint: POST /api/compare
• Content-Type: application/json
• Request Schema:

  {
    "documentIds": ["doc-id-1", "doc-id-2"],
    "query": "Compare operating revenue metrics.",
    "model": "groq"
  }

---

Knowledge Graph & Analytics

1. Get Global Knowledge Graph

• Endpoint: GET /api/knowledge-graph
• Response Payload (200 OK):

  {
    "success": true,
    "data": {
      "nodes": [
        { "id": "e-101", "name": "Google", "type": "organization", "val": 4 },
        { "id": "e-102", "name": "Q1 Report", "type": "concept", "val": 2 }
      ],
      "links": [
        { "source": "e-101", "target": "e-102", "type": "PUBLISHED_BY", "confidence": 0.9 }
      ]
    }
  }

2. Get Analytics Overview

• Endpoint: GET /api/analytics/overview
• Response Payload (200 OK):

  {
    "success": true,
    "data": {
      "totalDocuments": 4,
      "totalChunks": 240,
      "totalEntities": 85,
      "totalQueries": 18,
      "documentsByType": {
        "pdf": 2,
        "docx": 1,
        "txt": 1,
        "md": 0
      },
      "topEntities": [
        { "name": "Enterprise Corp", "type": "organization", "count": 12 }
      ],
      "recentQueries": [
        { "query": "What is our operational growth rate?", "timestamp": "2026-05-27T12:30:15.000Z", "model": "groq" }
      ]
    }
  }

---

6. Local Setup & Quickstart Guide

Follow these exact commands to clone the repository and run the application on your local machine instantly.

Prerequisites

• Node.js: v18.0.0 or higher installed.
• Git: Installed on your system.

Quickstart Installation

1. Clone the repository:

   git clone https://github.com/AyushCoder9/Multi-Document-Enterprise-Search-Assistant.git
   cd Multi-Document-Enterprise-Search-Assistant

2. Install all dependencies: (This single command automatically installs packages for the frontend, backend, and shared libraries simultaneously).

   npm install

3. Configure Environment Variables:

   cp .env.example .env

   Open the newly created .env file in your code editor and insert your GROQ_API_KEY and GEMINI_API_KEY.

4. Launch the Application in Development Mode:

   npm run dev

The Express API backend will initialize on port 3001, and the React frontend will open instantly at http://localhost:5173.

---

Engineered for unparalleled data retrieval precision and architectural resilience.