RAG Process Chatbot

A high-performance Retrieval-Augmented Generation (RAG) chatbot designed for processing enterprise documents with role-based access control. Accelerated by NVIDIA GPUs (RTX A5000).

🚀 Features

• Multi-Format Ingestion: Supports PDF (text & scanned/OCR), DOCX, XLSX, XLS, CSV, TXT, and PPTX.
• Role-Based Access: Restrict document visibility to specific user roles (e.g., Retail, All Access).
• GPU Acceleration: Utilizes CUDA-enabled embeddings for rapid document indexing and retrieval.
• Admin Dashboard: Comprehensive management interface to monitor ingestion status, manage documents, and view system health.
• Enterprise-Grade Security: Integrated SSL/TLS support with FastAPI and production-ready architecture.
• Hybrid Retrieval: Combines semantic vector search (ChromaDB) with keyword search for pinpoint accuracy.

🛠️ Tech Stack

• Backend: FastAPI (Python)
• Frontend: Vanilla JS, HTML5, CSS3 (Modern Glassmorphic UI)
• Vector Database: ChromaDB
• Embeddings: Sentence-Transformers (BGE-Small/v1.5)
• AI Models: Groq Cloud API for Chat, Local GPU for Embeddings & OCR.

📂 Project Structure

• /rag: Core RAG logic, loaders, and configurations.
• /data: Source documents organized by role folders.
• /chroma_db: Persistent vector database (excluded from Git).
• server.py: Main API server and secure endpoint handling.
• admin.html: Project administration panel.
• frontend.html: Chatbot user interface.

📄 Documentation

• Detailed installation and server setup instructions can be found in SETUP.md.

⚖️ License

Internal Use Only - IIFL Samasta

Evaluation Workflow

The repository now includes an eval/ package for retrieval-quality measurement, PDF-level ranking, and LLM-as-judge scoring.

What's Included

• eval/logger.py: logs top-k retrievals for each query into the shared SQLite database table retrieval_logs.
• eval/metrics.py: pure functions for recall@k, precision@k, and MRR.
• eval/pdf_ranker.py: aggregates chunk rankings into PDF rankings with configurable strategies.
• eval/llm_judge.py: scores faithfulness, groundedness, and relevance with the existing LM Studio client.
• eval/run_eval.py: CLI entrypoint that runs retrieval metrics, PDF ranking, and judge scoring, then writes JSON and CSV reports.
• eval/build_dataset.py: scaffolds a labelable dataset from recent retrieval logs.
• eval/dataset.json: starter dataset file for labeled evaluation queries.

Retrieval Logging

Production chat queries now log their retrieval candidates automatically. Each log record stores:

• query_id
• query_text
• timestamp
• user_role
• k
• latency_ms
• normalized retrieved chunk metadata (chunk_id, source_pdf, page_number, rank, similarity_score)

Build a Dataset

Use recent retrieval logs to scaffold examples for manual labeling:

python -m eval.build_dataset --limit 50

That writes eval/dataset_from_logs.json. Fill in relevant_chunk_ids and/or relevant_pdfs for each query.

Run Evaluation

Run the full evaluation pipeline with:

python -m eval.run_eval

Useful flags:

• --skip-judge to run only retrieval and PDF ranking metrics.
• --top-k 5 to change how many chunks are used for answer generation during evaluation.
• --pdf-strategy max_score to switch PDF aggregation.
• --log-retrievals to also persist eval-run retrievals into retrieval_logs.

Reports are written to eval/results/ as timestamped JSON and CSV files.