A Retrieval-Augmented Generation (RAG) based Document Intelligence System for PDF Question Answering with Semantic Search, Citations, Confidence Scoring, and Guardrails.
A lightweight Retrieval-Augmented Generation (RAG) based document intelligence system that enables users to upload PDF documents, ask natural language questions, and receive grounded answers with citations and confidence scoring.
The system emphasizes accuracy, reliability, and modular architecture through semantic retrieval, retrieval confidence validation, source citations, and guardrails that reduce unsupported responses.
The system supports two response modes:
- Answer – concise grounded answers
- Answer + Insights – answer followed by evidence-based insights and recommendations
The implementation is intentionally optimized for Accuracy rather than latency.
- Upload multiple PDF documents
- Automatic PDF parsing
- Text chunking with overlap
- Sentence Transformer embeddings
- FAISS vector indexing
- Natural language queries
- Semantic retrieval
- Context-aware answer generation
- Source citations
- Retrieval confidence score
- Processing time measurement
- Retrieval validation layer
- Confidence propagation
- Guardrails against unsupported answers
- Duplicate document detection
- Hash-based duplicate prevention
- Comprehensive error handling
- Browser-based interface
- PDF upload
- Question answering
- Citation display
- Confidence visualization
The architecture follows a modular Retrieval-Augmented Generation (RAG) pipeline where each component is independently responsible for ingestion, retrieval, validation, generation, and response delivery.
PDF Upload
│
▼
PDF Parser
│
▼
Text Chunking
│
▼
Embedding Generation
│
▼
FAISS Vector Store
│
▼
Semantic Retrieval
│
▼
Retrieval Validator
│
▼
Prompt Construction
│
▼
FLAN-T5 Generator
│
▼
Answer + Citations + Confidence
app/
│
├── api/
├── core/
├── schemas/
├── services/
├── utils/
│
frontend/
│
architecture/
│
uploads/
vector_db/
logs/
README.md
requirements.txt
| Category | Technology |
|---|---|
| Backend | FastAPI |
| Frontend | HTML • CSS • JavaScript |
| Embeddings | Sentence Transformers |
| Vector Database | FAISS |
| Language Model | Google FLAN-T5 Base |
| PDF Parsing | PyMuPDF |
| ML Framework | HuggingFace Transformers |
- Upload PDF documents
↓
- Parse PDF text
↓
- Chunk documents
↓
- Generate embeddings
↓
- Store vectors in FAISS
↓
- Ask natural language question
↓
- Retrieve Top-K relevant chunks
↓
- Validate retrieval confidence
↓
- Construct grounded prompt
↓
- Generate response using FLAN-T5
↓
- Return
- Answer
- Citations
- Confidence Mode
- Confidence Score
- Processing Time
This implementation intentionally prioritizes Accuracy, as the primary objective of a Document Intelligence System is to provide reliable, evidence-backed answers rather than simply responding quickly.
To improve answer reliability, the system introduces multiple validation layers including semantic retrieval, retrieval confidence validation, grounded prompt construction, citations, and guardrails before answer generation.
This design reduces unsupported responses, increases user trust, and provides greater transparency by exposing both confidence scores and source citations with every generated answer.
Each pipeline stage is implemented as an independent service (Parser, Chunker, Embeddings, Retriever, Generator, Validator), making the system maintainable and easily extensible.
Instead of sending retrieved chunks directly to the LLM, a Retrieval Validator evaluates retrieval quality, computes similarity-based confidence, and propagates confidence to the final response.
The language model only receives retrieved document context and is explicitly instructed not to use external knowledge.
Every answer includes filename, page number, and source preview to improve explainability and user trust.
- Selected FLAN-T5 Base to enable fully local inference without external APIs.
- Used FAISS for simplicity and fast semantic retrieval on moderate document collections.
- Chose fixed-size chunking with overlap for predictable retrieval performance.
- Chose CPU-based inference, making deployment easier while increasing response latency compared to GPU inference.
- Built a lightweight HTML/CSS/JavaScript frontend to minimize dependencies and keep the project simple.
- Avoided external services to keep the project lightweight and reproducible.
The system handles:
- Empty uploads
- Upload limit exceeded
- Non-PDF files
- Duplicate documents
- Empty document parsing
- Empty retrieval results
- Irrelevant questions
- Embedding generation failures
- Vector database failures
- File save failures
- Invalid request modes
If insufficient evidence is available, the system returns:
"I could not find sufficient information in the uploaded documents."
instead of generating unsupported answers.
The current implementation is suitable for moderate document collections and single-user workloads.
Potential scaling challenges include:
- Large document collections (10k+ PDFs)
- High concurrent user requests
- CPU-only LLM inference
- Single-node FAISS vector database
For production-scale deployments, distributed vector databases, hybrid retrieval, GPU inference, and asynchronous processing would be recommended.
A production deployment could introduce:
- Distributed vector databases
- Hybrid retrieval
- Cross-encoder reranking
- GPU inference
- Async ingestion
- Response caching
- Stronger instruction-tuned LLMs (Llama, Gemma, Mistral)
- OCR support for scanned PDFs
- Streaming responses
- Docker deployment
- Authentication
- Cloud object storage
- Monitoring dashboard
- Multi-user document collections
- Upgrade to a larger instruction-tuned LLM to improve response quality and structured answer generation while reusing the existing RAG pipeline.
git clone <repository-url>
cd document-intelligence-system
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000Start the frontend:
cd frontend
python -m http.server 5500Open:
http://localhost:5500
-
Upload one or more PDF documents.
-
Choose:
- Answer
- Answer + Insights
- Ask a natural language question.
The system returns:
- Grounded answer
- Source citations
- Confidence
- Confidence score
- Processing time
What cloud services does Microsoft Azure provide?
Answer
Azure provides cloud computing services including compute,
storage, networking, AI, analytics, security, and hybrid cloud
solutions based on the uploaded document context.
Confidence
High (0.81)
Sources
2025_MicrosoftAnnualReport.pdf
Page 13
2025_MicrosoftAnnualReport.pdf
Page 14
- Improve retrieval quality using hybrid search and reranking.
- Replace the lightweight local LLM with a stronger instruction-tuned model.
- Scale the system using distributed vector databases and GPU inference.
- Implement semantic response caching for repeated or similar queries to reduce latency and improve throughput.
- Add Docker deployment, authentication, and cloud storage support.
-
This project uses Google FLAN-T5 Base, a lightweight instruction-tuned model selected to enable fully local, offline inference without external API dependencies.
-
The retrieval, validation, citation, and guardrail pipeline provides grounded evidence to the model. Final answer quality is therefore influenced by the capabilities of the underlying LLM rather than the retrieval architecture itself.
-
The architecture is intentionally model-agnostic, allowing stronger instruction-tuned models (Llama 3, Gemma, Mistral, etc.) to replace the generation component with minimal code changes.
-
The Answer + Insights mode is more dependent on instruction-following ability. A stronger local or hosted LLM would significantly improve structured insight generation while preserving the same retrieval pipeline.