🧠 Markdown RAG Knowledgebase

A Retrieval-Augmented Generation (RAG) system that transforms large Markdown documentation sets into an AI-powered chatbot knowledgebase.
It ingests 2000+ .md files, cleans and embeds them into a vector database (ChromaDB), and serves semantic Q&A via FastAPI and Streamlit.

---

🚀 Features

• Markdown ingestion with YAML front-matter parsing
• Content cleaning, chunking, and hashing for reproducible indexing
• Vector embeddings using SentenceTransformers
• Persistent storage via ChromaDB (local or Docker)
• FastAPI backend with REST endpoints for ingestion and querying
• Streamlit chat UI for interactive Q&A with source citations
• BM25 re-ranking for precision improvement
• Subset selection script to identify the highest-impact Markdown files
• Optional OpenAI GPT integration for natural answers with citations

---

🧩 Architecture

markdown-rag-knowledgebase/
│
├── src/
│   ├── api.py                     # FastAPI backend
│   └── pipeline/
│       ├── ingest.py              # Markdown parsing and cleaning
│       ├── store.py               # Chroma vector store client
│       └── rag.py                 # Retriever, BM25, and LLM logic
│
├── scripts/
│   ├── ingest.py                  # Batch ingestion for full dataset
│   └── select_subset.py           # Rank Markdown files for MVP subset
│
├── web/
│   └── app.py                     # Streamlit chat interface
│
├── data/                          # Vector DB persistence & reports
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
└── README.md

---

⚙️ Installation

1. Clone the repository

git clone https://github.com/yourusername/markdown-rag-knowledgebase.git
cd markdown-rag-knowledgebase

2. Create and activate a virtual environment

python -m venv venv
source venv/bin/activate   # macOS/Linux
venv\Scripts\activate    # Windows

3. Install dependencies

pip install -r requirements.txt

4. (Optional) Add OpenAI key for LLM responses

export OPENAI_API_KEY=your_openai_api_key

---

🧱 Build the Knowledgebase

Ingest Markdown files

python scripts/ingest.py --md_dir path/to/markdown --collection kb --persist_dir data/chroma

This will:

• Parse and clean .md, .markdown, and .mdx files
• Chunk and embed content
• Store vectors in a ChromaDB collection
• Create a report in data/ingest_report.csv

---

💬 Run the API

uvicorn src.api:app --host 0.0.0.0 --port 8001

API Endpoints

Endpoint	Method	Description
/health	GET	Health check
/stats	GET	Returns DB stats
/ingest_dir	POST	Ingest all Markdown files in a directory
/query	POST	Ask semantic questions

Example query

curl -X POST "http://localhost:8001/query"      -H "Content-Type: application/json"      -d '{"question": "How to install ExampleProduct?", "top_k": 5}'

---

🖥️ Launch the Chat Interface

streamlit run web/app.py

Access at: http://localhost:8501

Chat UI Features

• Ask natural questions about your docs
• Get AI answers with citations
• Inspect original Markdown sources

---

🧮 Select Initial MVP Subset

Prioritize high-impact files for your first knowledgebase iteration:

python scripts/select_subset.py --md_dir path/to/markdown --top_n 200 --out data/subset.json

Outputs a ranked JSON of key docs based on keyword relevance and metadata.

---

🐳 Docker Deployment

Run everything (API + UI) with one command:

docker compose up --build

Then visit:

• API → http://localhost:8001
• UI → http://localhost:8502

---

🧠 Tech Stack

Component	Technology
Programming	Python 3.11
Backend	FastAPI
Frontend	Streamlit
Vector DB	ChromaDB
Embeddings	SentenceTransformers
LLM (optional)	OpenAI GPT models
Ranking	BM25

---

🔒 Environment Variables

Variable	Description	Default
OPENAI_API_KEY	Optional key for LLM answers	–
EMBEDDING_MODEL	Embedding model name	all-MiniLM-L6-v2
PERSIST_DIR	Vector DB path	data/chroma
COLLECTION	Chroma collection name	kb
API_BASE	Base URL for Streamlit API calls	http://localhost:8001

---

🧠 Example Workflow

1. Place your Markdown documentation in a folder
2. Ingest it into the knowledgebase
3. Start the API server
4. Launch the Streamlit chat UI
5. Ask product, guide, or compliance questions — get context-based answers with citations

---

🧩 Future Enhancements

• Pinecone / Weaviate backend option
• LangChain and LlamaIndex pipeline support
• Incremental updates and file watchers
• Admin analytics dashboard
• Multi-tenant knowledgebase support

---

👨‍💻 Author

Tayyub Yaqoob
AI & Data Solutions Engineer
LinkedIn | GitHub

---

📝 License

MIT License

Copyright (c) 2025 Tayyub Yaqoob

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.