RAG Eval is an open-source toolkit for evaluating Retrieval-Augmented Generation (RAG) applications. It provides a comprehensive framework for assessing both retrieval and generation quality, helping developers build more accurate, reliable, and trustworthy AI systems. By measuring retrieval effectiveness, context relevance, answer faithfulness, and response quality, RAG Eval makes it easier to identify weaknesses and optimize every stage of a RAG pipeline.
Whether you're developing AI-powered search, question-answering systems, knowledge assistants, or enterprise chatbots, RAG Eval offers configurable evaluation workflows and detailed reports to support continuous improvement.
- Evaluate retrieval accuracy and document relevance
- Measure answer faithfulness and groundedness
- Assess response quality using customizable metrics
- Compare prompts, embedding models, and retrievers
- Benchmark multiple LLMs and RAG pipelines
- Automated batch evaluation for large datasets
- Generate detailed reports and performance summaries
- JSON-based configuration for flexible evaluation workflows
- Export results for further analysis and visualization
- Retrieval-Augmented Generation (RAG) evaluation
- Enterprise knowledge assistants
- AI-powered search applications
- Question-answering systems
- Prompt engineering and optimization
- Embedding model comparison
- Regression testing for LLM applications
- AI quality assurance and benchmarking
- Python
- LangChain
- Vector Databases
- OpenAI API
- JSON
- Pandas
- NumPy
- Pytest
rag-eval/
├── evaluators/
├── metrics/
├── retrievers/
├── reports/
├── datasets/
├── configs/
├── tests/
├── examples/
├── requirements.txt
└── README.md
git clone https://github.com/your-username/rag-eval.git
cd rag-eval
pip install -r requirements.txtpython evaluate.py \
--dataset datasets/sample.json \
--config configs/default.jsonRAG Eval supports multiple evaluation dimensions, including:
- Retrieval Precision
- Context Relevance
- Answer Correctness
- Faithfulness
- Groundedness
- Response Completeness
- Response Consistency
- Latency and Throughput
Contributions are welcome! Feel free to submit issues, feature requests, or pull requests to help improve the project.
This project is licensed under the MIT License.