LLM Evaluation Platform

An open-source toolkit for evaluating Large Language Model (LLM) responses using automated quality metrics, regression testing, prompt comparison, and customizable evaluation pipelines.

Overview

LLM Evaluation Platform helps developers measure the quality of AI-generated responses instead of relying solely on manual review.

The project provides reusable evaluation pipelines that score responses based on multiple quality dimensions including:

Accuracy
Relevance
Faithfulness
Completeness
Hallucination Risk
Consistency
Response Similarity

The toolkit supports automated regression testing, prompt experimentation, and evaluation reporting for AI-powered applications.

Why?

Traditional software testing verifies deterministic outputs.

Large Language Models generate probabilistic responses.

Testing LLMs therefore requires a completely different workflow.

Instead of checking:

Input
↓

Expected Output

↓

Pass

LLM evaluation becomes:

Question
        │
        ▼
LLM Response
        │
        ▼
Evaluation Pipeline
        │
 ┌──────┼────────┐
 │      │        │
 ▼      ▼        ▼
Accuracy Relevance Hallucination
        │
        ▼
Quality Score

This project automates that process.

Features

Response Evaluation

Evaluate AI responses using configurable metrics.

Accuracy
Relevance
Completeness
Faithfulness
Fluency
Hallucination Detection
Semantic Similarity

Prompt Comparison

Compare multiple prompts against the same dataset.

Example:

Prompt	Overall Score
Prompt A	81.4
Prompt B	90.2
Prompt C	87.1

Automatically determine the highest-performing prompt.

Regression Testing

Prevent prompt regressions after updates.

Version 1
     │
     ▼
Prompt Tests

Version 2
     │
     ▼
Prompt Tests

↓

Compare Scores

↓

Regression Report

Dataset Evaluation

Evaluate hundreds or thousands of prompts automatically.

Supported formats:

CSV
JSON
JSONL

Example CSV

question,expected_answer
What is Python?,Programming language
Capital of Japan?,Tokyo
Largest planet?,Jupiter

Custom Metrics

Create your own evaluation metrics.

Example:

class ToxicityMetric(BaseMetric):

    def evaluate(self, response):

        return score

Batch Evaluation

Evaluate entire datasets.

python evaluate.py \
--dataset data/questions.csv \
--model gpt-4 \
--output report.json

Report Generation

Automatically generate:

Markdown Reports
HTML Reports
JSON Reports
CSV Reports

Architecture

                   Dataset

                      │

                      ▼

             Prompt Generator

                      │

                      ▼

               LLM Provider

          OpenAI / Claude / Gemini

                      │

                      ▼

             Generated Response

                      │

                      ▼

            Evaluation Pipeline

       ┌─────────┬──────────┬──────────┐

       ▼         ▼          ▼

 Accuracy   Relevance   Hallucination

       │         │          │

       └─────────┴──────────┘

                ▼

          Final Quality Score

                ▼

         HTML / JSON Report

Technology Stack

Backend

Python 3.11+
FastAPI
Pydantic
Pandas
NumPy

AI

OpenAI API
Anthropic Claude
Google Gemini
Ollama

Testing

pytest
pytest-cov

Database

SQLite
PostgreSQL

Visualization

Plotly
Matplotlib

Project Structure

llm-evaluation-platform/

│

├── app/

│   ├── api/

│   ├── evaluation/

│   ├── metrics/

│   ├── prompts/

│   ├── reports/

│   ├── services/

│   └── utils/

│

├── datasets/

├── reports/

├── tests/

├── examples/

├── docs/

├── scripts/

└── README.md

Installation

Clone the repository

git clone https://github.com/ForwardAva/llm-evaluation-platform.git

Enter project

cd llm-evaluation-platform

Create virtual environment

python -m venv .venv

Activate

Windows

.venv\Scripts\activate

Linux / macOS

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

Quick Start

Set your API key

OPENAI_API_KEY=your_key

Run evaluation

python evaluate.py

Expected output

Accuracy ............ 94%

Relevance ........... 96%

Faithfulness ........ 91%

Hallucination ....... Low

Overall Score ....... 93.7

Next Section

REST API
CLI Commands
Configuration
Evaluation Metrics
Screenshots
Contributing
Roadmap
License
FAQ

REST API

Start the server

uvicorn app.main:app --reload

Default URL

http://localhost:8000

Interactive API Documentation

http://localhost:8000/docs

ReDoc

http://localhost:8000/redoc

API Endpoints

Health Check

GET /health

Response

{
  "status": "healthy",
  "version": "1.0.0"
}

Evaluate Response

POST /api/v1/evaluate

Request

{
  "question": "What is Artificial Intelligence?",
  "expected_answer": "Artificial Intelligence is...",
  "generated_answer": "AI is..."
}

Response

{
  "overall_score": 91.8,
  "metrics": {
    "accuracy": 94,
    "relevance": 96,
    "faithfulness": 90,
    "completeness": 88,
    "hallucination": 0.04
  }
}

Compare Prompts

POST /api/v1/prompts/compare

Response

{
  "winner": "Prompt B",
  "scores": [
    {
      "prompt":"Prompt A",
      "score":86.2
    },
    {
      "prompt":"Prompt B",
      "score":92.4
    }
  ]
}

Batch Evaluation

POST /api/v1/evaluate/batch

Input

[
  {
    "question":"...",
    "expected":"..."
  }
]

Output

{
  "processed":250,
  "average_score":90.7
}

Command Line Interface

Evaluate one dataset

python evaluate.py \
--dataset datasets/general.csv \
--model gpt-4

Generate HTML report

python evaluate.py \
--dataset datasets/general.csv \
--report html

Compare prompts

python compare.py \
--prompt prompts/v1.txt \
--candidate prompts/v2.txt

Generate regression report

python regression.py

Run all evaluations

python run_all.py

Configuration

Example .env

OPENAI_API_KEY=

ANTHROPIC_API_KEY=

GOOGLE_API_KEY=

DEFAULT_MODEL=gpt-4

DATABASE_URL=sqlite:///database.db

REPORT_FORMAT=html

MAX_CONCURRENT_REQUESTS=5

Supported Models

Provider	Supported
GPT-4	✅
GPT-4.1	✅
GPT-4o	✅
Claude 3	✅
Gemini	✅
Ollama	✅
Local LLMs	✅

Evaluation Metrics

Metric	Description
Accuracy	Measures factual correctness
Relevance	Measures whether the answer addresses the question
Completeness	Measures coverage of expected information
Faithfulness	Measures grounding in supplied context
Hallucination	Detects unsupported claims
Consistency	Compares repeated generations
Semantic Similarity	Embedding similarity score

Sample Evaluation Report

Question

"What is Retrieval-Augmented Generation?"

────────────────────────────────────

Model

GPT-4

────────────────────────────────────

Accuracy

96%

Relevance

98%

Faithfulness

93%

Completeness

95%

Hallucination Risk

Low

Overall Score

95.5

Status

PASS

Example Workflow

Dataset

↓

Generate Responses

↓

Evaluate Responses

↓

Generate Scores

↓

Generate Reports

↓

Review Failures

↓

Improve Prompt

↓

Run Again

Example Directory

datasets/

├── finance.csv

├── healthcare.csv

├── legal.csv

├── coding.csv

└── general.csv

Reports

Generated reports include

HTML
Markdown
CSV
JSON

Example

reports/

├── evaluation.html

├── evaluation.md

├── evaluation.csv

└── evaluation.json

Testing

Run unit tests

pytest

Run coverage

pytest --cov

Run integration tests

pytest tests/integration

Performance

Average evaluation speed

Dataset Size	Time
10	< 5 sec
100	< 40 sec
500	< 3 min
1000	< 7 min

Screenshots

Replace these placeholders with your own screenshots.

/docs/images/dashboard.png

/docs/images/report.png

/docs/images/evaluation.png

/docs/images/api.png

Roadmap

Version 1.0

Version 1.1

PDF report export
Docker support
Model benchmarking
Authentication

Version 2.0

Contributing

Contributions are welcome.

Fork the repository
Create a feature branch

git checkout -b feature/my-feature

Commit changes

git commit -m "Add new evaluation metric"

Push

git push origin feature/my-feature

Open a Pull Request

License

Distributed under the MIT License.

See the LICENSE file for more information.

Acknowledgements

This project was inspired by the growing need for reliable evaluation tools for Large Language Models and AI-powered applications.

Special thanks to the open-source AI community for advancing evaluation methodologies and responsible AI engineering.

Author

Oscar Garcia

AI Quality Engineer • LLM Evaluation Specialist • AI Application Developer

⭐ Support

If you find this project useful, consider giving it a ⭐ on GitHub.

It helps others discover the project and supports future development.

Built with ❤️ for the AI Engineering and LLM Evaluation community.

Name		Name	Last commit message	Last commit date
data		data
output		output
qa_generator_outputs		qa_generator_outputs
src		src
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
sample_ranked_b7c6163791d24cb082c6407163185b04.json		sample_ranked_b7c6163791d24cb082c6407163185b04.json

Latest commit

History

Folders and files

Repository files navigation

LLM Evaluation Platform

Overview

Why?

Features

Response Evaluation

Prompt Comparison

Regression Testing

Dataset Evaluation

Custom Metrics

Batch Evaluation

Report Generation

Architecture

Technology Stack

Backend

AI

Testing

Database

Visualization

Project Structure

Installation

Quick Start

Next Section

REST API

API Endpoints

Health Check

Evaluate Response

Compare Prompts

Batch Evaluation

Command Line Interface

Configuration

Supported Models

Evaluation Metrics

Sample Evaluation Report

Example Workflow

Example Directory

Reports

Testing

Performance

Screenshots

Roadmap

Version 1.0

Version 1.1

Version 2.0

Contributing

License

Acknowledgements

Author

⭐ Support

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages