██████╗ ██████╗ ███████╗███╗ ██╗███████╗██╗ ██╗██████╗ ██╗
██╔══██╗██╔══██╗██╔════╝████╗ ██║██╔════╝██║ ██║██╔══██╗██║
██║ ██║██████╔╝█████╗ ██╔██╗ ██║█████╗ ██║ ██║██████╔╝██║
██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║██╔══╝ ██║ ██║██╔══██╗██║
██████╔╝██║ ███████╗██║ ╚████║███████╗╚██████╔╝██║ ██║███████╗
╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝
Zero Judge-LLM Cost • 100% Reproducible Verdicts • Zero API Latency • Instant CI Gating
"Every existing evaluation tool evaluates what the agent said. OpenEval evaluates what the agent did — deterministically, with zero judge-LLM cost and zero flakiness."
- 💡 Executive Overview & Vision
- ⚡ Performance & Benchmark Comparison
- ⚔️ OpenEval vs. Traditional LLM-as-a-Judge
- 🏗️ System Architecture & Execution Pipeline
- 📦 Installation Matrix
- 🚀 Quickstart Tutorials (4 Adapters)
- 📊 Mathematical Specification of All 7 Deterministic Metrics
- 🖥️ Command Line Interface (CLI) Guide
- ⚙️ GitHub Actions CI/CD Gating
- 🔑 Core Data Model API Reference
- 📖 Repository Structure & Proposals
- ❓ Frequently Asked Questions (FAQ)
- 📄 License
OpenEval is a high-speed, purely deterministic evaluation engine designed specifically for AI agent trajectories.
As AI agents transition from simple single-prompt text generators to complex autonomous tool-calling loops (interacting with file systems, databases, payment APIs, and web search), evaluating their behavior requires inspecting actual execution traces.
OpenEval bypasses prompt-based LLM judges entirely. It consumes structured execution steps (TraceStep) and evaluates exact tool calls, argument correctness, trajectory step efficiency, final state transitions, permission denial recoveries, Verified Replay divergences, and environment fingerprint staleness using 100% pure mathematical Python functions.
Benchmark executed on 1,000 synthetic agent evaluation cases:
| Benchmark Metric | OpenEval ⚡ | LLM-as-a-Judge (GPT-4o) 🐢 | Performance Advantage |
|---|---|---|---|
| Execution Throughput | > 15,000 evals / sec | ~ 0.2 evals / sec | 75,000x Faster |
| Average Latency | < 0.08 milliseconds | 2,500 - 8,000 milliseconds | Instant Evaluation |
| API Cost per 1k Evals | $0.00 (Zero) | $15.00 - $60.00 | 100% Free |
| Variance / Flakiness | 0.0% (Deterministic) | 8.5% - 14.2% Non-Deterministic | Flawless Reproducibility |
| Network Dependency | 100% Offline Capable | Requires Cloud Internet | Zero Network Overhead |
Traditional Evaluation (LLM-as-a-Judge):
[ Agent Run ] ──> [ Secondary Prompt ] ──> [ GPT-4 API Call ] ──> [ Flaky Text Verdict ($$$) ]
OpenEval Deterministic Engine:
[ Agent Trace ] ──> [ Pure Python Logic ] ──> [ Instant Math Score (1.0 / 0.0 / None) ($0) ]
+---------------------------------------------------------------------------------------------------+
| INPUT TRAJECTORY INGESTION |
| +--------------------------+ +---------------------------+ +--------------------------+ |
| | Fixtura .trace File | | LangChain Run Tree | | OpenAI Messages List | |
| | (Compressed JSONL) | | (Traced Agent Runs) | | (ChatCompletion API) | |
| +------------+-------------+ +-------------+-------------+ +------------+-------------+ |
+----------------|--------------------------------|-------------------------------|-----------------+
| | |
v v v
+---------------------------------------------------------------------------------------------------+
| ADAPTER SUBSYSTEM |
| +--------------------------+ +---------------------------+ +--------------------------+ |
| | from_fixtura_trace() | | from_langchain_run() | | from_openai_messages() | |
| +------------+-------------+ +-------------+-------------+ +------------+-------------+ |
+----------------|--------------------------------|-------------------------------|-----------------+
+--------------------------------+-------------------------------+
|
v
+---------------------------------------------------------------------------------------------------+
| CORE UNIFIED DATA MODEL (Pure Dataclasses) |
| |
| AgentTrace |
| ├── task_id: str |
| ├── input: str | final_output: str | actual_state: dict | metadata: dict |
| └── steps: list[TraceStep] |
| ├── step_id: int | type: "thought" | "tool_call" | "tool_result" |
| ├── tool_name: str | tool_args: dict | tool_result: str |
| ├── denied: bool (Permission denial / validation error tracking) |
| ├── finish_reason: str | provider: str | model: str | tokens: dict | latency_ms: float |
| └── divergent: bool (Verified Replay trajectory divergence marker) |
+-------------------------------------------------|-------------------------------------------------+
|
v
+---------------------------------------------------------------------------------------------------+
| PURE DETERMINISTIC EVAL ENGINE |
| |
| EvalTestCase |
| ├── expected_tool_calls: list[dict] | expected_final_state: dict |
| └── max_steps: int | timeout_seconds: float |
| |
| +-------------------------------------------------------------------------------------------+ |
| | METRIC PIPELINE MODULES | |
| | 1. ToolSelectionAccuracy (Exact tool invocation ratio) | |
| | 2. ArgumentCorrectness (Key-value schema match precision) | |
| | 3. StepEfficiency (Optimal vs actual steps ratio) | |
| | 4. GoalCompletionRate (Final state transition accuracy) | |
| | 5. DenialRecoveryRate (Permission denial recovery without loops) | |
| | 6. DivergenceScore (Verified Replay trajectory step fidelity) | |
| | 7. FixtureFreshness (Environment fingerprint staleness check) | |
| +---------------------------------------------+---------------------------------------------+ |
+-------------------------------------------------|-------------------------------------------------+
|
v
+---------------------------------------------------------------------------------------------------+
| VERDICT & REPORT GENERATOR |
| +---------------------------+ +---------------------------+ +-----------------------+ |
| | MetricResult | --> | Suite Runner | --> | CLI & GitHub Action | |
| | (score, passed, details) | | (openeval.runner) | | (openeval.report) | |
| | passed: True/False/None | | Excludes passed=None | | Pure Markdown/JSON | |
| +---------------------------+ +---------------------------+ +-----------------------+ |
+---------------------------------------------------------------------------------------------------+
OpenEval requires zero external framework dependencies to run its core engine:
# 1. Core Engine (Zero external dependencies — pure Python)
pip install openeval-core
# 2. With Native Fixtura Trace Support (.trace compressed JSONL)
pip install "openeval-core[fixtura]"
# 3. With LangChain Run Adapter Support
pip install "openeval-core[langchain]"
# 4. With All Optional Adapters Installed
pip install "openeval-core[fixtura,langchain]"
# 5. Local Editable Installation (From Source Checkout)
pip install -e ".[fixtura]"Note
The PyPI distribution package name is openeval-core, while the CLI executable command is openeval.
The core engine runs anywhere without external frameworks or SDK dependencies:
from openeval.metrics import ToolSelectionAccuracy, ArgumentCorrectness
from openeval.models import AgentTrace, EvalTestCase, TraceStep
# 1. Define expectations (what the agent was supposed to do)
test_case = EvalTestCase(
task_id="quickstart-1",
input="Search for weather in Tokyo",
expected_tool_calls=[{"tool": "search", "args": {"query": "weather in Tokyo"}}],
expected_final_state={"searched": True},
expected_output_contains=[],
max_steps=5,
timeout_seconds=10.0
)
# 2. Provide actual trace (what the agent actually executed)
trace = AgentTrace(
task_id="quickstart-1",
input="Search for weather in Tokyo",
steps=[
TraceStep(
step_id=1,
type="tool_call",
content="",
tool_name="search",
tool_args={"query": "weather in Tokyo"},
tool_result="85 degrees and sunny",
timestamp=0.0
)
],
final_output="The weather in Tokyo is 85 degrees and sunny.",
actual_state={"searched": True},
metadata={}
)
# 3. Score deterministically (0.0ms execution time)
metric = ToolSelectionAccuracy()
result = metric.score(trace, test_case)
print(f"Metric: {result.metric_name}")
print(f"Score: {result.score} (Passed: {result.passed})")
print(f"Details: {result.details}")Ingest zstd-compressed .trace files recorded by Fixtura. OpenEval natively parses permission denials, completion finish reasons, token counts, Verified Replay divergence markers, and fingerprint drift verdicts:
from openeval.adapters.fixtura import from_fixtura_trace
from openeval.metrics import (
ToolSelectionAccuracy,
DenialRecoveryRate,
DivergenceScore,
FixtureFreshness
)
# Ingest Fixtura trace file
trace = from_fixtura_trace(
trace_path="fixtures/checkout.trace",
task_id="task-101",
input_text="Execute user checkout",
final_output="Order placed successfully",
actual_state={"order_created": True},
metadata={}
)
# Evaluate against test case
metrics = [
ToolSelectionAccuracy(),
DenialRecoveryRate(),
DivergenceScore(),
FixtureFreshness()
]
for m in metrics:
res = m.score(trace, test_case)
print(f"{m.name:25s}: Score = {res.score:.2f} | Passed = {res.passed!s:5s} | {res.details}")Convert LangChain Run trees directly into OpenEval traces:
from openeval.adapters.langchain import from_langchain_run
from langchain_core.tracers.context import collect_runs
with collect_runs() as cb:
agent.invoke({"input": "Search for weather"})
trace = from_langchain_run(cb.traced_runs[0])Convert OpenAI ChatCompletion message lists into structured AgentTrace trajectories:
from openeval.adapters.openai import from_openai_messages
messages = [
{"role": "user", "content": "Fetch weather in Tokyo"},
{"role": "assistant", "tool_calls": [{"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"location\": \"Tokyo\"}"}}]},
{"role": "tool", "tool_call_id": "call_1", "content": "85F sunny"}
]
trace = from_openai_messages(messages)OpenEval ships out-of-the-box with seven pure, deterministic evaluation metrics.
Evaluates the ratio of expected tool calls executed by the agent.
Denial Guard: Tool calls with
step.denied == True(permission denials or validation errors) are excluded from$T_{\text{executed}}$ , ensuring forbidden calls don't count as successful tool selections.
Evaluates the key-value argument match precision for executed tool calls.
Evaluates step count efficiency relative to the optimal step count specified in the test case.
Evaluates the accuracy of final environment state key transitions against expected target state.
Evaluates whether an agent that encountered a permission denial or validation error:
- Did not retry the identical forbidden tool call (
tool_name+tool_args) anywhere in the remainder of the trajectory. - Successfully recovered by executing an allowed tool or producing output.
$$\text{Score} = \frac{\text{Count}(\text{recovered_denials})}{\text{Total Denials Encountered}}$$
Zero Denials: Returns
score = 1.0, passed = True, details = "No permission denials...".
Evaluates trajectory agreement during Verified Replay offline comparison.
No Divergence: Returns
score = 1.0, passed = True, details = "No Verified Replay divergence detected.".
Evaluates whether the trace fixture's tool registry fingerprint verdict indicates a fresh spec:
-
verdict == "PASS"$\rightarrow$ score = 1.0, passed = True -
verdict == "DRIFTED"$\rightarrow$ score = 0.0, passed = False(genuine drift regression) -
verdict == "UNVERIFIED"$\rightarrow$ score = 0.0, passed = False(check-drift failed / config error) -
No Fingerprint Metadata
$\rightarrow$ score = 1.0, passed = None(NOT EVALUATED)
Important
Disambiguated Verdicts: FixtureFreshness returns passed = None when a trace was never drift-checked. In aggregate suite reporting, metrics with passed is None are excluded from pass-rate denominators, eliminating false-positive and false-negative reporting bugs in CI gates!
OpenEval provides a high-speed CLI binary (openeval).
# Run a single evaluation test case against a trace
openeval run --trace examples/simple_agent/trace.json --testcase examples/simple_agent/testcase.json
# Run an entire evaluation suite directory
openeval run --suite tests/ --output results/
# Generate a pure Markdown evaluation report
openeval report --input results/ --format markdown
# Generate a structured JSON summary report
openeval report --input results/ --format jsonWarning
The openeval report command exits with exit code 1 if any JSON result file in the input directory is malformed or corrupted, guaranteeing pipeline failures on corrupted evals.
Gate pull requests deterministically in CI:
name: Agent Trajectory Evaluation CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
eval-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Grade Agent Execution
uses: organization/openeval-core@v1
with:
suite: path/to/eval_suite_dir
fail-under: '0.8' # Fail CI build if average pass rate < 80%OpenEval relies on pure, strongly typed dataclasses in openeval/models.py:
@dataclass
class TraceStep:
step_id: int
type: Literal["thought", "tool_call", "tool_result", "output"]
content: str
tool_name: str | None
tool_args: dict | None
tool_result: str | None
timestamp: float
error: str | None = None
# Extended telemetry fields
denied: bool = False
finish_reason: str | None = None
provider: str | None = None
model: str | None = None
tokens: dict[str, int] | None = None
latency_ms: float | None = None
divergent: bool = False@dataclass
class MetricResult:
metric_name: str
score: float
passed: bool | None # True (Passed), False (Failed), None (Not Applicable / Un-Evaluated)
details: str| Document | Description / Purpose |
|---|---|
| 🏗️ ARCHITECTURE.md | Technical architecture, component diagrams & data models |
| 📜 CHANGELOG.md | Full version history and release notes |
| 🤝 CONTRIBUTING.md | Guidelines for contributing custom deterministic metrics |
📄 docs/proposals/001_extended_trace_model.md |
Architectural proposal for additive trace fields |
📄 docs/proposals/002_fixtura_adapter_package.md |
Architectural proposal for native Fixtura trace adapter |
📄 docs/proposals/003_metric_result_not_evaluated_verdict.md |
Architectural proposal for `passed: bool |
Q: Does OpenEval require any API keys (OpenAI, Anthropic, etc.) to run?
No. OpenEval contains zero LLM calls, zero API clients, and zero network calls. Every metric is a pure mathematical Python function that scores traces offline in <1ms with $0.00 API cost.
Q: How does OpenEval handle permission denials from Fixtura or custom agents?
Unlike general eval frameworks that wipe tool names or treat denials as raw string errors, OpenEval preserves
denied=True on the TraceStep alongside raw tool_name and tool_args. This allows DenialRecoveryRate to evaluate whether the agent adapted to the refusal without repeating forbidden calls.
Q: What happens if a trace was never checked for drift?
FixtureFreshness returns score = 1.0, passed = None, details = "NOT EVALUATED: Trace was not checked for drift...". OpenEval's report generator excludes passed: None metrics from pass-rate denominators, preventing un-checked traces from causing false failures or false passes.
This project is licensed under the MIT License.