Trace-native evaluation
-Built on OpenTelemetry traces so you can evaluate real production-like runs without replaying agent execution.
+No reruns
+Use recorded traces to evaluate real executions after the fact.
- Why AgentEvals
-
-
Evaluation that matches how agents actually run.
-Traditional evals re-run entire workflows. AgentEvals scores the traces you already collect, so you can measure behavior in realistic conditions.
+Why agentevals
+Evaluate behavior from the telemetry you already collect
++ Score agents against consistent rubrics using OpenTelemetry traces rather than replaying runs. + Keep evaluations close to your production workflows and compare changes over time. +
Flexible scoring
-Combine built-in evaluators with custom Python logic to measure correctness, tool usage, memory behavior, and more.
+Behavior-first scoring
+Measure task completion, tool use quality, handoffs, latency, and more.
Works in your workflow
-Run locally with the CLI, automate in CI/CD, or explore results visually in the web UI.
+Built on OpenTelemetry
+Plug into existing observability pipelines instead of inventing a parallel eval stack.
- How it works
-
-
- From traces to scores in three steps.
+How it works
+Two ways to evaluate
-
+
-
- 01
-
- Collect traces
-Instrument your agent with OpenTelemetry and emit traces for prompts, tool calls, memory operations, and outputs.
-
- 02
-
- Define evaluators
-Choose built-in evaluators or create your own to score the behaviors that matter for your agent.
-
- 03
-
+ Run evaluations
-Score trace datasets through the CLI or web UI and compare results across prompts, models, or tool strategies.
-
+
+ 1
+
+
+ 2
+
-
+ CLI workflow
++ Run evaluations locally or in CI with config files and reproducible commands. +
+ Open the CLI guide → +Web workflow
++ Explore traces, inspect scores, and review rubric results in the browser. +
+ Open the Web guide → +
- Docs
-
-
Start with the path that fits your workflow.
+Docs
+Start where you are
- {{ range where .Site.RegularPages "Section" "docs" }}
-
-
-
-
-
-
- →
-
- {{ end }}
+
+ {{ .Title }}
-{{ .Description }}
-Quick start
+Install agentevals, run your first scoring pass, and inspect the output.
+ + +Integrations
+Connect agentevals with your existing tracing and observability stack.
+ + +Custom evaluators
+Define your own scoring logic and tailor rubrics to your agents.
+ + +Web walkthrough
+See how to inspect traces and scores with the browser-based interface.
+
- Usage
-
-
- Two ways to evaluate.
-Use the CLI for fast, scriptable scoring or the Web UI for visual exploration of evaluation results.
-
-
-
-
-
-
- CLI
-Run evaluations locally or in CI with straightforward commands and structured outputs.
-agentevals eval run config.yamlWeb UI
-Inspect trace datasets, compare runs, and review evaluator outputs in a visual interface.
-agentevals ui
- Get started
-
- Bring evaluation into your agent development loop.
-Install AgentEvals, connect your traces, and start measuring how your agent behaves in the real world.
-
- Read the docs
- View on GitHub
-
-