diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..9601b67 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,208 @@ +# OpenEval Architecture & System Specification + +**High-Performance, Purely Deterministic AI Agent Trajectory Evaluation System.** + +--- + +## 1. Executive Design Philosophy + +OpenEval evaluates **what an AI agent did**, not merely what it said. + +Modern AI agent evaluation is dominated by "LLM-as-a-Judge" frameworks (e.g. DeepEval, RAGAS, Braintrust). While useful for qualitative copy generation, LLM judges introduce three fundamental flaws to CI/CD pipelines: +1. **Flakiness & Non-Determinism:** The same agent trace scored twice by an LLM judge can yield different scores or verdicts. +2. **High Latency & API Cost:** Scoring 100 test cases with an LLM judge requires hundreds of secondary LLM calls, incurring latency and recurring API costs. +3. **Inability to Verify System State:** LLM judges evaluate text outputs; they cannot inspect whether a database operation succeeded, whether a permission policy was respected, or whether a tool call argument matched exact schema boundaries. + +OpenEval replaces judge LLMs with **100% deterministic pure functions**. Given an `AgentTrace` and an `EvalTestCase`, OpenEval computes scores reproducibly in `<1ms` with `$0.00` API cost. + +--- + +## 2. System Architecture & Components + +``` ++-----------------------------------------------------------------------------------+ +| INPUT TRAJECTORIES | +| +-------------------+ +--------------------+ +------------------------+ | +| | Fixtura .trace | | LangChain Run | | OpenAI Messages List | | +| | (Compressed) | | (Run Tree) | | (ChatCompletion API) | | +| +---------+---------+ +---------+----------+ +-----------+------------+ | ++-------------|------------------------|---------------------------|----------------+ + | | | + v v v ++-----------------------------------------------------------------------------------+ +| ADAPTER SUBSYSTEM | +| +-------------------+ +--------------------+ +------------------------+ | +| | from_fixtura_trace| | from_langchain_run | | from_openai_messages | | +| +---------+---------+ +---------+----------+ +-----------+------------+ | ++-------------|------------------------|---------------------------|----------------+ + +------------------------+---------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| CORE DATA MODEL (Pure Dataclasses) | +| | +| AgentTrace | +| ├── task_id: str | +| ├── input: str | +| ├── final_output: str | +| ├── actual_state: dict | +| ├── metadata: dict {"fingerprint", "verdict"} | +| └── steps: list[TraceStep] | +| ├── step_id: int | +| ├── type: "thought" | "tool_call" | "tool_result" | "output" | +| ├── tool_name: str | None | +| ├── tool_args: dict | None | +| ├── tool_result: str | None | +| ├── denied: bool (permission denial or validation error) | +| ├── finish_reason: str | None | +| ├── provider, model, tokens, latency_ms | +| └── divergent: bool (Verified Replay divergence marker) | ++--------------------------------------|--------------------------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| DETERMINISTIC EVAL ENGINE | +| | +| EvalTestCase | +| ├── expected_tool_calls: list[dict] | +| ├── expected_final_state: dict | +| └── max_steps / timeout_seconds | +| | +| +---------------------------------------------------------------------------+ | +| | METRIC PIPELINE | | +| | 1. ToolSelectionAccuracy (ratio of correct tool calls) | | +| | 2. ArgumentCorrectness (key-value exact argument match) | | +| | 3. StepEfficiency (optimal vs actual steps ratio) | | +| | 4. GoalCompletionRate (state transition verification) | | +| | 5. DenialRecoveryRate (permission denial recovery without loops) | | +| | 6. DivergenceScore (Verified Replay trajectory fidelity) | | +| | 7. FixtureFreshness (environment fingerprint drift check) | | +| +-------------------------------------+-------------------------------------+ | ++-----------------------------------------|-----------------------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| OUTPUT & REPORTING | +| +--------------------+ +--------------------+ +---------------------+ | +| | MetricResult | --> | Suite Runner | --> | CLI & GitHub Action | | +| | (score, passed, | | (openeval.runner) | | (openeval.report) | | +| | details) | +--------------------+ +---------------------+ | +| +--------------------+ | ++-----------------------------------------------------------------------------------+ +``` + +--- + +## 3. Data Models Specification (`openeval/models.py`) + +### 3.1 `TraceStep` +Represents an individual step in an agent's execution sequence. + +```python +@dataclass +class TraceStep: + step_id: int + type: Literal["thought", "tool_call", "tool_result", "output"] + content: str + tool_name: str | None + tool_args: dict | None + tool_result: str | None + timestamp: float + error: str | None = None + + # Extended telemetry fields + denied: bool = False + finish_reason: str | None = None + provider: str | None = None + model: str | None = None + tokens: dict[str, int] | None = None + latency_ms: float | None = None + divergent: bool = False +``` + +### 3.2 `AgentTrace` +The complete recorded trajectory of an agent run. + +```python +@dataclass +class AgentTrace: + task_id: str + input: str + steps: list[TraceStep] + final_output: str + actual_state: dict + metadata: dict +``` + +### 3.3 `EvalTestCase` +Defines the expected ground-truth requirements for a task. + +```python +@dataclass +class EvalTestCase: + task_id: str + input: str + expected_tool_calls: list[dict] + expected_final_state: dict + expected_output_contains: list[str] + max_steps: int + timeout_seconds: float +``` + +### 3.4 `MetricResult` +The outcome of evaluating a metric against a trace and test case. + +```python +@dataclass +class MetricResult: + metric_name: str + score: float + passed: bool | None # True (Passed), False (Failed), None (Not Applicable) + details: str +``` + +--- + +## 4. Complete Metric Specification + +### 4.1 `ToolSelectionAccuracy` +- **Formula:** $\text{Score} = \frac{\text{Count}(\text{actual\_tools} \cap \text{expected\_tools})}{\text{Count}(\text{expected\_tools})}$ +- **Denial Guard:** Excludes steps where `step.denied is True`. Denied tool calls are preserved on the step for denial metrics, but do not count as successfully executed tool selections. + +### 4.2 `ArgumentCorrectness` +- **Formula:** $\text{Score} = \frac{\text{Count}(\text{matching\_key\_value\_pairs})}{\text{Total Evaluated Expected Arguments}}$ +- **Denial Guard:** Evaluates arguments of executed (non-denied) tool calls. + +### 4.3 `StepEfficiency` +- **Formula:** $\text{Score} = \min\left(1.0, \frac{\text{optimal\_steps}}{\text{actual\_steps\_taken}}\right)$ +- **Behavior:** Penalizes unnecessary extra tool steps. + +### 4.4 `GoalCompletionRate` +- **Formula:** $\text{Score} = \frac{\text{Count}(\text{actual\_state}[k] == \text{expected\_state}[k])}{\text{Total Expected Keys}}$ +- **Behavior:** Verifies environment state transitions. + +### 4.5 `DenialRecoveryRate` +- **Formula:** $\text{Score} = \frac{\text{Count}(\text{recovered\_denials})}{\text{Total Denials Encountered}}$ +- **Behavior:** Evaluates if the agent experienced a permission denial or validation error and: + 1. Did **not** retry the exact same forbidden call (`tool_name` + `tool_args`) anywhere in the remainder of the trace. + 2. Subsequently executed an allowed tool call or produced final output. +- **Zero Denials:** Returns `score = 1.0, passed = True, details = "No permission denials..."`. + +### 4.6 `DivergenceScore` +- **Formula:** $\text{Score} = \frac{\text{Step Index of First Divergence}}{\text{Total Steps}}$ +- **Behavior:** Evaluates trajectory agreement during Verified Replay. Returns `1.0` if zero divergence occurred. + +### 4.7 `FixtureFreshness` +- **Behavior:** Evaluates `trace.metadata.get("verdict")`: + - `"PASS"` $\rightarrow$ `score = 1.0, passed = True` + - `"DRIFTED"` $\rightarrow$ `score = 0.0, passed = False` (genuine drift regression) + - `"UNVERIFIED"` $\rightarrow$ `score = 0.0, passed = False` (test setup / config error) + - **No Fingerprint Metadata** $\rightarrow$ `score = 1.0, passed = None` (`NOT EVALUATED`) + +--- + +## 5. Execution & Reporting Pipeline + +1. **`openeval.runner`:** Loads trace JSON or converts `.trace` files, pairs them with `EvalTestCase` definitions, executes selected metrics, and produces structured result dictionaries. +2. **`openeval.report`:** Formats raw result dictionaries into deterministic Markdown or JSON reports. +3. **Verdict Filtering:** Metrics returning `passed: None` (`NOT EVALUATED`) are excluded from failure lists and suite pass-rate denominators, eliminating false positives and false failures in CI gates. diff --git a/CHANGELOG.md b/CHANGELOG.md index 791dcbb..df1a923 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.2.0] - 2026-07-26 + +### Added +- **Extended Data Model (`TraceStep` & `AgentTrace`):** Additive support for `denied`, `finish_reason`, `provider`, `model`, `tokens`, `latency_ms`, and replay `divergent` fields. +- **Three New Deterministic Metrics:** Added `DenialRecoveryRate` (permission denial recovery), `DivergenceScore` (Verified Replay trajectory agreement), and `FixtureFreshness` (tool registry fingerprint drift detection). +- **Native Fixtura Trace Adapter:** Added `openeval.adapters.fixtura.from_fixtura_trace` for native zstd `.trace` ingestion. +- **Disambiguated Verdicts (`passed: bool | None`):** `MetricResult.passed` extended to support `None` (`NOT EVALUATED` / `N/A`), preventing un-drift-checked traces from distorting aggregate suite pass rates. +- **Comprehensive Technical Docs:** Added `ARCHITECTURE.md` and detailed architectural proposals in `docs/proposals/`. + ## [0.1.2] - 2026-07-03 ### Added diff --git a/README.md b/README.md index 5d934c2..6a5ca37 100644 --- a/README.md +++ b/README.md @@ -6,55 +6,165 @@ [![CI](https://github.com/yash161004/OpenEval/actions/workflows/ci.yml/badge.svg)](https://github.com/yash161004/OpenEval/actions/workflows/ci.yml) [![codecov](https://codecov.io/gh/yash161004/OpenEval/branch/main/graph/badge.svg)](https://codecov.io/gh/yash161004/OpenEval) -Every existing tool evaluates what the agent said. OpenEval evaluates what the agent did. +**Pure deterministic agent-trajectory evaluation library — zero judge-LLM cost, zero LLM flakiness.** -## Installation +> Every existing tool evaluates what the agent *said*. OpenEval evaluates what the agent *did*. -```bash -pip install openeval-core -``` +--- -*Note: While the package is installed as `openeval-core`, the CLI command to run it is `openeval`.* +## Table of Contents -## Usage +- [What this is](#what-this-is) +- [Why OpenEval Exists](#why-openeval-exists) +- [Architecture & System Flow](#architecture--system-flow) +- [A-to-Z Feature Directory](#a-to-z-feature-directory) +- [Installation Matrix](#installation-matrix) +- [Quickstart Tutorials](#quickstart-tutorials) + - [Tier 1: Core Engine (Zero Dependencies)](#tier-1-core-engine-zero-dependencies) + - [Tier 2: Native Fixtura Trace Integration](#tier-2-native-fixtura-trace-integration) + - [Tier 3: LangChain Run Integration](#tier-3-langchain-run-integration) + - [Tier 4: OpenAI Tool Calling Messages](#tier-4-openai-tool-calling-messages) +- [Built-in Deterministic Metrics Reference](#built-in-deterministic-metrics-reference) +- [Command Line Interface (CLI) Guide](#command-line-interface-cli-guide) +- [GitHub Actions CI/CD Gating](#github-actions-cicd-gating) +- [Core Data Models Reference](#core-data-models-reference) +- [Repository Structure & Proposals](#repository-structure--proposals) +- [License](#license) -Run a single test case using the included example: -```bash -openeval run --trace examples/simple_agent/trace.json --testcase examples/simple_agent/testcase.json -``` +--- -Run a test suite: -```bash -openeval run --suite tests/ --output results/ -``` +## What this is -Generate a report: -```bash -openeval report --input results/ --format markdown -``` +OpenEval is a lightweight, framework-agnostic trajectory evaluation engine for AI agents. -*Note: The `report` command will exit with a non-zero status code (1) if any JSON file in the input directory is malformed or fails to load, guaranteeing pipeline failures on corrupted evals.* +Unlike general-purpose LLM evaluation frameworks that rely on expensive, flaky, non-deterministic "LLM-as-a-Judge" prompts, OpenEval evaluates agent trajectories using **100% deterministic pure functions**. It scores exact tool selections, argument correctness, trajectory step efficiency, goal state transitions, permission denial recoveries, Verified Replay divergences, and environment fingerprint staleness. -## Quickstart +It features **native integration with Fixtura** (`fixtura-core`), allowing recorded and replayed agent traces to be scored instantly in local testing or CI pipelines. -Get your first deterministic evaluation running in under 2 minutes. OpenEval is completely framework-agnostic. +--- -### Tier 1: The Core Engine (No Framework Required) +## Why OpenEval Exists -The core engine requires zero dependencies on external frameworks or LLM SDKs. You just feed it plain Python data. +1. **Zero LLM-Judge Flakiness:** Evals built on top of LLM-as-judge prompts drift across model versions, introduce non-reproducible scoring variance, and cost money on every test run. OpenEval guarantees `Score = 1.0` or `0.0` is reproducible forever. +2. **Depth Over Breadth:** We do not chase 50+ fuzzy NLP prompt metrics or heavy web UIs. OpenEval focuses on depth: strict execution correctness and native support for signals produced by Fixtura's permission and replay pipeline. +3. **Disambiguated Verdicts (`passed: bool | None`):** Un-evaluated metrics (e.g. `FixtureFreshness` on a trace without drift metadata) return `passed = None` (`NOT EVALUATED`), excluding them from suite pass-rate denominators and eliminating false-positive and false-negative reporting bugs in CI. -**1. Install the package** -```bash -pip install openeval-core +--- + +## Architecture & System Flow + +``` ++-----------------------------------------------------------------------------------+ +| INPUT TRAJECTORIES | +| +-------------------+ +--------------------+ +------------------------+ | +| | Fixtura .trace | | LangChain Run | | OpenAI Messages List | | +| | (Compressed) | | (Run Tree) | | (ChatCompletion API) | | +| +---------+---------+ +---------+----------+ +-----------+------------+ | ++-------------|------------------------|---------------------------|----------------+ + | | | + v v v ++-----------------------------------------------------------------------------------+ +| ADAPTER SUBSYSTEM | +| +-------------------+ +--------------------+ +------------------------+ | +| | from_fixtura_trace| | from_langchain_run | | from_openai_messages | | +| +---------+---------+ +---------+----------+ +-----------+------------+ | ++-------------|------------------------|---------------------------|----------------+ + +------------------------+---------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| CORE DATA MODEL (Pure Dataclasses) | +| | +| AgentTrace | +| ├── task_id: str | +| ├── input: str | +| ├── final_output: str | +| ├── actual_state: dict | +| ├── metadata: dict {"fingerprint", "verdict"} | +| └── steps: list[TraceStep] | +| ├── step_id: int | +| ├── type: "thought" | "tool_call" | "tool_result" | "output" | +| ├── tool_name: str | None | +| ├── tool_args: dict | None | +| ├── tool_result: str | None | +| ├── denied: bool (permission denial or validation error) | +| ├── finish_reason: str | None | +| ├── provider, model, tokens, latency_ms | +| └── divergent: bool (Verified Replay divergence marker) | ++--------------------------------------|--------------------------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| DETERMINISTIC EVAL ENGINE | +| | +| EvalTestCase | +| ├── expected_tool_calls: list[dict] | +| ├── expected_final_state: dict | +| └── max_steps / timeout_seconds | +| | +| +---------------------------------------------------------------------------+ | +| | METRIC PIPELINE | | +| | 1. ToolSelectionAccuracy (ratio of correct tool calls) | | +| | 2. ArgumentCorrectness (key-value exact argument match) | | +| | 3. StepEfficiency (optimal vs actual steps ratio) | | +| | 4. GoalCompletionRate (state transition verification) | | +| | 5. DenialRecoveryRate (permission denial recovery without loops) | | +| | 6. DivergenceScore (Verified Replay trajectory fidelity) | | +| | 7. FixtureFreshness (environment fingerprint drift check) | | +| +-------------------------------------+-------------------------------------+ | ++-----------------------------------------|-----------------------------------------+ + | + v ++-----------------------------------------------------------------------------------+ +| OUTPUT & REPORTING | +| +--------------------+ +--------------------+ +---------------------+ | +| | MetricResult | --> | Suite Runner | --> | CLI & GitHub Action | | +| | (score, passed, | | (openeval.runner) | | (openeval.report) | | +| | details) | +--------------------+ +---------------------+ | +| +--------------------+ | ++-----------------------------------------------------------------------------------+ ``` -**2. Score a trace deterministically** -Create a file named `hello_eval.py` and run it: +--- + +## A-to-Z Feature Directory + +- **Zero LLM-Judge Overhead:** Every metric is a pure function executed in Python with zero network calls or API costs. +- **Permission Denial Telemetry:** Preserves denied tool call attempts with raw `tool_name` and `tool_args`, allowing structural recovery scoring instead of string dumps. +- **Denial Recovery Rate Scoring:** Detects if an agent recovered from permission denials or validation errors without repeating identical forbidden calls anywhere in the trajectory. +- **Verified Replay Divergence Tracking:** Evaluates trajectory step fidelity against recorded baseline traces from Fixtura's offline replay engine. +- **Environment Fingerprint Staleness Check:** Reads tool registry fingerprints and evaluates `PASS`, `DRIFTED`, and `UNVERIFIED` verdicts. +- **Disambiguated `passed: bool | None` Verdicts:** Excludes un-evaluated metrics (`passed = None`) from aggregate pass-rate calculations, eliminating false-positive CI gates. +- **Framework-Agnostic Adapters:** Built-in converters for Fixtura (`.trace`), LangChain (`Run`), and OpenAI (`messages`). +- **Deterministic CLI & CI Reporter:** Pure Markdown and JSON report generator with strict exit code guarantees for CI gating. + +--- + +## Installation Matrix + +| Environment / Feature | Installation Command | +| :--- | :--- | +| **Core Engine** (Zero external dependencies) | `pip install openeval-core` | +| **With Fixtura `.trace` Support** | `pip install "openeval-core[fixtura]"` | +| **With LangChain Support** | `pip install "openeval-core[langchain]"` | +| **All Extras** | `pip install "openeval-core[fixtura,langchain]"` | +| **Local Editable Install** | `pip install -e ".[fixtura]"` | + +*Note: Package name is `openeval-core`, while the CLI executable is `openeval`.* + +--- + +## Quickstart Tutorials + +### Tier 1: Core Engine (Zero Dependencies) + +Score plain Python dictionaries deterministically: + ```python -from openeval.metrics import ToolSelectionAccuracy +from openeval.metrics import ToolSelectionAccuracy, ArgumentCorrectness from openeval.models import AgentTrace, EvalTestCase, TraceStep -# 1. Define what the agent was supposed to do +# 1. Define expectations test_case = EvalTestCase( task_id="quickstart-1", input="Search for the weather in Tokyo", @@ -65,7 +175,7 @@ test_case = EvalTestCase( timeout_seconds=10.0 ) -# 2. Provide the raw trace of what the agent actually did +# 2. Ingest agent trace trace = AgentTrace( task_id="quickstart-1", input="Search for the weather in Tokyo", @@ -85,69 +195,107 @@ trace = AgentTrace( metadata={} ) -# 3. Score it deterministically (no LLM required) +# 3. Score deterministically (0.0ms, 0 API tokens) metric = ToolSelectionAccuracy() result = metric.score(trace, test_case) -print(f"Metric: {result.metric_name}") -print(f"Score: {result.score} (Passed: {result.passed})") +print(f"Metric: {result.metric_name}") +print(f"Score: {result.score} (Passed: {result.passed})") print(f"Details: {result.details}") ``` -**Expected Output:** +**Output:** ``` -Metric: Tool Selection Accuracy -Score: 1.0 (Passed: True) +Metric: Tool Selection Accuracy +Score: 1.0 (Passed: True) Details: Correctly called 1 of 1 expected tools. ``` -### Tier 2: Using OpenEval with LangChain +--- -If your traces come from LangChain, you can use our built-in adapter to automatically convert LangChain runs into OpenEval traces. +### Tier 2: Native Fixtura Trace Integration -**1. Install with LangChain support** -```bash -pip install "openeval-core[langchain]" +Ingest zstd-compressed JSONL `.trace` files generated by Fixtura: + +```python +from openeval.adapters.fixtura import from_fixtura_trace +from openeval.metrics import DenialRecoveryRate, DivergenceScore, FixtureFreshness + +trace = from_fixtura_trace( + trace_path="fixtures/checkout.trace", + task_id="task-1", + input_text="Execute checkout", + final_output="Order placed", + actual_state={"order_created": True}, + metadata={} +) + +metrics = [DenialRecoveryRate(), DivergenceScore(), FixtureFreshness()] +for m in metrics: + res = m.score(trace, test_case) + print(f"{m.name:25s}: Score = {res.score:.2f} | Passed = {res.passed!s:5s} | {res.details}") ``` -**2. Convert and Score** -*(See the `examples/` directory in our GitHub repository for full, runnable agent scripts using this adapter.)* +--- + +### Tier 3: LangChain Run Integration + +Convert LangChain `Run` trees into OpenEval traces: ```python from openeval.adapters.langchain import from_langchain_run from langchain_core.tracers.context import collect_runs with collect_runs() as cb: - # Run your langchain agent - agent.invoke({"input": "task input"}) - -# Convert the run to an OpenEval AgentTrace -trace = from_langchain_run(cb.traced_runs[0]) + agent.invoke({"input": "Search for weather"}) -# Score it exactly like Tier 1 -# metric.score(trace, test_case) +trace = from_langchain_run(cb.traced_runs[0]) ``` -### OpenAI Tool Calling +--- -OpenEval can automatically convert raw OpenAI chat completions message lists into `AgentTrace` objects. +### Tier 4: OpenAI Tool Calling Messages -*LIMITATION: This adapter flattens the message list. Only the first `user` message is used as the input, and the final `assistant` message without tool calls is used as the output. Any intermediate non-tool conversational turns are not captured structurally.* +Convert raw OpenAI message arrays into structured `AgentTrace` instances: ```python from openeval.adapters.openai import from_openai_messages -messages = [ - # ... your OpenAI messages list -] +trace = from_openai_messages(messages_list) +``` + +--- + +## Built-in Deterministic Metrics Reference + +| Metric | Class Name | What it evaluates | Score | `passed` | +| :--- | :--- | :--- | :--- | :--- | +| **Tool Selection Accuracy** | `ToolSelectionAccuracy` | Ratio of expected tools called vs total expected. | `0.0 - 1.0` | `True` / `False` | +| **Argument Correctness** | `ArgumentCorrectness` | Precision of key-value argument matching on executed tools. | `0.0 - 1.0` | `True` / `False` | +| **Step Efficiency** | `StepEfficiency` | Step efficiency ratio (`optimal_steps / actual_steps`). | `0.0 - 1.0` | `True` / `False` | +| **Goal Completion Rate** | `GoalCompletionRate` | Match accuracy of final actual state vs expected state. | `0.0 - 1.0` | `True` / `False` | +| **Denial Recovery Rate** | `DenialRecoveryRate` | Ratio of permission/validation denials recovered without repeating forbidden calls. | `0.0 - 1.0` | `True` / `False` | +| **Divergence Score** | `DivergenceScore` | Trajectory agreement score during Verified Replay offline comparison. | `0.0 - 1.0` | `True` / `False` | +| **Fixture Freshness** | `FixtureFreshness` | Evaluates tool registry fingerprint verdict (`PASS`, `DRIFTED`, `UNVERIFIED`). | `1.0` / `0.0` | `True` / `False` / `None` (`NOT EVALUATED`) | -# Convert the messages -trace = from_openai_messages(messages) +--- + +## Command Line Interface (CLI) Guide + +```bash +# Run single test case +openeval run --trace trace.json --testcase testcase.json + +# Run test suite directory +openeval run --suite tests/ --output results/ + +# Generate Markdown or JSON report +openeval report --input results/ --format markdown ``` -### GitHub Action +--- -OpenEval provides a GitHub Action to seamlessly gate PRs based on your agents' performance. +## GitHub Actions CI/CD Gating ```yaml steps: @@ -155,6 +303,31 @@ steps: - name: Grade Agent Execution uses: organization/openeval-core@v1 with: - suite: path/to/your/test_suite_dir - fail-under: '0.8' # Fail the step if average metric score is < 80% + suite: path/to/eval_suite_dir + fail-under: '0.8' # Fail CI step if suite pass rate < 80% ``` + +--- + +## Core Data Models Reference + +See [ARCHITECTURE.md](file:///d:/OpenEval/ARCHITECTURE.md) for full dataclass definitions (`TraceStep`, `AgentTrace`, `EvalTestCase`, `MetricResult`). + +--- + +## Repository Structure & Proposals + +| Document | Purpose | +| :--- | :--- | +| [ARCHITECTURE.md](file:///d:/OpenEval/ARCHITECTURE.md) | Technical architecture & metric algorithms | +| [CONTRIBUTING.md](file:///d:/OpenEval/CONTRIBUTING.md) | Contributing guidelines | +| [CHANGELOG.md](file:///d:/OpenEval/CHANGELOG.md) | Version history | +| `docs/proposals/001_extended_trace_model.md` | Additive fields on TraceStep/AgentTrace | +| `docs/proposals/002_fixtura_adapter_package.md` | Native `from_fixtura_trace` adapter design | +| `docs/proposals/003_metric_result_not_evaluated_verdict.md` | `passed: bool | None` verdict specification | + +--- + +## License + +[MIT License](file:///d:/OpenEval/LICENSE) diff --git a/action.yml b/action.yml index 0999db4..649da93 100644 --- a/action.yml +++ b/action.yml @@ -17,53 +17,24 @@ runs: python-version: '3.11' - name: Install OpenEval shell: bash - run: pip install openeval-core + run: pip install "${{ github.action_path }}" - name: Run OpenEval shell: bash run: | mkdir -p .openeval_results openeval run --suite "${{ inputs.suite }}" --output .openeval_results || echo "OpenEval CLI returned non-zero exit code, continuing to threshold check..." - cat << 'EOF' > evaluate_threshold.py - import json - import sys - from pathlib import Path + echo "Generating report and evaluating threshold..." + # Capture exit code and output to a file + set +e + openeval report --input .openeval_results --fail-under "${{ inputs.fail-under }}" > openeval_report.md + EXIT_CODE=$? + set -e - fail_under = float('${{ inputs.fail-under }}') - results_dir = Path('.openeval_results') + # Write to step summary + cat openeval_report.md >> $GITHUB_STEP_SUMMARY - if not results_dir.exists() or not any(results_dir.iterdir()): - print("Error: No results generated. Check if the suite directory is correct.") - sys.exit(1) - - total_score = 0.0 - total_metrics = 0 - - for file_path in results_dir.glob("*.json"): - with open(file_path, "r", encoding="utf-8") as f: - data = json.load(f) - if "error" in data: - print(f"Error: Test case {data.get('task_id', 'unknown')} failed with error: {data['error']}") - print("A hard error occurred during evaluation. Failing the action outright.") - sys.exit(1) - - for metric in data.get("metrics", {}).values(): - total_score += metric.get("score", 0.0) - total_metrics += 1 - - if total_metrics == 0: - print("No metrics evaluated.") - sys.exit(1) - - avg_score = total_score / total_metrics - print(f"Overall Average Score: {avg_score:.2f}") - print(f"Required Threshold: {fail_under:.2f}") - - if avg_score < fail_under: - print("Eval score is below the fail-under threshold. Failing the action.") - sys.exit(1) - else: - print("Eval score meets or exceeds the threshold. Passed.") - EOF - - python evaluate_threshold.py + if [ $EXIT_CODE -ne 0 ]; then + echo "OpenEval failed (either due to errors or score below threshold). Check the step summary for details." + exit $EXIT_CODE + fi diff --git a/docs/proposals/001_extended_trace_model.md b/docs/proposals/001_extended_trace_model.md new file mode 100644 index 0000000..9e47a7e --- /dev/null +++ b/docs/proposals/001_extended_trace_model.md @@ -0,0 +1,108 @@ +# Proposal 001: Additive Extension of OpenEval Trace Data Model for Rich Agent Signals + +**Status:** Proposed +**Author:** Antigravity AI & Fixtura Core Team +**Date:** 2026-07-26 + +## Executive Summary + +This proposal defines an additive extension to OpenEval's core data models (`TraceStep` and `AgentTrace` in `openeval/models.py`). The extension enables OpenEval to ingest rich trace telemetry generated by Fixtura—specifically permission denials, LLM completion metadata (`finish_reason`, token counts, latency), Verified Replay divergences, and fingerprint drift verdicts—without breaking any existing adapters, metrics, or API contracts. + +--- + +## 1. Problem Statement + +### 1.1 Deceptive Permission Denial Coercion +Previously, when Fixtura converted trace events into OpenEval's `AgentTrace`, permission denials (`permission_decision == "denied"`) were coerced by setting `tool_name = None` and `tool_args = None`. This was intended to dodge `ToolSelectionAccuracy`, but it resulted in a scoring bug: +- An agent that attempted forbidden actions scored identically to an agent that never attempted them. +- Crucial structural data (`tool_name` and `tool_args`) was dumped into a unstructured `content` string, making it impossible for deterministic metrics to analyze attempted vs. recovered tool calls. +- `validation_error` decisions fell through conditionally and were silently dropped. + +### 1.2 Discarded LLM Execution Metadata +Fixtura `llm_call` events contain `finish_reason` (which records provider refusals or tool-call triggers), `provider`, `model`, token counts (`input_tokens`, `output_tokens`), and `latency_ms`. These were concatenated into a raw `content` string, discarding structured metadata. + +### 1.3 Missing Replay & Drift Signals +Fixtura's Verified Replay (divergence points between live execution and recordings) and drift detection (`PASS` / `DRIFTED` / `UNVERIFIED` verdicts against tool fingerprints) had no representation in OpenEval's `AgentTrace` or `TraceStep` models. + +--- + +## 2. Proposed Data Model Changes + +All changes to `openeval/models.py` are strictly **additive** with backwards-compatible default values. + +### 2.1 `TraceStep` Extension + +```python +@dataclass +class TraceStep: + step_id: int + type: Literal["thought", "tool_call", "tool_result", "output"] + content: str + tool_name: str | None + tool_args: dict | None + tool_result: str | None + timestamp: float + error: str | None = None + + # Additive fields for rich trace telemetry + denied: bool = False + finish_reason: str | None = None + provider: str | None = None + model: str | None = None + tokens: dict[str, int] | None = None # e.g., {"input": 150, "output": 45} + latency_ms: float | None = None + divergent: bool = False +``` + +#### Field Specifications: +- `denied: bool = False`: Set to `True` when a tool call attempt was blocked by permission policy or failed validation. +- `finish_reason: str | None = None`: The completion finish reason reported by the LLM provider (e.g. `"stop"`, `"tool_calls"`, `"length"`, `"content_filter"`). +- `provider: str | None = None`: The LLM provider identifier (e.g. `"openai"`, `"anthropic"`). +- `model: str | None = None`: Model identifier string (e.g. `"gpt-4o"`). +- `tokens: dict[str, int] | None = None`: Dictionary containing token counts `{"input": int, "output": int}`. +- `latency_ms: float | None = None`: Execution duration in milliseconds. +- `divergent: bool = False`: Set to `True` if this step represents a divergence point during Verified Replay. + +### 2.2 `AgentTrace` Metadata Convention + +`AgentTrace` already contains a `metadata: dict` field. We formalize standard key conventions for Fixtura integration: +- `metadata["fingerprint"]`: The string hash of the tool registry fingerprint. +- `metadata["verdict"]`: Drift check verdict (`"PASS"`, `"DRIFTED"`, or `"UNVERIFIED"`). + +--- + +## 3. Fixtura Adapter Update Specification (`tools/openeval_adapter.py`) + +### 3.1 Tool Call & Denial Handling +For `event_type == "tool_call"`: +- **`allowed`**: `denied = False`, `tool_name`, `tool_args`, `tool_result`, `latency_ms` mapped cleanly. +- **`denied` and `validation_error`**: + - Preserve actual `tool_name` and `arguments` in `tool_args`. + - Set `denied = True`. + - Set `error = event.get("permission_reason", "Denied")`. + - Store human note in `content`. + +### 3.2 LLM Call Handling +For `event_type == "llm_call"`: +- Map `type = "thought"`. +- Set `finish_reason`, `provider`, `model`, `tokens = {"input": input_tokens, "output": output_tokens}`, and `latency_ms`. + +### 3.3 Header & Replay Lineage +- Read `trace_header` via `reader.read_header()` to populate `metadata["fingerprint"]` and `metadata["verdict"]`. +- If `divergence_step_id` is present in header, flag matching step with `divergent = True`. + +--- + +## 4. Impact on Existing Metrics & Adapters + +1. **Existing Adapters (`LangChain`, `OpenAI`):** Instantiated `TraceStep` objects will use default values (`denied=False`, `finish_reason=None`, etc.). Zero breaking changes. +2. **`ToolSelectionAccuracy` Metric:** Updated to filter out `denied` steps when measuring executed tool accuracy (`if step.type == "tool_call" and step.tool_name is not None and not step.denied`), preventing denied attempts from counting as successful tool selections while preserving `tool_name` on the step for denial recovery metrics. +3. **Future Metrics (Phase 2):** `DenialRecoveryRate`, `DivergenceScore`, and `FixtureFreshness` will consume these structured fields directly. + +--- + +## 5. Verification Plan + +- Unit tests in OpenEval testing `TraceStep` initialization with and without extended fields. +- Unit tests for `ToolSelectionAccuracy` verifying behavior on traces with allowed vs. denied steps. +- Adapter tests in Fixtura verifying loss-less trace conversion. diff --git a/docs/proposals/002_fixtura_adapter_package.md b/docs/proposals/002_fixtura_adapter_package.md new file mode 100644 index 0000000..d2d824c --- /dev/null +++ b/docs/proposals/002_fixtura_adapter_package.md @@ -0,0 +1,77 @@ +# Proposal 002: Dedicated Fixtura Adapter Package (`openeval.adapters.fixtura`) + +**Status:** Proposed +**Author:** Antigravity AI & Fixtura Core Team +**Date:** 2026-07-26 + +## Executive Summary + +This proposal defines the native Fixtura trace adapter in OpenEval (`openeval.adapters.fixtura.from_fixtura_trace`). Moving trace ingestion logic into OpenEval core establishes native support for Fixtura's deterministic replay, drift detection, and permission signals, while converting Fixtura's `tools/openeval_adapter.py` into a thin wrapper. + +--- + +## 1. Public API Interface + +```python +# openeval/adapters/fixtura.py + +from pathlib import Path +from typing import Any, Union +from openeval.models import AgentTrace + +def from_fixtura_trace( + trace_path: Union[str, Path], + task_id: str = "fixtura_task", + input_text: str = "", + final_output: str = "", + actual_state: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None +) -> AgentTrace: + """ + Converts a Fixtura .trace compressed JSONL file into an OpenEval AgentTrace. + + Parameters: + - trace_path: Path to the .trace zstd-compressed JSONL file. + - task_id: Identifier for the evaluation task. + - input_text: Top-level prompt/input text given to the agent. + - final_output: Final string answer produced by the agent. + - actual_state: Final environment state dictionary. + - metadata: Optional evaluation metadata (e.g. {"verdict": "PASS"}). + """ +``` + +--- + +## 2. Dependency & Packaging Strategy + +- **`zstandard` Dependency:** Fixtura trace files (`.trace`) are zstd-compressed JSONL. To maintain OpenEval's zero-dependency core requirement, `zstandard` is defined as an optional extra: `pip install "openeval-core[fixtura]"`. +- **Runtime Guard:** If `zstandard` is not installed when `from_fixtura_trace()` is invoked, a descriptive `RuntimeError` / `ImportError` is raised with installation instructions. + +--- + +## 3. Delegation & Backwards Compatibility + +Fixtura's existing adapter `tools/openeval_adapter.py` will be updated to a 3-line wrapper: + +```python +from openeval.adapters.fixtura import from_fixtura_trace + +def trace_to_agent_trace( + trace_path, task_id, input_text, final_output, actual_state, metadata=None +) -> AgentTrace: + return from_fixtura_trace( + trace_path=trace_path, + task_id=task_id, + input_text=input_text, + final_output=final_output, + actual_state=actual_state, + metadata=metadata + ) +``` + +--- + +## 4. Verification Plan + +- Unit test in `OpenEval` (`tests/test_fixtura_adapter.py`) testing `from_fixtura_trace` on simple, denied, and multi-step traces. +- Full E2E regression check of Fixtura's test suite (`132/132` passing). diff --git a/docs/proposals/003_metric_result_not_evaluated_verdict.md b/docs/proposals/003_metric_result_not_evaluated_verdict.md new file mode 100644 index 0000000..02b9769 --- /dev/null +++ b/docs/proposals/003_metric_result_not_evaluated_verdict.md @@ -0,0 +1,65 @@ +# Proposal 003: Disambiguating Non-Applicable Metrics with `passed: bool | None` + +**Status:** Proposed +**Author:** Antigravity AI & OpenEval Core Team +**Date:** 2026-07-26 + +## Executive Summary + +This proposal addresses metric aggregation distortion caused by non-applicable or un-evaluated trace metrics. By allowing `MetricResult.passed` to accept `None` (representing `Not Applicable` / `Un-Evaluated`), OpenEval prevents un-checked metrics (such as `FixtureFreshness` on a trace without drift metadata) from skewing aggregate suite pass rates as either false positives or false failures. + +--- + +## 1. Problem Statement + +Previously, `MetricResult.passed` was strictly typed as a non-nullable `bool`. When a metric encountered a trace where evaluation was not applicable: +- Setting `passed = False` produced **false failures** (e.g. flagging un-checked traces as drift regressions). +- Setting `passed = True` produced **false positives** (e.g. reporting 100% pass rates for suites where drift checking was never performed). + +In aggregate suite reporting (`openeval report` / CI gates), binary booleans conflate *confirmed clean execution* with *un-evaluated execution*. + +--- + +## 2. Proposed Data Model Extension (`openeval/models.py`) + +```python +@dataclass +class MetricResult: + metric_name: str + score: float + passed: bool | None # True = Passed, False = Failed, None = Not Applicable / Un-Evaluated + details: str +``` + +### Verdict Semantics: +- **`True` (Passed):** The metric evaluated the trace and confirmed it met criteria. +- **`False` (Failed):** The metric evaluated the trace and detected a failure or regression. +- **`None` (Not Applicable / Un-Evaluated):** The metric could not be evaluated due to missing input preconditions (e.g. no fingerprint metadata present for `FixtureFreshness`). + +--- + +## 3. Aggregate Reporting Rules (`openeval/runner.py` / CLI) + +When calculating suite-level pass rates: +1. Metrics returning `passed is None` are **excluded from the denominator**. +2. **Suite Pass Rate Formula:** + $$\text{Pass Rate} = \frac{\text{Count}(\text{passed is True})}{\text{Count}(\text{passed is True}) + \text{Count}(\text{passed is False})}$$ +3. Reporting outputs (`openeval report`) render `passed: None` clearly as `N/A (Not Evaluated)`. + +--- + +## 4. `FixtureFreshness` Implementation + +| Metadata Condition | `score` | `passed` | Details | +| :--- | :--- | :--- | :--- | +| `verdict == "PASS"` | `1.0` | `True` | `Fixture fingerprint matches current agent spec (PASS).` | +| `verdict == "DRIFTED"` | `0.0` | `False` | `DRIFTED: Fixture fingerprint has drifted from current agent spec.` | +| `verdict == "UNVERIFIED"` | `0.0` | `False` | `UNVERIFIED: Fixture fingerprint status is UNVERIFIED.` | +| **No fingerprint metadata** | `1.0` | `None` | `NOT EVALUATED: Trace was not checked for drift (no fingerprint metadata present).` | + +--- + +## 5. Backwards Compatibility + +- Existing metrics and custom metrics returning `bool` (`True` or `False`) continue working without modification. +- Evaluators checking `if result.passed:` evaluate `None` as falsy, while explicit checks (`if result.passed is True:`) cleanly separate passed tests. diff --git a/openeval/adapters/__init__.py b/openeval/adapters/__init__.py index f65001d..e8a1e0a 100644 --- a/openeval/adapters/__init__.py +++ b/openeval/adapters/__init__.py @@ -1,4 +1,6 @@ from .langchain import from_langchain_run from .openai import from_openai_messages +from .fixtura import from_fixtura_trace + +__all__ = ["from_langchain_run", "from_openai_messages", "from_fixtura_trace"] -__all__ = ["from_langchain_run", "from_openai_messages"] diff --git a/openeval/adapters/fixtura.py b/openeval/adapters/fixtura.py new file mode 100644 index 0000000..9f8670a --- /dev/null +++ b/openeval/adapters/fixtura.py @@ -0,0 +1,186 @@ +import io +import json +import re +from pathlib import Path +from typing import Any, Union, Set + +try: + import zstandard as zstd +except ImportError: + zstd = None + +from openeval.models import AgentTrace, TraceStep + +def extract_step_id(step_id_str: str) -> int: + digits = re.sub(r'\D', '', str(step_id_str)) + if not digits: + raise ValueError(f"Could not extract integer from step_id: {step_id_str}") + return int(digits) + +def _read_fixtura_events(trace_file: Path) -> tuple[dict | None, list[dict]]: + if zstd is None: + raise RuntimeError( + "zstandard is required to read Fixtura traces. " + "Install it via: pip install zstandard or pip install 'openeval-core[fixtura]'" + ) + + if not trace_file.exists(): + return None, [] + + dctx = zstd.ZstdDecompressor() + events = [] + header = None + + with open(trace_file, "rb") as f: + with dctx.stream_reader(f, read_across_frames=True) as reader: + text_stream = io.TextIOWrapper(reader, encoding="utf-8") + for line in text_stream: + line = line.strip() + if not line: + continue + event = json.loads(line) + if event.get("event_type") == "trace_header": + if header is None: + header = event + else: + events.append(event) + + if header and header.get("parent_trace_id") and header.get("divergence_step_id"): + parent_id = header["parent_trace_id"] + divergence_id = header["divergence_step_id"] + parent_file = trace_file.parent / f"{parent_id}.trace" + if parent_file.exists(): + _, p_events = _read_fixtura_events(parent_file) + parent_prefix = [] + for pe in p_events: + parent_prefix.append(pe) + if pe.get("step_id") == divergence_id: + break + events = parent_prefix + events + + return header, events + +def from_fixtura_trace( + trace_path: Union[str, Path], + task_id: str = "fixtura_task", + input_text: str = "", + final_output: str = "", + actual_state: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None +) -> AgentTrace: + """ + Converts a Fixtura .trace file into an OpenEval AgentTrace. + """ + if metadata is None: + metadata = {} + else: + metadata = dict(metadata) + + if actual_state is None: + actual_state = {} + + trace_file = Path(trace_path) + header, events = _read_fixtura_events(trace_file) + + divergence_step_id = None + if header: + if "fingerprint" in header and "fingerprint" not in metadata: + metadata["fingerprint"] = header["fingerprint"] + if "verdict" in header and "verdict" not in metadata: + metadata["verdict"] = header["verdict"] + divergence_step_id = header.get("divergence_step_id") + + steps: list[TraceStep] = [] + seen_step_ids: Set[int] = set() + + for event in events: + raw_step_id = event["step_id"] + step_id_int = extract_step_id(raw_step_id) + + if step_id_int in seen_step_ids: + raise ValueError(f"Step ID collision after int conversion: {raw_step_id} -> {step_id_int}") + seen_step_ids.add(step_id_int) + + event_type = event.get("event_type") + timestamp = event.get("timestamp", 0.0) + is_divergent = (raw_step_id == divergence_step_id or step_id_int == divergence_step_id) + + if event_type == "llm_call": + prompt = event.get("prompt", "") + completion = event.get("completion", "") + content = f"PROMPT:\n{prompt}\n\nCOMPLETION:\n{completion}" + + input_tokens = event.get("input_tokens") + output_tokens = event.get("output_tokens") + tokens = None + if input_tokens is not None or output_tokens is not None: + tokens = { + "input": input_tokens if input_tokens is not None else 0, + "output": output_tokens if output_tokens is not None else 0 + } + + step = TraceStep( + step_id=step_id_int, + type="thought", + content=content, + tool_name=None, + tool_args=None, + tool_result=None, + timestamp=timestamp, + error=None, + finish_reason=event.get("finish_reason"), + provider=event.get("provider"), + model=event.get("model"), + tokens=tokens, + latency_ms=event.get("latency_ms"), + divergent=is_divergent + ) + steps.append(step) + + elif event_type == "tool_call": + decision = event.get("permission_decision") + tool_name = event.get("tool_name") + arguments = event.get("arguments") + + if decision == "allowed": + step = TraceStep( + step_id=step_id_int, + type="tool_call", + content="", + tool_name=tool_name, + tool_args=arguments, + tool_result=event.get("response"), + timestamp=timestamp, + error=None, + denied=False, + latency_ms=event.get("latency_ms"), + divergent=is_divergent + ) + steps.append(step) + elif decision in ("denied", "validation_error"): + reason = event.get("permission_reason", f"Call {decision}") + human_note = f"ATTEMPTED TOOL: {tool_name}\nATTEMPTED ARGS: {arguments}" + + step = TraceStep( + step_id=step_id_int, + type="tool_call", + content=human_note, + tool_name=tool_name, + tool_args=arguments, + tool_result=None, + timestamp=timestamp, + error=reason, + denied=True, + latency_ms=event.get("latency_ms"), + divergent=is_divergent + ) + steps.append(step) + + return AgentTrace( + task_id=task_id, + input=input_text, + steps=steps, + final_output=final_output, + actual_state=actual_state, + metadata=metadata + ) diff --git a/openeval/cli.py b/openeval/cli.py index e17ef0d..2333b7c 100644 --- a/openeval/cli.py +++ b/openeval/cli.py @@ -114,14 +114,15 @@ def run( @app.command() def report( - input: Path = typer.Option(..., help="Path to results directory") + input: Path = typer.Option(..., help="Path to results directory"), + fail_under: Optional[float] = typer.Option(None, "--fail-under", help="Minimum average metric score (0.0 to 1.0) required to pass.") ): """ Generate a report from evaluation results. Exit Code Contract: - - 0: All valid JSON files parsed successfully and report generated. - - 1: One or more JSON files failed to load (corrupted/malformed), or invalid arguments/missing directory. + - 0: All valid JSON files parsed successfully, report generated, and score meets threshold (if provided). + - 1: One or more JSON files failed to load, invalid arguments, or score below threshold. """ if not input.exists() or not input.is_dir(): typer.echo(f"Error: Input directory not found: {input}", err=True) @@ -131,11 +132,23 @@ def report( test_cases = [] has_errors = False + total_score = 0.0 + total_metrics = 0 + for file_path in input.glob("*.json"): try: with open(file_path, "r", encoding="utf-8") as f: data = json.load(f) test_cases.append(data) + + # Check for test execution errors (hard failures) + if "error" in data: + has_errors = True + + # Aggregate scores + for metric in data.get("metrics", {}).values(): + total_score += metric.get("score", 0.0) + total_metrics += 1 except Exception as e: typer.echo(f"Warning: Failed to load {file_path}: {e}", err=True) has_errors = True @@ -147,6 +160,20 @@ def report( typer.echo(format_report(results)) + # Calculate average score + avg_score = (total_score / total_metrics) if total_metrics > 0 else 0.0 + + if fail_under is not None: + if total_metrics == 0: + typer.echo("Error: No metrics evaluated.", err=True) + raise typer.Exit(code=1) + + typer.echo(f"\nOverall Average Score: {avg_score:.2f}") + typer.echo(f"Required Threshold: {fail_under:.2f}") + if avg_score < fail_under: + typer.echo("Eval score is below the fail-under threshold.", err=True) + raise typer.Exit(code=1) + if has_errors: raise typer.Exit(code=1) diff --git a/openeval/metrics.py b/openeval/metrics.py index 5c58a29..3f6de58 100644 --- a/openeval/metrics.py +++ b/openeval/metrics.py @@ -23,7 +23,7 @@ class ToolSelectionAccuracy(BaseMetric): description = "Measures the accuracy of tool selection against expected tools." def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: - actual_tools = [step.tool_name for step in trace.steps if step.type == "tool_call" and step.tool_name is not None] + actual_tools = [step.tool_name for step in trace.steps if step.type == "tool_call" and step.tool_name is not None and not step.denied] expected_tools = [call["tool"] for call in test_case.expected_tool_calls] if not expected_tools: @@ -64,7 +64,7 @@ def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: if not expected_tool_calls: return MetricResult(self.name, 1.0, True, "No expected tools, no arguments to evaluate.") - actual_tools = [step for step in trace.steps if step.type == "tool_call" and step.tool_name is not None] + actual_tools = [step for step in trace.steps if step.type == "tool_call" and step.tool_name is not None and not step.denied] # Since duplicate names are asserted away, we can just use the first instance safely in a dict actual_by_name = {step.tool_name: (step.tool_args or {}) for step in actual_tools} @@ -175,3 +175,118 @@ def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: def explain(self, result: MetricResult) -> str: raise NotImplementedError() + + +class DenialRecoveryRate(BaseMetric): + """ + DenialRecoveryRate: ratio of permission denials or validation errors the agent successfully recovered from. + """ + name = "Denial Recovery Rate" + description = "Measures the agent's ability to recover from permission denials or validation errors." + + def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: + denied_steps = [(i, step) for i, step in enumerate(trace.steps) if step.type == "tool_call" and step.denied] + + if not denied_steps: + return MetricResult( + metric_name=self.name, + score=1.0, + passed=True, + details="No permission denials or validation errors encountered." + ) + + recovered_count = 0 + total_denials = len(denied_steps) + + for i, denied_step in denied_steps: + subsequent_steps = trace.steps[i + 1:] + + # A denial is NOT recovered if the agent retried the identical denied tool call anywhere later in the trace + repeated_later = any( + s.type == "tool_call" and + s.tool_name == denied_step.tool_name and + s.tool_args == denied_step.tool_args and + s.denied + for s in subsequent_steps + ) + + has_successful_subsequent = any( + (s.type == "tool_call" and s.tool_name is not None and not s.denied) or (s.type == "output") or bool(trace.final_output) + for s in subsequent_steps + ) + + if has_successful_subsequent and not repeated_later: + recovered_count += 1 + + score = recovered_count / total_denials + passed = score == 1.0 + details = f"Recovered from {recovered_count} of {total_denials} permission denials/validation errors." + return MetricResult(self.name, float(score), passed, details) + + def explain(self, result: MetricResult) -> str: + return f"{self.name}: {result.details} Score: {result.score:.2f}." + + +class DivergenceScore(BaseMetric): + """ + DivergenceScore: trajectory fidelity score in Verified Replay comparisons. + """ + name = "Divergence Score" + description = "Measures trajectory agreement with recorded baseline trace in Verified Replay." + + def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: + if not trace.steps: + return MetricResult(self.name, 1.0, True, "Trace has no steps to evaluate for divergence.") + + divergent_indices = [i for i, step in enumerate(trace.steps) if step.divergent] + + if not divergent_indices: + return MetricResult(self.name, 1.0, True, "No Verified Replay divergence detected.") + + first_div_idx = divergent_indices[0] + total_steps = len(trace.steps) + score = first_div_idx / total_steps + passed = score == 1.0 + details = f"Divergence detected at step {first_div_idx + 1} of {total_steps}." + return MetricResult(self.name, float(score), passed, details) + + def explain(self, result: MetricResult) -> str: + return f"{self.name}: {result.details} Score: {result.score:.2f}." + + +class FixtureFreshness(BaseMetric): + """ + FixtureFreshness: checks if the trace fixture's fingerprint verdict indicates a fresh agent spec (PASS). + Explicitly distinguishes fresh specs (PASS), genuine drift regressions (DRIFTED), test setup errors (UNVERIFIED), + and un-evaluated traces (no fingerprint metadata present). + """ + name = "Fixture Freshness" + description = "Evaluates whether the trace fixture's tool registry and environment fingerprint is fresh (PASS) vs stale (DRIFTED), unverified (UNVERIFIED), or not evaluated." + + def score(self, trace: AgentTrace, test_case: EvalTestCase) -> MetricResult: + has_fingerprint = "fingerprint" in trace.metadata + has_verdict = "verdict" in trace.metadata or "fingerprint_verdict" in trace.metadata + + if not has_fingerprint and not has_verdict: + return MetricResult( + self.name, + 1.0, + None, + "NOT EVALUATED: Trace was not checked for drift (no fingerprint metadata present)." + ) + + verdict = trace.metadata.get("verdict") or trace.metadata.get("fingerprint_verdict") or "UNVERIFIED" + verdict_str = str(verdict).upper() + + if verdict_str == "PASS": + return MetricResult(self.name, 1.0, True, "Fixture fingerprint matches current agent spec (PASS).") + elif verdict_str == "DRIFTED": + return MetricResult(self.name, 0.0, False, "DRIFTED: Fixture fingerprint has drifted from current agent spec (genuine drift regression).") + else: + return MetricResult(self.name, 0.0, False, f"UNVERIFIED: Fixture fingerprint status is {verdict_str} (check-drift could not verify against agent spec — test setup or config issue).") + + def explain(self, result: MetricResult) -> str: + return f"{self.name}: {result.details} Score: {result.score:.2f}." + + + diff --git a/openeval/models.py b/openeval/models.py index 344143a..23de590 100644 --- a/openeval/models.py +++ b/openeval/models.py @@ -11,6 +11,13 @@ class TraceStep: tool_result: str | None timestamp: float error: str | None = None + denied: bool = False + finish_reason: str | None = None + provider: str | None = None + model: str | None = None + tokens: dict[str, int] | None = None + latency_ms: float | None = None + divergent: bool = False @dataclass class AgentTrace: @@ -35,5 +42,6 @@ class EvalTestCase: class MetricResult: metric_name: str score: float - passed: bool + passed: bool | None details: str + diff --git a/openeval/report.py b/openeval/report.py index e1f5617..55a78e6 100644 --- a/openeval/report.py +++ b/openeval/report.py @@ -26,7 +26,8 @@ def format_report(results: Dict[str, Any], *, timestamp: Optional[str] = None) - has_failures = True tool_sel = arg_corr = step_eff = goal_comp = "-" else: - all_passed = all(m.get("passed", False) for m in metrics.values()) + evaluable_metrics = [m for m in metrics.values() if m.get("passed") is not None] + all_passed = all(m.get("passed") is True for m in evaluable_metrics) if evaluable_metrics else True status = "PASS" if all_passed else "FAIL" if not all_passed: has_failures = True @@ -86,7 +87,7 @@ def get_score(name: str) -> str: else: for m_name in sorted(tc["metrics"].keys()): m = tc["metrics"][m_name] - if not m.get("passed", False): + if m.get("passed") is False: score = m.get("score", 0.0) details = m.get("details", "") lines.append(f"- **{m_name}:** {score:.2f}") diff --git a/openeval/runner.py b/openeval/runner.py index af1588f..8d6b5a0 100644 --- a/openeval/runner.py +++ b/openeval/runner.py @@ -73,7 +73,15 @@ def run_suite(suite_dir: Path, metrics: list[BaseMetric]) -> dict[str, dict]: tool_name=step_data.get("tool_name"), tool_args=step_data.get("tool_args"), tool_result=step_data.get("tool_result"), - timestamp=step_data["timestamp"] + timestamp=step_data["timestamp"], + error=step_data.get("error"), + denied=step_data.get("denied", False), + finish_reason=step_data.get("finish_reason"), + provider=step_data.get("provider"), + model=step_data.get("model"), + tokens=step_data.get("tokens"), + latency_ms=step_data.get("latency_ms"), + divergent=step_data.get("divergent", False) )) trace = AgentTrace( diff --git a/tests/test_fixtura_adapter.py b/tests/test_fixtura_adapter.py new file mode 100644 index 0000000..280a6fb --- /dev/null +++ b/tests/test_fixtura_adapter.py @@ -0,0 +1,25 @@ +import os +import pytest +from pathlib import Path + +from openeval.adapters.fixtura import from_fixtura_trace, extract_step_id +from openeval.models import AgentTrace + +try: + import zstandard as zstd +except ImportError: + zstd = None + +def test_extract_step_id(): + assert extract_step_id("step-000001") == 1 + assert extract_step_id("step_42") == 42 + with pytest.raises(ValueError): + extract_step_id("no_digits_here") + +@pytest.mark.skipif(zstd is None, reason="zstandard optional extra required for fixtura trace conversion") +def test_from_fixtura_trace_missing_file(tmp_path: Path): + missing = tmp_path / "non_existent.trace" + trace = from_fixtura_trace(missing, task_id="t1", input_text="hello") + assert isinstance(trace, AgentTrace) + assert trace.task_id == "t1" + assert len(trace.steps) == 0 diff --git a/tests/test_metrics.py b/tests/test_metrics.py index 96276fe..272f5d5 100644 --- a/tests/test_metrics.py +++ b/tests/test_metrics.py @@ -202,3 +202,173 @@ def test_goal_completion_rate_empty_expected(): tr_empty = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={"a": 1}, metadata={}) assert metric.score(tr_empty, tc_empty).score == 1.0 +def test_tool_selection_accuracy_with_denied_step(): + metric = ToolSelectionAccuracy() + tc = EvalTestCase( + task_id="denied-test", + input="", + expected_tool_calls=[{"tool": "read_file"}], + expected_final_state={}, + expected_output_contains=[], + max_steps=5, + timeout_seconds=5.0 + ) + # Step 1 attempted 'delete_file' but was denied. Step 2 succeeded 'read_file'. + steps = [ + TraceStep(step_id=1, type="tool_call", content="ATTEMPTED TOOL: delete_file", tool_name="delete_file", tool_args={}, tool_result=None, timestamp=0.0, denied=True, error="Permission denied"), + TraceStep(step_id=2, type="tool_call", content="", tool_name="read_file", tool_args={}, tool_result="ok", timestamp=1.0, denied=False) + ] + trace = AgentTrace(task_id="denied-test", input="", steps=steps, final_output="", actual_state={}, metadata={}) + result = metric.score(trace, tc) + # Only read_file should be counted in actual_tools, matching expected 1/1 + assert result.score == 1.0 + assert result.passed is True + # Verify raw tool_name and tool_args remain intact on the denied step + assert steps[0].tool_name == "delete_file" + assert steps[0].tool_args == {} + assert steps[0].denied is True + +def test_argument_correctness_with_denied_step(): + metric = ArgumentCorrectness() + tc = EvalTestCase( + task_id="denied-arg-test", + input="", + expected_tool_calls=[{"tool": "read_file", "args": {"path": "a.txt"}}], + expected_final_state={}, + expected_output_contains=[], + max_steps=5, + timeout_seconds=5.0 + ) + steps = [ + TraceStep(step_id=1, type="tool_call", content="", tool_name="read_file", tool_args={"path": "forbidden.txt"}, tool_result=None, timestamp=0.0, denied=True, error="Denied"), + TraceStep(step_id=2, type="tool_call", content="", tool_name="read_file", tool_args={"path": "a.txt"}, tool_result="data", timestamp=1.0, denied=False) + ] + trace = AgentTrace(task_id="denied-arg-test", input="", steps=steps, final_output="", actual_state={}, metadata={}) + result = metric.score(trace, tc) + # Denied step argument path="forbidden.txt" should be excluded, allowed step path="a.txt" matched -> 1.0 + assert result.score == 1.0 + assert result.passed is True + +from openeval.metrics import DenialRecoveryRate, DivergenceScore, FixtureFreshness + +def test_denial_recovery_rate(): + metric = DenialRecoveryRate() + tc = EvalTestCase(task_id="t", input="", expected_tool_calls=[], expected_final_state={}, expected_output_contains=[], max_steps=5, timeout_seconds=5.0) + + # 1. Zero denials -> score 1.0 + tr_clean = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={}, metadata={}) + res_clean = metric.score(tr_clean, tc) + assert res_clean.score == 1.0 + assert res_clean.passed is True + + # 2. Denied then recovered via subsequent allowed tool call -> score 1.0 + tr_rec = AgentTrace( + task_id="t", + input="", + steps=[ + TraceStep(step_id=1, type="tool_call", content="", tool_name="delete", tool_args={}, tool_result=None, timestamp=0.0, denied=True), + TraceStep(step_id=2, type="tool_call", content="", tool_name="read", tool_args={}, tool_result="ok", timestamp=1.0, denied=False) + ], + final_output="Done", + actual_state={}, + metadata={} + ) + res_rec = metric.score(tr_rec, tc) + assert res_rec.score == 1.0 + assert res_rec.passed is True + + # 3. Repeated exact denial immediately or non-adjacently -> unrecovered, score 0.0 + tr_fail = AgentTrace( + task_id="t", + input="", + steps=[ + TraceStep(step_id=1, type="tool_call", content="", tool_name="delete", tool_args={"path": "sys"}, tool_result=None, timestamp=0.0, denied=True), + TraceStep(step_id=2, type="tool_call", content="", tool_name="delete", tool_args={"path": "sys"}, tool_result=None, timestamp=1.0, denied=True) + ], + final_output="", + actual_state={}, + metadata={} + ) + res_fail = metric.score(tr_fail, tc) + assert res_fail.score == 0.0 + assert res_fail.passed is False + + # 4. Non-adjacent retry (denied at step 4, took other steps, retried same call at step 10) -> unrecovered, score 0.0 + tr_non_adj = AgentTrace( + task_id="t", + input="", + steps=[ + TraceStep(step_id=4, type="tool_call", content="", tool_name="delete", tool_args={"path": "sys"}, tool_result=None, timestamp=0.0, denied=True), + TraceStep(step_id=5, type="tool_call", content="", tool_name="read", tool_args={"path": "notes.txt"}, tool_result="ok", timestamp=1.0, denied=False), + TraceStep(step_id=10, type="tool_call", content="", tool_name="delete", tool_args={"path": "sys"}, tool_result=None, timestamp=2.0, denied=True) + ], + final_output="Done", + actual_state={}, + metadata={} + ) + res_non_adj = metric.score(tr_non_adj, tc) + assert res_non_adj.score == 0.0 + assert res_non_adj.passed is False + +def test_divergence_score(): + metric = DivergenceScore() + tc = EvalTestCase(task_id="t", input="", expected_tool_calls=[], expected_final_state={}, expected_output_contains=[], max_steps=5, timeout_seconds=5.0) + + # No divergence -> 1.0 + tr_no_div = AgentTrace( + task_id="t", input="", + steps=[TraceStep(step_id=1, type="thought", content="", tool_name=None, tool_args=None, tool_result=None, timestamp=0.0)], + final_output="", actual_state={}, metadata={} + ) + assert metric.score(tr_no_div, tc).score == 1.0 + + # Divergence at step 3 of 4 steps -> score 2/4 = 0.5 + tr_div = AgentTrace( + task_id="t", input="", + steps=[ + TraceStep(step_id=1, type="tool_call", content="", tool_name="t1", tool_args={}, tool_result="ok", timestamp=0.0), + TraceStep(step_id=2, type="tool_call", content="", tool_name="t2", tool_args={}, tool_result="ok", timestamp=1.0), + TraceStep(step_id=3, type="tool_call", content="", tool_name="t3", tool_args={}, tool_result="diff", timestamp=2.0, divergent=True), + TraceStep(step_id=4, type="thought", content="", tool_name=None, tool_args=None, tool_result=None, timestamp=3.0) + ], + final_output="", actual_state={}, metadata={} + ) + res_div = metric.score(tr_div, tc) + assert res_div.score == 0.5 + assert res_div.passed is False + +def test_fixture_freshness(): + metric = FixtureFreshness() + tc = EvalTestCase(task_id="t", input="", expected_tool_calls=[], expected_final_state={}, expected_output_contains=[], max_steps=5, timeout_seconds=5.0) + + # 1. PASS -> 1.0 + tr_pass = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={}, metadata={"verdict": "PASS", "fingerprint": {}}) + res_pass = metric.score(tr_pass, tc) + assert res_pass.score == 1.0 + assert res_pass.passed is True + + # 2. DRIFTED -> 0.0 + tr_drift = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={}, metadata={"verdict": "DRIFTED", "fingerprint": {}}) + res_drift = metric.score(tr_drift, tc) + assert res_drift.score == 0.0 + assert res_drift.passed is False + assert "DRIFTED" in res_drift.details + + # 3. UNVERIFIED (fingerprint present, check-drift failed/unverified) -> 0.0 + tr_unv = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={}, metadata={"verdict": "UNVERIFIED", "fingerprint": {}}) + res_unv = metric.score(tr_unv, tc) + assert res_unv.score == 0.0 + assert res_unv.passed is False + assert "UNVERIFIED" in res_unv.details + + # 4. Zero fingerprint metadata at all (trace was never check-drifted) -> 1.0, NOT EVALUATED (passed=None) + tr_none = AgentTrace(task_id="t", input="", steps=[], final_output="", actual_state={}, metadata={}) + res_none = metric.score(tr_none, tc) + assert res_none.score == 1.0 + assert res_none.passed is None + assert "NOT EVALUATED" in res_none.details + + + + +