feat: CI gate wiring

ARCHITECTURE.md

@@ -0,0 +1,208 @@
+# OpenEval Architecture & System Specification
+
+**High-Performance, Purely Deterministic AI Agent Trajectory Evaluation System.**
+
+---
+
+## 1. Executive Design Philosophy
+
+OpenEval evaluates **what an AI agent did**, not merely what it said.
+
+Modern AI agent evaluation is dominated by "LLM-as-a-Judge" frameworks (e.g. DeepEval, RAGAS, Braintrust). While useful for qualitative copy generation, LLM judges introduce three fundamental flaws to CI/CD pipelines:
+1. **Flakiness & Non-Determinism:** The same agent trace scored twice by an LLM judge can yield different scores or verdicts.
+2. **High Latency & API Cost:** Scoring 100 test cases with an LLM judge requires hundreds of secondary LLM calls, incurring latency and recurring API costs.
+3. **Inability to Verify System State:** LLM judges evaluate text outputs; they cannot inspect whether a database operation succeeded, whether a permission policy was respected, or whether a tool call argument matched exact schema boundaries.
+
+OpenEval replaces judge LLMs with **100% deterministic pure functions**. Given an `AgentTrace` and an `EvalTestCase`, OpenEval computes scores reproducibly in `<1ms` with `$0.00` API cost.
+
+---
+
+## 2. System Architecture & Components
+
+```
++-----------------------------------------------------------------------------------+
+|                                 INPUT TRAJECTORIES                                |
+|   +-------------------+    +--------------------+    +------------------------+   |
+|   |  Fixtura .trace   |    |   LangChain Run    |    |   OpenAI Messages List |   |
+|   |  (Compressed)     |    |   (Run Tree)       |    |   (ChatCompletion API) |   |
+|   +---------+---------+    +---------+----------+    +-----------+------------+   |
++-------------|------------------------|---------------------------|----------------+
+              |                        |                           |
+              v                        v                           v
++-----------------------------------------------------------------------------------+
+|                                ADAPTER SUBSYSTEM                                  |
+|   +-------------------+    +--------------------+    +------------------------+   |
+|   | from_fixtura_trace|    | from_langchain_run |    | from_openai_messages   |   |
+|   +---------+---------+    +---------+----------+    +-----------+------------+   |
++-------------|------------------------|---------------------------|----------------+
+              +------------------------+---------------------------+
+                                       |
+                                       v
++-----------------------------------------------------------------------------------+
+|                            CORE DATA MODEL (Pure Dataclasses)                     |
+|                                                                                   |
+|   AgentTrace                                                                      |
+|   ├── task_id: str                                                                |
+|   ├── input: str                                                                  |
+|   ├── final_output: str                                                           |
+|   ├── actual_state: dict                                                          |
+|   ├── metadata: dict  {"fingerprint", "verdict"}                                  |
+|   └── steps: list[TraceStep]                                                      |
+|       ├── step_id: int                                                            |
+|       ├── type: "thought" | "tool_call" | "tool_result" | "output"                 |
+|       ├── tool_name: str | None                                                   |
+|       ├── tool_args: dict | None                                                   |
+|       ├── tool_result: str | None                                                 |
+|       ├── denied: bool (permission denial or validation error)                    |
+|       ├── finish_reason: str | None                                               |
+|       ├── provider, model, tokens, latency_ms                                     |
+|       └── divergent: bool (Verified Replay divergence marker)                     |
++--------------------------------------|--------------------------------------------+
+                                       |
+                                       v
++-----------------------------------------------------------------------------------+
+|                              DETERMINISTIC EVAL ENGINE                            |
+|                                                                                   |
+|   EvalTestCase                                                                    |
+|   ├── expected_tool_calls: list[dict]                                             |
+|   ├── expected_final_state: dict                                                  |
+|   └── max_steps / timeout_seconds                                                 |
+|                                                                                   |
+|   +---------------------------------------------------------------------------+   |
+|   |                            METRIC PIPELINE                                |   |
+|   |  1. ToolSelectionAccuracy (ratio of correct tool calls)                   |   |
+|   |  2. ArgumentCorrectness (key-value exact argument match)                  |   |
+|   |  3. StepEfficiency (optimal vs actual steps ratio)                        |   |
+|   |  4. GoalCompletionRate (state transition verification)                    |   |
+|   |  5. DenialRecoveryRate (permission denial recovery without loops)         |   |
+|   |  6. DivergenceScore (Verified Replay trajectory fidelity)                 |   |
+|   |  7. FixtureFreshness (environment fingerprint drift check)                |   |
+|   +-------------------------------------+-------------------------------------+   |
++-----------------------------------------|-----------------------------------------+
+                                          |
+                                          v
++-----------------------------------------------------------------------------------+
+|                               OUTPUT & REPORTING                                  |
+|   +--------------------+     +--------------------+     +---------------------+   |
+|   | MetricResult       | --> | Suite Runner       | --> | CLI & GitHub Action |   |
+|   | (score, passed,    |     | (openeval.runner)  |     | (openeval.report)   |   |
+|   |  details)          |     +--------------------+     +---------------------+   |
+|   +--------------------+                                                          |
++-----------------------------------------------------------------------------------+
+```
+
+---
+
+## 3. Data Models Specification (`openeval/models.py`)
+
+### 3.1 `TraceStep`
+Represents an individual step in an agent's execution sequence.
+
+```python
+@dataclass
+class TraceStep:
+    step_id: int
+    type: Literal["thought", "tool_call", "tool_result", "output"]
+    content: str
+    tool_name: str | None
+    tool_args: dict | None
+    tool_result: str | None
+    timestamp: float
+    error: str | None = None
+
+    # Extended telemetry fields
+    denied: bool = False
+    finish_reason: str | None = None
+    provider: str | None = None
+    model: str | None = None
+    tokens: dict[str, int] | None = None
+    latency_ms: float | None = None
+    divergent: bool = False
+```
+
+### 3.2 `AgentTrace`
+The complete recorded trajectory of an agent run.
+
+```python
+@dataclass
+class AgentTrace:
+    task_id: str
+    input: str
+    steps: list[TraceStep]
+    final_output: str
+    actual_state: dict
+    metadata: dict
+```
+
+### 3.3 `EvalTestCase`
+Defines the expected ground-truth requirements for a task.
+
+```python
+@dataclass
+class EvalTestCase:
+    task_id: str
+    input: str
+    expected_tool_calls: list[dict]
+    expected_final_state: dict
+    expected_output_contains: list[str]
+    max_steps: int
+    timeout_seconds: float
+```
+
+### 3.4 `MetricResult`
+The outcome of evaluating a metric against a trace and test case.
+
+```python
+@dataclass
+class MetricResult:
+    metric_name: str
+    score: float
+    passed: bool | None  # True (Passed), False (Failed), None (Not Applicable)
+    details: str
+```
+
+---
+
+## 4. Complete Metric Specification
+
+### 4.1 `ToolSelectionAccuracy`
+- **Formula:** $\text{Score} = \frac{\text{Count}(\text{actual\_tools} \cap \text{expected\_tools})}{\text{Count}(\text{expected\_tools})}$
+- **Denial Guard:** Excludes steps where `step.denied is True`. Denied tool calls are preserved on the step for denial metrics, but do not count as successfully executed tool selections.
+
+### 4.2 `ArgumentCorrectness`
+- **Formula:** $\text{Score} = \frac{\text{Count}(\text{matching\_key\_value\_pairs})}{\text{Total Evaluated Expected Arguments}}$
+- **Denial Guard:** Evaluates arguments of executed (non-denied) tool calls.
+
+### 4.3 `StepEfficiency`
+- **Formula:** $\text{Score} = \min\left(1.0, \frac{\text{optimal\_steps}}{\text{actual\_steps\_taken}}\right)$
+- **Behavior:** Penalizes unnecessary extra tool steps.
+
+### 4.4 `GoalCompletionRate`
+- **Formula:** $\text{Score} = \frac{\text{Count}(\text{actual\_state}[k] == \text{expected\_state}[k])}{\text{Total Expected Keys}}$
+- **Behavior:** Verifies environment state transitions.
+
+### 4.5 `DenialRecoveryRate`
+- **Formula:** $\text{Score} = \frac{\text{Count}(\text{recovered\_denials})}{\text{Total Denials Encountered}}$
+- **Behavior:** Evaluates if the agent experienced a permission denial or validation error and:
+  1. Did **not** retry the exact same forbidden call (`tool_name` + `tool_args`) anywhere in the remainder of the trace.
+  2. Subsequently executed an allowed tool call or produced final output.
+- **Zero Denials:** Returns `score = 1.0, passed = True, details = "No permission denials..."`.
+
+### 4.6 `DivergenceScore`
+- **Formula:** $\text{Score} = \frac{\text{Step Index of First Divergence}}{\text{Total Steps}}$
+- **Behavior:** Evaluates trajectory agreement during Verified Replay. Returns `1.0` if zero divergence occurred.
+
+### 4.7 `FixtureFreshness`
+- **Behavior:** Evaluates `trace.metadata.get("verdict")`:
+  - `"PASS"` $\rightarrow$ `score = 1.0, passed = True`
+  - `"DRIFTED"` $\rightarrow$ `score = 0.0, passed = False` (genuine drift regression)
+  - `"UNVERIFIED"` $\rightarrow$ `score = 0.0, passed = False` (test setup / config error)
+  - **No Fingerprint Metadata** $\rightarrow$ `score = 1.0, passed = None` (`NOT EVALUATED`)
+
+---
+
+## 5. Execution & Reporting Pipeline
+
+1. **`openeval.runner`:** Loads trace JSON or converts `.trace` files, pairs them with `EvalTestCase` definitions, executes selected metrics, and produces structured result dictionaries.
+2. **`openeval.report`:** Formats raw result dictionaries into deterministic Markdown or JSON reports.
+3. **Verdict Filtering:** Metrics returning `passed: None` (`NOT EVALUATED`) are excluded from failure lists and suite pass-rate denominators, eliminating false positives and false failures in CI gates.

CHANGELOG.md

@@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-07-26
+
+### Added
+- **Extended Data Model (`TraceStep` & `AgentTrace`):** Additive support for `denied`, `finish_reason`, `provider`, `model`, `tokens`, `latency_ms`, and replay `divergent` fields.
+- **Three New Deterministic Metrics:** Added `DenialRecoveryRate` (permission denial recovery), `DivergenceScore` (Verified Replay trajectory agreement), and `FixtureFreshness` (tool registry fingerprint drift detection).
+- **Native Fixtura Trace Adapter:** Added `openeval.adapters.fixtura.from_fixtura_trace` for native zstd `.trace` ingestion.
+- **Disambiguated Verdicts (`passed: bool | None`):** `MetricResult.passed` extended to support `None` (`NOT EVALUATED` / `N/A`), preventing un-drift-checked traces from distorting aggregate suite pass rates.
+- **Comprehensive Technical Docs:** Added `ARCHITECTURE.md` and detailed architectural proposals in `docs/proposals/`.
+
 ## [0.1.2] - 2026-07-03
 
 ### Added