Measuring what makes AI agents reliable in production-like scenarios. Open benchmarks and controlled harness experiments.

We study one question:

What makes an AI agent remain useful when tasks become long, tools become risky, context becomes crowded, and execution gets interrupted?

Longer context windows expand what an agent can access. They do not decide what belongs in the active working set, what should be persisted or forgotten, when an action needs permission, how sub-agents should hand work back, or whether execution can recover after interruption.

Agent Reliability Lab turns those choices into reproducible system and product evaluations.

The Deep Research Harness Eval compares four cumulative configurations:

1. ReAct baseline
2. Baseline + context compaction
3. Compaction + permission gate
4. Permission gate + structured sub-agents

The primary metrics are:

• Evidence-Grounded Task Success Rate
• Cost per Successful Task

The primary evaluation uses a frozen twenty-task suite. A separate cross-model subset tests whether configuration rankings transfer without mixing model effects into the main causal claim.

1. Freeze inputs and version source snapshots.
2. Separate model capability from infrastructure failure.
3. Measure systems from raw traces, not screenshots.
4. Change one architectural decision at a time.
5. Publish badcases and failure metadata.
6. Report quality and cost together.
7. State limitations explicitly.

Built by Melody Ling.