From 4e1386aca2429e59818cc4638be8e78d175ce10d Mon Sep 17 00:00:00 2001
From: itseffi <15998472+itseffi@users.noreply.github.com>
Date: Sun, 24 May 2026 21:49:07 +0200
Subject: [PATCH] v2

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---
 README.md           | 29 ++++++++++++++++++++++++++---
 docs/task-schema.md | 22 ++++++++++++++++++++++
 2 files changed, 48 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 3df906c..c7d5af4 100644
--- a/README.md
+++ b/README.md
@@ -136,7 +136,30 @@ Generate local baseline artifacts for an environment:
 python3 -m agenticevals env-baselines examples.tau_retail_env:TauRetailEnv --agents scripted,noop --max-items 3 --trials 2
 ```
 
-Baseline artifacts include pass@1, pass^k, bootstrap confidence intervals, and cost-per-success when the adapter exposes token/cost accounting.
+Baseline artifacts include pass@1, pass^k, confidence intervals, and cost-per-success when the adapter exposes token/cost accounting. Proportion metrics (`pass_rate`, `pass_at_1`, `pass_power_k`) use a **Wilson score interval**, which stays non-degenerate at 0%/100% where a percentile bootstrap collapses to zero width; continuous metrics keep bootstrap CIs. `pass^k` is the unbiased `C(c,k)/C(n,k)` estimator. A `saturated` flag marks suites where every agent lands at the same extreme (no discriminative signal).
+
+## Judge Calibration & Release Gating
+
+LLM-judge verifiers (`llm_rubric`) return a reason-first **binary verdict**. Useful per-rubric config keys:
+
+- `rubric`: the grading instruction.
+- `threshold`: pass threshold when only a `score` is returned (default `0.5`).
+- `repetitions`: judge the trajectory N times and majority-vote (ties break on mean score), surfacing variance a single cached call hides.
+- `transcript_max_chars` / `transcript_max_steps`: how much trajectory the judge sees (default 500 chars/step).
+- `provider` / `model` / `timeout` / `max_tokens`.
+
+Validate a judge before trusting its scores:
+
+```bash
+# 1. Sample real judge decisions from runs into a balanced labeling template
+python3 -m agenticevals make-calibration-set runs/<suite-run-dir> -o labels.jsonl --size 100
+# 2. Fill in human_passed for each row, then compute agreement
+python3 -m agenticevals calibrate-judge labels.jsonl
+# 3. Gate a release on baselines + judge agreement
+python3 -m agenticevals release-gate --baselines runs/<baselines-dir>/baselines.json --calibration labels.calibration.json
+```
+
+The calibration report includes accuracy, Cohen's kappa, and per-class **TPR/TNR**. The release gate enforces kappa, TPR/TNR (≥0.70 when present), a minimum labeled-sample size, and rejects a saturated baseline.
 
 ## Data Export
 
@@ -199,8 +222,8 @@ AGENTICEVALS_TRACES_PATH="/path/to/agenticevals/traces"
 AGENTICEVALS_ENV_TIMEOUT=10000
 AGENTICEVALS_ACTION_SHORT_TIMEOUT=60
 AGENTICEVALS_ACTION_LONG_TIMEOUT=10000
-AGENTICEVALS_AGENT_MAX_STEPS=50
-AGENTICEVALS_MODEL_MAX_RETRIES=3
+AGENTICEVALS_AGENT_MAX_STEPS=50      # effective ceiling: caps each task's limits.max_steps
+AGENTICEVALS_MODEL_MAX_RETRIES=3     # retries on 429/5xx/network with exponential backoff
 
 AGENTICEVALS_DEFAULT_AGENT=scripted
 AGENTICEVALS_HTTP_AGENT_URL="http://127.0.0.1:8000/run"
diff --git a/docs/task-schema.md b/docs/task-schema.md
index 9fe8f96..3ab102e 100644
--- a/docs/task-schema.md
+++ b/docs/task-schema.md
@@ -153,6 +153,28 @@ Tasks can declare tools and the HTTP endpoints that execute them:
 
 Tool dispatches are recorded in `trajectory.jsonl` as `tool.dispatch` events and in `trajectory.json` as typed tool-call/tool-result steps.
 
+## Score Weights
+
+The optional `score` block sets the point budget for each default verifier dimension. Magnitudes are relative — the reward is the weight-normalized average across all emitted criteria.
+
+```json
+{
+  "score": {
+    "command_checks": 40,
+    "file_checks": 20,
+    "browser_checks": 20,
+    "git_policy": 20,
+    "expected_actions": 25,
+    "audit_safety": 25,
+    "tool_dispatch": 25,
+    "tool_argument": 25,
+    "tool_safety": 25
+  }
+}
+```
+
+All nine are configurable; the values above are the defaults. Raise `tool_*` relative to `file_checks`, for example, to weight tool correctness more heavily.
+
 ## Verifiers
 
 Runs write `reward.json` and `reward-details.json` from verifier components. If `verifiers` is omitted, the runner builds default verifiers from `checks`, `policies`, `tools`, `expected_actions`, and `safety_checks`.