fix(eng-l1-a): distinguish exact and approximate issue matches - #12
fix(eng-l1-a): distinguish exact and approximate issue matches#12mrojas54 wants to merge 10 commits into
Conversation
…atches Criterion 6 fails on every trial because "related engineering issue" is undefined. It admits a structural reading (shares the component field) and a semantic one (describes the same defect). The rubric scores the first; the prompt's wording invites the second. An agent taking the semantic reading finds the exactly-matching issue for nine of the 31 tickets, discovers it is marked status=done, and has no choice but to report "none" -- which Criterion 6 treats as an automatic fail. Six of those nine (ISS-012, ISS-015, ISS-021, ISS-024, ISS-027, ISS-030) are purpose-built same-component, same-defect counterparts sitting at status=done. The fix is at the instruction level rather than in the data: an enterprise controls its prompts, not its issue-tracker hygiene, so a benchmark solvable only against tidy data measures something the customer cannot act on. - instruction.md: define "open", require ranking by defect similarity, and add exact/approximate/none labels. Every affected component carries 69-100 open issues, so "none" becomes unreachable while semantic ranking is still tested. - criteria.yaml: drop Criterion 6's premise that all 31 tickets have matching open issues, which is false under the semantic reading; bound the "lists all of them" weighted criterion to DISTINCT issues so it stops rewarding enumeration of ~69 duplicate backlog chores; add a weighted criterion for honest exact/approximate labelling. - trajectory.json: update the reference response, which the judge receives verbatim, so it grades the new spec rather than the old one. No data rows changed. make validate passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two documents from running the suite with claude-code / claude-opus-4-8. field-report-opus-4-8.md -- operational findings: - 30% of recorded trials were not attempts. They cost $0, submitted nothing, and scored reward 0.0, which is indistinguishable from a wrong answer. Four tasks looked like failures on that basis; re-run properly, all four passed. A run that dies on billing publishes a silently depressed score. - Job-level cost_usd excludes errored-attempt spend, so a budget guard built on it does not fire. Per-trial agent_result.cost_usd is accurate. - Per-task costs vary 7x ($0.30-$2.22/trial), so a blended average is not a usable budget input. - Reproduction section: versions, host, Docker allocation, exact commands. eng-l1-a-criterion-6.md -- the task analysis: - 8 valid trials, 0 passed, every one failing only Criterion 6. - Root cause traced through the trajectories to an undefined term intersecting a status filter, with the six done-status counterpart issues identified. - Leaderboard evidence: computer clears 14/14 at pass@10 while both claude-code entries sit at exactly 12/14 across two model generations -- the signature of a spec defect rather than a capability limit. - Proposed refinement to the SKILL.md heuristic, which currently cannot distinguish task difficulty from a spec or data defect. WIP: 7 TODO markers remain, all pending run data (k=10 tables and the patched-task verification line). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corrects a claim made in de58efd and documents two further harness issues found while trying to reproduce it. field-report-opus-4-8.md: - Section 2 rewritten. The previous claim -- that job-level cost_usd excludes errored-attempt spend -- does not reproduce and is withdrawn. In job 2026-07-27__00-43-07 the job figure equals the sum of its per-trial costs to the cent, and both of its errored trials genuinely cost $0.00. A later Opus 5 run settles the other direction: an errored trial that burned $0.9956 was included in its job's cost_usd. The ~$21 account draw was a cross-job total ($1.87 + $15.93 + $3.69 = $21.48) compared against a single job's figure. What survives is narrower but real: cost_usd is per-job and harbor emits no run-level total, so with no harbor resume every interrupted suite spans several job directories; and it reports null rather than 0 when no trial completes, which breaks a guard doing arithmetic on it. - New section 3: a run that never starts still reports a score. With the base image absent, five trials died at container build in ten seconds. The job still counted the same five as both completed and errored, and emitted pass_at_k values of 0.0 from n_trials: 0. A total infrastructure failure is therefore indistinguishable, in the scored fields, from an agent that attempted every trial and got every one wrong. This also reproduces section 1 deterministically -- no credit exhaustion and no API spend required. - New section 4: make run-task omits -k, so ATTEMPTS is silently ignored and a single-task invocation always runs at harbor's default attempt count. Per-task top-up is the only recovery path for a halted run, so anyone repairing a partial suite gets a k that does not match the suite being repaired. - Reproduction: the harbor commands were missing --ae, without which the agent starts inside Docker with no credentials (README.md:229, Makefile:28-30). Also notes the base image must still be present -- a pruned image falls through to a registry pull and kills every trial in seconds. Summary block now lists six findings; intro corrected to four harness issues. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Section 3 cited a local job directory as its evidence, which a reviewer cannot
inspect. Replaced with the commands that produce the failure, so the defect can
be verified on any machine in about ten seconds with no API spend:
docker rmi enterprise-bench/conversational-base:latest
harbor run -p tasks/eng-l1-a ... -k 5
make build-image # restore
Every trial then dies at container build -- Docker falls through to a registry
pull for an image that only ever existed locally -- and the resulting
result.json still counts the same trials as both completed and errored while
emitting pass_at_k values of 0.0 from n_trials: 0.
The eval key in the sample output is genericised to <model>, since the recipe is
model-independent.
Also adds jobs-*/ to .gitignore. The existing entry covers the default jobs/
directory but not the output of a custom --jobs-dir, which left run artifacts
untracked but committable by accident.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…irement Ran the section 3 reproduction recipe end to end against claude-opus-4-8. It reproduces the counters exactly as recorded for the earlier claude-opus-5 job -- n_completed_trials and n_errored_trials both 5 against n_total_trials 5, n_trials 0 with pass_at_k populated as 0.0, cost_usd null -- in 8 seconds with no API spend. Section 3 now notes both verifications; it previously rested on a single accidental observation. Reproduction: both API keys are needed in two places, and supplying only one fails in a way the previous text did not predict. Harbor's [verifier.env] phase reads them from the shell environment, while --ae is what forwards them into the agent container. Passing only --ae aborts the run before any trial starts with "Missing Environment Variables: OPENAI_API_KEY [verifier.env]". The Makefile covers both via the ifndef guards (Makefile:109-115) and AGENT_ENV_ARGS (Makefile:28-30); a hand-written harbor invocation must do the same. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field report told readers its figures were reproducible with scripts/verify_estimate.py "included so you can audit these numbers". The script did not exist. This adds it. It derives every figure in the report from harbor's own result.json files: trial counts, per-task cost, and pass rate, separating valid attempts from trials that died at $0 (the distinction section 1 is about). Standard library only, no third-party dependencies. Output on the run behind the report reproduces its section 1 block verbatim -- 37 recorded, 26 valid, 11 at $0, 14/26 = 54% -- and all 14 per-task costs. jobs/ is 384 MB and gitignored, so the script's output is committed alongside it at docs/verify-estimate-output.txt. That keeps the numbers checkable against the method that produced them without shipping the raw data. Co-Authored-By: Claude <noreply@anthropic.com>
…table Three things, all in the two field docs. No task or data files change. 1. Remove the seven <!-- TODO --> markers. They were invisible in rendered markdown but fully visible in the diff, and several stood in for numbers a planned k=5 suite run was meant to supply. That run did not happen, so each is resolved by stating what is actually measured rather than by deferring: coverage is 37 recorded / 26 valid across four jobs, sample sizes are 1-8 per task, and the k=10 figure is labelled as the extrapolation it is. 2. Correct section 2's job table. It omitted jobs/2026-07-27__00-02-22 -- the 23-trial suite sweep whose results populate the appendix -- and therefore reported "Total drawn $21.48" when the four jobs behind the 37 trials drew $34.05. The corrected figures strengthen the section's own point: a guard watching one job's $15.93 was blind to $14.44 already spent, not the $5.55 previously claimed. Also splits the withdrawn cost_usd claim into the two checks that settle it, and notes that run used -r 0, so retry spend remains an open question this report does not answer. 3. Add the Opus 5 replication to the criterion-6 analysis. The task was re-run unmodified on claude-opus-5: 2 valid trials, both 0.0, both failing only Criterion 6, criteria 1-5/7/8 passing on each. Combined with the eight Opus 4.8 trials that is 10 valid trials across two model generations with one shared failure. The judge's own explanation is quoted, because it states the mechanism directly -- the agent "asserts those related issues are Done". Also states plainly that the diagnosis is measured but the proposed fix is not: no trial has yet been run against the patched spec, and the analysis makes no claim about its pass rate. Co-Authored-By: Claude <noreply@anthropic.com>
|
@mrojas54 thank you for this. Very good observations. We are reviewing this and will have a full reply later today. |
|
Michelle — this is an excellent piece of work, and thank you for the rigor behind it. In particular, thank you for tracing why it fails, separating the two distinct causes, and backing it with cross-model trials and a trace. That's exactly the kind of contribution that makes a benchmark better. TL;DR: your diagnosis is correct. We reproduced and verified your findings:
What we're going to do — and why it's a little different from the PR as written. We want to keep
In other words: you didn't just find a bug — you surfaced the difference between two levels of the benchmark for this type of query, and that is now the basis for two new tasks. That's a bigger contribution than the original fix. We'll credit you for the finding and for the L2/L3 foundation. I'll follow up directly about collaborating on authoring these. I think you're the right person to introduce the L2 task given you've already done the hard analytical part. One small mechanical note for anyone reading later: the ID stability point in the PR is right — object display IDs are handled ID-independently in grading, and the discrepancy set is defined on stable attributes (subject → counterpart title → status), so it holds at 256x (our primary scale). Thank you again — this is a real improvement to Enterprise-Bench. |
Having your tests match mine exactly is extremely validating — thank you! I like the idea of different levels increasing the cognitive difficulty, and potentially the verifier being refined at that level too. It's a layered complexity approach that I think gets to the heart of agent debugging quickly and efficiently. I'm happy to author the L2 audit task! One ask regardless of what happens to this PR
First, what I'm not arguing: the audit work is harder than the retrieval work, and that's why it deserves its own level. No argument with the L1/L2 split. Two separate questions get blurred here:
The agents did it right and got marked down for it. That isn't a difficulty signal, but the heuristic answers question 2 with question 1's answer, so it reads like one. The tell is the shape of the failure. Three shapes:
The practical fix: read the trajectory, not just the score. All three shapes are identical in Could the three-branch version land separately? Once On this PRFour files, and your plan supersedes three of them:
Unless you'd rather I close it outright, I'll retarget this PR to the diagnosis doc alone. |
3 participants
Scope
This PR now contains only
docs/eng-l1-a-criterion-6.md, the diagnosis of theeng-l1-aCriterion 6 failure.The original instruction, rubric, and reference-trajectory edits were removed per maintainer direction. The structural wording/rubric fix for L1 will land separately; the
exact/approximate/nonecontract is retained here as the foundation for a follow-up L2 audit task, not applied to L1.Diagnosis
Across 10 valid trials on two model generations, every run failed only Criterion 6. Agents converged on the same semantic counterparts, but six were marked
done; three tickets had no same-defect issue in any status. The document separates those spec/data findings from timeout and judgeability accounting and records the reproduction evidence.Verification
make validate: passed (14 tasks and 14 dataset entries validated)docs/eng-l1-a-criterion-6.mdThis write-up is AI-assisted; the runs, diagnosis, and trace verification were performed against the raw Harbor artifacts.