PLAN: benchmark — from v0 outline to operational /benchmark:review-acceleration

[mmcky]

Development plan for taking /benchmark:review-acceleration from its v0 outline to an operational skill. This is workstream B of the benchmarking programme (QuantEcon/meta#335); the evaluation system it packages was designed and validated by @xuanguang-li on lecture-python.myst#717.

Body updated 2026-08-03 to the state on main. The original plan, written before #5, is in this issue's edit history; the comments below record how each item actually resolved.

Status note 2026-08-13 (supersedes "Where we are" below where they differ): benchmark is now 0.4.0 — #33 merged as f035a51 and tagged benchmark--v0.4.0. Triage is promoted to the skill's front door and every output leads with the verdict; no scoring semantics moved and both anchors reproduce byte-identically. That fixes #14 finding 6; findings 1–5 there stay open, as does #7. lecture-python.myst#717 was closed 2026-08-07 with a formal no-conversion verdict, so the §4 acceptance test (#10) has no live target until a genuinely new conversion PR appears. The "Pending PR #30" item below merged 2026-08-03. Two small follow-ups from Kenko's #33 review: the skill should state which mode it chose and ask when genuinely ambiguous, and the qualitative triage decision rule gets revisited together with the triage-scaffold call (#14 finding 2 / #7).

Where we are

benchmark is 0.3.1 on main. The evaluation system landed in 0.3.0 (#5, merged 2026-07-27); 0.3.1 (#28) added benchmark/CHANGELOG.md and corrected a wrong version in the plugin README's status line, and changed no procedure.

Landed	Still open
The skill body, operational for workspace runs: scaffold with verbatim extraction → extraction/replay diff against the lecture's cells → measurement → evidence → deterministic score → report, with the plugin read-only at ${CLAUDE_PLUGIN_ROOT} and the evaluation built under <workspace>/benchmark-eval/<lecture>/	Triage mode ships no scaffold — SKILL.md documents it, nothing executable backs it (#14, finding 2). The other four findings from that run are also unfixed on main: no timeout= and still capture_output=True in both run_all.py (1, 5), env_stamp.py unchanged (3), the 1 s materiality floor unchanged (4)
The rubric as benchmark/references/EVALUATION_FRAMEWORK.md, machine-encoded in benchmark/scripts/scoring/ (rubric.py, score.py, env_stamp.py, EVIDENCE_TEMPLATE.json) — no score is typed by hand	The rubric-v2 documentation-honesty pass, and the two items needing @xuanguang-li's design input (#7)
Measurement scripts as per-lecture templates under benchmark/references/examples/<lecture>/scripts/, plus the shared benchmark/scripts/calibration/bellman_bench.py	The acceptance test (#10), both halves — a from-scratch skill evaluation of lecture-python.myst#654 and the expert-parallel adjudication. That PR was closed unmerged on 2026-07-21, so the protocol's final step needs a new target decision
Two complete worked evaluations as regression anchors — ge_arrow 2.85/5, markov_asset 2.25/5, both no-conversion — regenerated from their evidence files by CI on every PR and diffed against the committed scorecards	The three housekeeping remainders from the PR #5 review, all still as filed: no committed environment record for the reference examples (E1), the readability worse-of-two asymmetry (E6), no __init__.py in scripts/scoring/ (E7)
	Distillation into the manual (QuantEcon.manual#104) and the workstream-B tick-off on QuantEcon/meta#335

Work items

1. The rubric — landed as references/EVALUATION_FRAMEWORK.md

• The rubric written down as the canonical standard — seven weighted dimensions, 1–5 anchors, structural checklists, verdict bands. It landed as benchmark/references/EVALUATION_FRAMEWORK.md, executable in benchmark/scripts/scoring/rubric.py, and not as the references/rubric.md this plan named — that path never existed in any branch. Weights and bands live in those two files; this issue deliberately does not restate them.
• Worked HIGH/LOW calibration cases with their measured numbers, in benchmark/references/examples/
• The measurement standard encoded: as-used fresh-process totals as the headline, warm timings never alone, crossover-n, recompile audit

2. Evaluation scripts — landed, as templates rather than a harness

• All eight named scripts exist in the plugin. references/examples/ge_arrow/scripts/ carries all seven per-lecture ones (check_equivalence.py, static_metrics.py, benchmark.py, cold_start.py, sweep_bench.py, as_used_total.py, run_all.py); references/examples/markov_asset/scripts/ carries five of them plus a lecture-specific smoke_test.py, because the set is adapted per lecture rather than shipped fixed.
• Not generalised into a lecture-agnostic CLI — this box will not be ticked as written. SKILL.md step 1 makes adapting the templates the skill's job ("there is deliberately no rigid harness"), because the call sequence that matters is each lecture's own. The need behind the item is met.
• Environment descriptor on every result — scripts/scoring/env_stamp.py writes results/env.json with python, platform and machine plus the versions of numpy, jax, jaxlib and quantecon, including the titles of failed pipeline steps so a partial run cannot claim full provenance. It does not record numba or scipy (TESTING: benchmark triage run on wald_friedman — six plugin gaps, and a floor that measures the wrong quantity #14, finding 3, open).

3. Operational skill body — landed in 0.3.0

• The driving procedure: extract both implementations verbatim, diff extraction and replay against the lecture's cells, adapt and run the measurement templates, fill evidence.json with cited answers, and let the engine compute every score and the verdict
• Report format suitable for posting as a PR review comment — the shape both worked reports follow
• The warm-only guardrail: the efficiency score reads the as-used speedup only, and benchmark/README.md states it as "the one rule to remember"

4. Validation

• lecture-python.myst#717 (ge_arrow) — evaluated, and re-validated end-to-end from a fresh checkout (reviews/validation-run-ge_arrow-2026-07-22.md). Under rubric v2 it scores 2.85, not the ≈2.95 this plan expected from the #717 thread; that number predates v2.
• lecture-python.myst#654 (markov_asset) — the packaged evaluation scores 2.25/5, no-conversion, and is committed as a regression anchor, but the from-scratch skill run of this PR called for by TESTING: review + expert-parallel protocol for the benchmark skill (@xuanguang-li) #10 has not been done, and PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7 records corrections of record to the original report. @jstac closed that PR unmerged on 2026-07-21 having reached the same keep-it-in-NumPy conclusion by his own review. The from-scratch evaluation plus the expert-parallel adjudication (TESTING: review + expert-parallel protocol for the benchmark skill (@xuanguang-li) #10) are the skill's acceptance test and both remain outstanding.
• HIGH anchor sanity-checked — benchmark/scripts/calibration/bellman_bench.py pins the aiyagari Bellman efficiency anchor

First exercise beyond this plan: the wald_friedman triage run (reviews/triage-run-wald_friedman-2026-07-27.md, #15) reached a defensible no-conversion verdict on a lecture with no candidate, and surfaced five plugin gaps (#14), none of them yet fixed.

5. Documentation + rollout — the remaining work

• Distill the rubric into the QuantEcon manual as the companion to the JAX style page — QuantEcon.manual#104, still open
• Tick off the corresponding workstream-B items on PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335
• Version bump: the plugin ships at 0.3.1, and every further change under benchmark/ on a PR into main now requires a version bump plus a changelog heading for that version, enforced in CI by scripts/check-version-bump.py (Release procedure: version bumps deliver, changelogs record, CI enforces #28)
• README status: benchmark/README.md still reads "evaluation system landed (v0.3.0); skill wiring tracked in skills#4", which the wiring landing in 0.3.0 made wrong. Pending PR Status: describe the plugins as they are, in the files that ship them #30 replaces it.

Boundary

/qe:check-jax (skills#3) checks how JAX is written against the style guide; this skill judges whether an accelerated implementation earns its place in a lecture — the boundary QuantEcon.manual#104 draws. The "should this lecture use JAX at all" question belongs here.

[mmcky]

Status update: @xuanguang-li delivered the package (2026-07-21) and it exceeds what this plan expected — not 8 raw scripts but a complete system with the rubric machine-encoded (scoring/rubric.py: scores are deterministic functions of a cited evidence.json, never hand-typed). Landing in #5. Consequences for the work items:

• Item 1 (rubric transcription) — superseded. EVALUATION_FRAMEWORK.md + scripts/scoring/ land as the canonical rubric; it post-dates the #717 thread version (structural 4-item checklists, override caps, a does-not-build gate; ge_arrow re-scored 2.85 vs the thread's 2.95). QuantEcon.manual#104 should distill from these files, not the thread.
• Item 2 (scripts) — landed, design revised. Per-lecture measurement scripts are adapted templates, not a generalised harness — template adaptation is precisely the step the skill automates, and lecture diversity (ge_arrow's λ-sweep vs markov_asset's checkify) genuinely resists a rigid CLI. The provenance stamp (results/env.json) is added in Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5, seeding the PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335 shared schema.
• Item 3 (skill body) — unblocked. SKILL.md's procedure now points at the real engine; remaining work is making the skill drive it end-to-end on a fresh PR.
• Item 4 (validation) — largely delivered. Both named validation cases are complete worked evaluations and now serve as the regression baseline (both scorecards regenerate byte-identically in Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5). markov_asset's findings (2.25/5, build-breaking bug) were reported to [markov_asset] Update to JAX and runtime comparison lecture-python.myst#654.
• Item 5 (manual docs + rollout) — unchanged, now unblocked once Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5 merges.

[mmcky]

Housekeeping remainders from the PR #5 review (response comment has the applied set; methodology items went to #7):

• E1 — results/env.json is gitignored for the reference examples, but evidence records source PRs and SHAs, not library versions, so there is no committed record of what produced the reference measurements. Suggest committing an env.reference.json for the two worked cases.
• E6 — in score_readability, the docstring sub-score bottoms out at 2 while the prereq sub-score reaches 1, so inside the worse-of-two a docstring catastrophe can never be the sole cause of a 1. Minor asymmetry, worth a deliberate yes/no.
• E7 — scripts/scoring/ has no __init__.py; the sys.path.insert works for the CLI but python -m does not. Matters only if the engine gets imported from elsewhere.

[mmcky]

Status after #5 merged — the "Where we are" table above is now stale in the plugin's favour.

Landed: the plugin is at 0.3.0 with the rubric in benchmark/references/EVALUATION_FRAMEWORK.md, the deterministic scoring engine in benchmark/scripts/scoring/ (rubric.py, score.py, env_stamp.py), rubric v2 in full (#7), two complete worked evaluations as regression baselines, synthetic fixtures pinning the v2 paths, and CI that reproduces every scorecard from its evidence file. The skill body is operational for workspace runs with ${CLAUDE_PLUGIN_ROOT} anchoring. The plugin guide and the evaluation framework are now published at https://quantecon.github.io/skills/.

Work item 2 resolved differently from the plan, which is worth recording. This issue anticipated landing eight generalised evaluation scripts as a fixed harness. What shipped instead is measurement templates inside the worked examples, adapted per lecture by the skill — SKILL.md states that adapting them is the skill's job and that there is deliberately no rigid harness, because the call sequence that matters is the lecture's own. So the checkbox for "receive and commit the 8 scripts / generalise from ge_arrow-specific code" will not be ticked as written; the need behind it is met. The environment descriptor from that item did land, as env_stamp.py.

Genuinely still open: the full v2 evaluation of lecture-python.myst#654 and its adjudication under the expert-parallel protocol (#10), which together are the skill's acceptance test. That PR thread still has its clean slate.