bench-history: add 2 local parsec records (v1 pilot, v2 honest) - #466
Merged
OsherElhadad merged 2 commits intoSep 8, 2026
Merged
Conversation
v1 (run 20260908001, experiment=v1, tier=pilot): the 2-task pilot from parsec-history/results/v1 — reward_base 0.317 -> reward_opt 0.326, train-fit only (train==val==test, no holdout), best=seed (both candidates rejected by the gate). n=2 of the source experiment's 30 tasks; the other 28 are baseline-only. v2 (run 20260908002, experiment=v2, tier=v2): the 10-task run from parsec-history/results/v2 — reward_base 0.822 -> reward_opt 0.962 (delta +0.139), the honest equal-n framing (seed re-measured at n=9, errored trials dropped) rather than cap-evolve's own gate output (+0.086, a mixed-n comparison against a reused n=3 baseline). No held-out test exists yet, so this is val-only, not comparable to v1. Both summary_url point at parsec-history's per-experiment summary.md, so the Source column links straight to the full narrative (task tables, cost recovery, caveats) rather than a bare "bcarmeli". Built with the current record.py (n_scored included, opt_infra-aware rollup — neither record has any opt_infra:true task so the reward figures are unaffected). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: boazc <boazc@il.ibm.com>
Both notes duplicated content already written out in full (with more precision) in results/v1/summary.md and results/v2/summary.md on parsec-history, which summary_url already links to. v2's note in particular was 1734 chars, by far the longest of any note across all 81 records (next is 1157; median ~400) — and site/benchmarks.js never reads or renders `note` anywhere, so the length bought nothing in the live UI. Trimmed both to the one caveat that matters for reading the number correctly (fit-only, not an accepted improvement / honest equal-n framing vs. the gate's raw output), with a pointer to the summary for everything else. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: boazc <boazc@il.ibm.com>
bcarmeli
force-pushed
the
bench-history/add-parsec-v1-v2-bcarmeli
branch
from
September 8, 2026 13:39
3a732fc to
98c83a4
Compare
OsherElhadad
approved these changes
Sep 8, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds parsec to the live dashboard, as two distinct Experiment rows (
v1,v2) — parsec has no CI job (see PR #465,parsec/benchmark-local-only, still open), so these are hand-built local-run records rather than CI-published ones, same mechanism as the existingbcarmelilocal skillsbench baseline/opus-optimize records already on this branch.What's in each record
v1 (
run_id=20260908001,bench=parsec,tier=pilot,experiment=v1)parsec-history/results/v1.reward_base 0.317 → reward_opt 0.326. Train-fit only —run_suite.shpins train==val==test to the same 2 tasks, so this is not a generalization claim.best=seed: both candidates (cand_0001mean 0.225,cand_0002mean 0.157) scored below the seed and were rejected by the paired-SE gate. The delta shown is two separate n=30 measurements of the byte-identical seed (baseline vs. finalize pass) — seed-measurement noise, not an improvement.v2 (
run_id=20260908002,bench=parsec,tier=v2,experiment=v2)parsec-history/results/v2.reward_base 0.822 → reward_opt 0.962(delta +0.139) — the honest equal-n framing (seed re-measured at n=9, errored trials dropped), not cap-evolve's own gate output (+0.086), which compared the n=9 candidate against a reused n=3 baseline measurement — a mixed-n comparison biased toward the newer number.parsec-history/artifacts/README.md).Both records carry
summary_urlpointing at the correspondingparsec-historysummary.md, so the Source column links straight to the full per-task table / cost breakdown / caveats instead of a barebcarmeli. Neither record setspr— parsec isn't CI-dispatchable (see PR #465's README changes), consistent with the other local-source records already on this branch.Schema note
Built with the current
record.py(origin/main, not the stale copy in theparsec-intake_v1worktree I originally built these against) — both records now carrysuite.n_scored, matching the new coverage-badge UI. Neither task set has anyopt_infra:truerow, so the reward figures themselves are unaffected by the fix; only the missingn_scoredfield needed correcting.Verification
records/*.jsonfor both runs load and theirsuiterollup matches a hand-computed mean of the per-task rows (checked both).origin/benchmark-history(671a39ce, includes bench: record transfer-eval-8fold skill-transfer pilot (run 20260902001) #459's transfer-eval-8fold record) before branching, so this isn't based on a stale snapshot.benchmarks.json/meta.jsonregenerated viarecord.py aggregateagainst the fullrecords/dir: 81 records, 75 runs (79 pre-existing + these 2).site/benchmarks.js(PR site(benchmarks): derive Benchmark/Type/Experiment filters, add Experiment column #457's Experiment column/filter): both records carryexperimentso they'll show up under av1/v2filter value oncebench=parsecis selected, no site code change needed.🤖 Generated with Claude Code