Skip to content

bench-history: add 2 local parsec records (v1 pilot, v2 honest) - #466

Merged
OsherElhadad merged 2 commits into
benchmark-historyfrom
bench-history/add-parsec-v1-v2-bcarmeli
Sep 8, 2026
Merged

bench-history: add 2 local parsec records (v1 pilot, v2 honest)#466
OsherElhadad merged 2 commits into
benchmark-historyfrom
bench-history/add-parsec-v1-v2-bcarmeli

Conversation

@bcarmeli

@bcarmeli bcarmeli commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Adds parsec to the live dashboard, as two distinct Experiment rows (v1, v2) — parsec has no CI job (see PR #465, parsec/benchmark-local-only, still open), so these are hand-built local-run records rather than CI-published ones, same mechanism as the existing bcarmeli local skillsbench baseline/opus-optimize records already on this branch.

What's in each record

v1 (run_id=20260908001, bench=parsec, tier=pilot, experiment=v1)

  • The 2-task pilot from parsec-history/results/v1.
  • reward_base 0.317 → reward_opt 0.326. Train-fit only — run_suite.sh pins train==val==test to the same 2 tasks, so this is not a generalization claim.
  • best=seed: both candidates (cand_0001 mean 0.225, cand_0002 mean 0.157) scored below the seed and were rejected by the paired-SE gate. The delta shown is two separate n=30 measurements of the byte-identical seed (baseline vs. finalize pass) — seed-measurement noise, not an improvement.
  • n=2 of the source experiment's 30 tasks; the other 28 are baseline-only (2 of 4 simulators were down during that sweep for most of them).

v2 (run_id=20260908002, bench=parsec, tier=v2, experiment=v2)

  • The 10-task run from parsec-history/results/v2.
  • reward_base 0.822 → reward_opt 0.962 (delta +0.139) — the honest equal-n framing (seed re-measured at n=9, errored trials dropped), not cap-evolve's own gate output (+0.086), which compared the n=9 candidate against a reused n=3 baseline measurement — a mixed-n comparison biased toward the newer number.
  • No held-out test split exists yet for v2, so this is val-only, not a generalization claim, and not comparable to v1 (different tasks, different simulators, different scoring).
  • The accepted diff is a tool-selection documentation fix, not a reasoning-strategy change (see parsec-history/artifacts/README.md).

Both records carry summary_url pointing at the corresponding parsec-history summary.md, so the Source column links straight to the full per-task table / cost breakdown / caveats instead of a bare bcarmeli. Neither record sets pr — parsec isn't CI-dispatchable (see PR #465's README changes), consistent with the other local-source records already on this branch.

Schema note

Built with the current record.py (origin/main, not the stale copy in the parsec-intake_v1 worktree I originally built these against) — both records now carry suite.n_scored, matching the new coverage-badge UI. Neither task set has any opt_infra:true row, so the reward figures themselves are unaffected by the fix; only the missing n_scored field needed correcting.

Verification

🤖 Generated with Claude Code

bcarmeli and others added 2 commits September 8, 2026 16:37
v1 (run 20260908001, experiment=v1, tier=pilot): the 2-task pilot from
parsec-history/results/v1 — reward_base 0.317 -> reward_opt 0.326, train-fit
only (train==val==test, no holdout), best=seed (both candidates rejected by
the gate). n=2 of the source experiment's 30 tasks; the other 28 are
baseline-only.

v2 (run 20260908002, experiment=v2, tier=v2): the 10-task run from
parsec-history/results/v2 — reward_base 0.822 -> reward_opt 0.962 (delta
+0.139), the honest equal-n framing (seed re-measured at n=9, errored trials
dropped) rather than cap-evolve's own gate output (+0.086, a mixed-n
comparison against a reused n=3 baseline). No held-out test exists yet, so
this is val-only, not comparable to v1.

Both summary_url point at parsec-history's per-experiment summary.md, so the
Source column links straight to the full narrative (task tables, cost
recovery, caveats) rather than a bare "bcarmeli". Built with the current
record.py (n_scored included, opt_infra-aware rollup — neither record has any
opt_infra:true task so the reward figures are unaffected).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: boazc <boazc@il.ibm.com>
Both notes duplicated content already written out in full (with more
precision) in results/v1/summary.md and results/v2/summary.md on
parsec-history, which summary_url already links to. v2's note in particular
was 1734 chars, by far the longest of any note across all 81 records (next is
1157; median ~400) — and site/benchmarks.js never reads or renders `note`
anywhere, so the length bought nothing in the live UI. Trimmed both to the one
caveat that matters for reading the number correctly (fit-only, not an
accepted improvement / honest equal-n framing vs. the gate's raw output),
with a pointer to the summary for everything else.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: boazc <boazc@il.ibm.com>
@bcarmeli
bcarmeli force-pushed the bench-history/add-parsec-v1-v2-bcarmeli branch from 3a732fc to 98c83a4 Compare September 8, 2026 13:39
@OsherElhadad
OsherElhadad merged commit fd80eec into benchmark-history Sep 8, 2026
2 checks passed
@OsherElhadad
OsherElhadad deleted the bench-history/add-parsec-v1-v2-bcarmeli branch September 8, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants