docs(results): point SkillsBench readers at the skillsbench-history branch - #423
Open
bcarmeli wants to merge 1 commit into
Open
docs(results): point SkillsBench readers at the skillsbench-history branch#423bcarmeli wants to merge 1 commit into
bcarmeli wants to merge 1 commit into
Conversation
…ranch The large per-task SkillsBench sweeps, their evolved skill packages and the analysis write-ups live on the skillsbench-history orphan branch, and nothing on main said so — review of skillberry-ai#411 found that as merged those artifacts would have been "a folder nobody will find". Adds a pointer in the intro and a section ahead of the existing SkillsBench sections naming that branch as the source of truth, with a table of what is on it and why it is not on main or on benchmark-history. Also states which SkillsBench number to quote, since the headline is easy to misread: the 87-task sweep's 64/87 = 73.6% comes from a metric that takes the better of val and test, 45 of those 64 passes are val-only, and only 34 of 87 tasks have a test number at all. The test-where-available cut is 62/87 = 71.3% and raw-only pass^5 is 67.1%. This is consistent with the "not directly comparable to EvoSkills' 71.1%" caveat already in the 87-task section below. Signed-off-by: Boaz Carmeli <boazc@il.ibm.com>
Contributor
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the discoverability gap that review of #411 identified.
#411 tried to land the 87-task SkillsBench sweep under
third_party/skillbench/. That PR is now closed:third_party/means vendored upstream code, and those artifacts are SkillsBench output (cap-evolve is merely the tool), not contributions to the library. They now live on theskillsbench-historyorphan branch.But the review's sharpest point was that as merged, those files would have been "a folder nobody will find" — nothing in the tree linked to them. Moving them to a separate branch makes that worse, not better, unless
mainsays where they went. This PR is that pointer.What it adds
Two things, both in
docs/RESULTS.md:skillsbench-historyas the source of truth for SkillsBench output.## SkillsBench sweeps: the full result setsection immediately ahead of the existing SkillsBench sections, with a table of what's on the branch (results/,recipes/,artifacts/,insights/,evidence/) and why it's on neithermainnorbenchmark-history— the latter being machine-written and periodically pruned, so hand-authored content there would eventually be deleted. Notes that SkillsBench dashboard metrics still go through the normalbenchmark-historypipeline.Which number to quote
The section also says plainly which SkillsBench figure is safe to cite, because the headline is easy to misread. The 87-task sweep reports 64/87 = 73.6%, but that metric takes the better of val and test: 45 of those 64 passes are val-only, and only 34 of 87 tasks have a test number at all. The test-where-available cut is 62/87 = 71.3%, and raw-only
pass^5is 67.1%.This is consistent with the caveat already committed a few lines below in the same file — "Not directly comparable to EvoSkills' 71.1%" — and the corresponding
summary.mdonskillsbench-historyhas been corrected to match (it previously claimed a +2.5 pp win, which is 0.52 SE, i.e. noise).Notes
Docs link checkjob (lychee --include-fragments). I verified the heading slug and every relative link and fragment in the file resolves before pushing.