docs: refresh benchmarks to 23 cross-repo tasks and 8 cal.com commits - #172
Merged
Merged
Conversation
Sync benchmarks/crossrepo/ with the current benchmark tree: 12 new tasks
(23 in total) and updated runs for the 11 existing ones, including the
Serena arm. The README tables now report recall@50 as percentages; the
headline chart and averages use the 12 hard tasks, where every arm ran
every task.
Add cal.com commit 79169de8d8 and the new Plumbline prompt-variant runs
to benchmarks/singlerepo/, and rebuild that section's tables over eight
commits at recall@50.
Estimate the Plumbline cost and wall time that failed-first-load and
lifecycle never recorded, from their session transcripts; the figures
are in each run's cost_estimate.json and marked with ≈ in the README.
Replace hard-coded MCP access tokens in run configs with
${PLUMBLINE_MCP_TOKEN}. Session transcripts, repository checkouts and
index dumps are not copied. Two raw CLI captures that are not valid JSON
are listed in .prettierignore so they stay byte-exact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refreshes both benchmark sections of the README from the current benchmark tree.
Cross-repo (
benchmarks/crossrepo/)failed-first-loadandlifecycle, which the runs never recorded, are estimated from their session transcripts (cost_estimate.json, marked ≈).cal.com single-repo (
benchmarks/singlerepo/)79169de8d8and the new Plumbline prompt-variant runs.Housekeeping
${PLUMBLINE_MCP_TOKEN}..prettierignoreso they stay byte-exact.🤖 Generated with Claude Code