Skip to content

chore: refresh benchmarks for v2.1.0 - #1372

Open
jdx wants to merge 1 commit into
mainfrom
bench-refresh
Open

chore: refresh benchmarks for v2.1.0#1372
jdx wants to merge 1 commit into
mainfrom
bench-refresh

Conversation

@jdx

@jdx jdx commented Aug 25, 2026

Copy link
Copy Markdown
Owner

🤖 Refreshed benchmarks

benchmarks/results.json was pinned to aube 2.0.1; the workspace is now 2.1.0. Re-ran mise run bench:bump on the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regenerated benchmarks/results.json plus the README BENCH_RATIOS block. The benchmark matrix pins aube's GVS mode via npm_config_enable_global_virtual_store=true|false (the auto-synthesized env alias for the enableGlobalVirtualStore setting), so GitHub Actions' inherited CI=true environment does not change whether aube runs with GVS enabled or disabled.

Benchmark changes

Versions:

  • aube: 2.0.1 -> 2.1.0
  • pnpm: 11.22.0 -> 11.23.0

Public ratios: warm installs vs Bun 3x -> 3x; warm installs vs pnpm 7x -> 7x; repeat test vs Bun 1.8x -> 1.8x; repeat test vs pnpm 20x -> 21x.

Benchmark aube bun deno pnpm npm yarn
Fresh install (warm cache) 214ms -> 218ms (+2%) 538ms -> 571ms (+6%) 498ms -> 763ms (+53%) 1479ms -> 1608ms (+9%) 3566ms -> 3666ms (+3%) 2846ms -> 2895ms (+2%)
Fresh install (cold cache) 6684ms -> 7314ms (+9%) 1502ms -> 1675ms (+12%) 1641ms -> 1702ms (+4%) 6497ms -> 9021ms (+39%) 5763ms -> 5786ms (+0%) 6268ms -> 7566ms (+21%)
npm install && npm run test 17ms -> 17ms (0%) 31ms -> 31ms (0%) 85ms -> 190ms (+124%) 341ms -> 351ms (+3%) 645ms -> 663ms (+3%) 1194ms -> 1215ms (+2%)

Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.

Once merged to main, the updated bench results flow into the next release-plz-pr run automatically.


Generated by the bench-refresh workflow.


Note

Low Risk
Docs and benchmark JSON only; no product or runtime behavior changes.

Overview
Re-runs the hermetic benchmark suite and updates pinned results for the 2.1.0 release (pnpm 11.23.0).

benchmarks/results.json gets new timings and stats across warm/cold fresh installs and repeat install && test. Notable shifts vs the prior pin include slower cold installs for pnpm (+39%) and noisier deno numbers on some scenarios; aube warm/cold and repeat-test rows stay in the same ballpark.

The auto-generated README.md BENCH_RATIOS block changes only the marketed repeat-test ratio vs pnpm from 20x to 21x; other headline ratios (7x vs pnpm warm, 3x vs Bun, 1.8x vs Bun on repeat test) are unchanged.

Reviewed by Cursor Bugbot for commit a072a05. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Documentation
    • Updated the reported benchmark result, showing repeat test commands as up to 21× faster than pnpm.
  • Benchmarks
    • Refreshed benchmark metadata, tool versions, measurements, and statistics across package managers.

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 86e47fd8-91f5-4f4f-8f4b-50d38081f244

📥 Commits

Reviewing files that changed from the base of the PR and between acfb152 and a072a05.

📒 Files selected for processing (2)
  • README.md
  • benchmarks/results.json

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

Changes

Benchmark refresh

Layer / File(s) Summary
Refresh benchmark data
benchmarks/results.json
Updated the timestamp, Aube and pnpm versions, and warm-cache, cold-cache, and install-test measurements for all listed package managers.
Align documented speedup
README.md
Updated the repeat-test benchmark claim from up to 20× to up to 21× faster than pnpm.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to a072a

This PR refreshes benchmark data and documentation without changing product or runtime behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

  • jdx/aube#947: Refreshes the same benchmark artifacts and pnpm comparisons.
  • jdx/aube#975: Refreshes benchmark results and performance claims.
  • jdx/aube#1257: Regenerates benchmark results and updates the repeat-test claim.

Suggested reviewers: mise-en-dev

Poem

A rabbit checks the numbers bright
New benchmark values hop in sight
Twenty-one bounces past the row
Fresh versions tell the tale below
The README nods, precise and clear

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: refreshing benchmark data for version 2.1.0.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Refreshes published benchmark data for aube 2.1.0 and pnpm 11.23.0.

  • Updates benchmark timing values and descriptive statistics across warm-install, cold-install, and repeat-test scenarios.
  • Regenerates the README performance ratios, increasing the pnpm repeat-test comparison from 20x to 21x.
  • The refreshed values, statistics, and public ratios are internally consistent.

Confidence Score: 5/5

The PR appears safe to merge because the refreshed benchmark data and generated README claim are internally consistent.

No concrete correctness, security, build, or documentation failure remains in the changed files; the notable measurement variance does not establish that the recorded benchmark means are invalid.

Important Files Changed

Filename Overview
README.md Updates the generated pnpm repeat-test ratio to 21x, consistent with the refreshed benchmark values.
benchmarks/results.json Refreshes tool versions and benchmark measurements; rounded values agree with their means and all public ratios reproduce correctly.

Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile

@github-actions

Copy link
Copy Markdown

Instruction counts

benchmark trend instructions Δ wall (min) Δ
graph ▇▇▁▅▅▁▁▁▅▄▁▂▁▄██▁▂ 13,280,410 → 13,284,934 +0.03% 4.88 → 4.86ms -0.31%
install ▂▁▁▁▂▁▂▁▁▁▁▂▁▁▂███ 110,266,721 → 110,268,539 +0.00% 27.43 → 27.00ms -1.56%
run ▁▅█▅▄▅▅▄█ 3,047,689 → 3,054,393 +0.22% 4.98 → 4.26ms -14.49%
startup ▁▁▁▄▃▃▃▂▃▃▅█▅▆▅▅▅█ 2,378,811 → 2,382,592 +0.16% 2.93 → 2.94ms +0.01%
tree ▇▇▁▁▂▂▁▁▇▁▂▂▂▂▁██▃ 12,679,978 → 12,610,545 -0.55% 4.75 → 4.30ms -9.53%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

a072a05c18d2 vs acfb1520a988 · measured on this runner, not pushed to the history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant