Skip to content

fix: use distinct critcmp group names so PR benchmark diff shows two columns#77

Merged
deepjoy merged 2 commits into
mainfrom
fix-benchmark-comparison-again
Mar 22, 2026
Merged

fix: use distinct critcmp group names so PR benchmark diff shows two columns#77
deepjoy merged 2 commits into
mainfrom
fix-benchmark-comparison-again

Conversation

@deepjoy

@deepjoy deepjoy commented Mar 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • Both the main baseline and PR benchmark results were exported under the same
    critcmp group name ("current"), causing critcmp baseline.json pr.json to
    merge them into a single column instead of producing a side-by-side comparison.
  • Rename the cached baseline group to "main" and the PR export group to "pr"
    so critcmp can diff them as two distinct columns.

deepjoy added 2 commits March 22, 2026 12:39
…columns

Both the main baseline and PR results were exported under the group name
"current", causing critcmp to merge them into a single column instead of
a side-by-side comparison. Rename to "main" and "pr" respectively.
@deepjoy deepjoy enabled auto-merge (squash) March 22, 2026 20:26
@deepjoy deepjoy merged commit ca4b3d5 into main Mar 22, 2026
2 checks passed
@github-actions github-actions Bot mentioned this pull request Mar 22, 2026
@github-actions

github-actions Bot commented Mar 22, 2026

Copy link
Copy Markdown
Contributor

Benchmark Comparison

Click to expand
group                                       current                                 pr
-----                                       -------                                 --
backoff_delay/constant                      1.00     49.8±0.11ns        ? ?/sec     1.01     50.2±0.80ns        ? ?/sec
backoff_delay/exponential                   1.00    187.2±1.33ns        ? ?/sec     1.00    187.3±1.23ns        ? ?/sec
backoff_delay/exponential_jitter            1.00    268.9±1.06ns        ? ?/sec     1.00    269.6±2.67ns        ? ?/sec
backoff_delay/linear                        1.00     76.1±0.99ns        ? ?/sec     1.00     76.3±0.62ns        ? ?/sec
batch_submit_1000                           1.00     32.9±2.47ms        ? ?/sec     1.00     32.9±2.56ms        ? ?/sec
byte_progress/byte_reporting_500            1.00    242.5±5.03ms        ? ?/sec     1.00    242.1±4.99ms        ? ?/sec
byte_progress/noop_500                      1.01    243.1±4.08ms        ? ?/sec     1.00    241.5±4.75ms        ? ?/sec
byte_progress_snapshot_100_tasks            1.01     81.0±3.60ms        ? ?/sec     1.00     80.1±2.07ms        ? ?/sec
concurrency_scaling/1                       1.02    378.2±5.61ms        ? ?/sec     1.00    370.7±5.72ms        ? ?/sec
concurrency_scaling/2                       1.01    312.0±3.67ms        ? ?/sec     1.00    307.7±4.39ms        ? ?/sec
concurrency_scaling/4                       1.02    269.8±5.60ms        ? ?/sec     1.00    264.8±7.24ms        ? ?/sec
concurrency_scaling/8                       1.00    243.5±3.74ms        ? ?/sec     1.00    243.1±5.28ms        ? ?/sec
count_by_tags/100                           1.00    128.7±2.81µs        ? ?/sec     1.00    128.8±2.66µs        ? ?/sec
count_by_tags/1000                          1.00    217.2±2.95µs        ? ?/sec     1.01    219.2±4.58µs        ? ?/sec
count_by_tags/5000                          1.00    598.1±7.69µs        ? ?/sec     1.03   617.9±12.03µs        ? ?/sec
dep_chain_dispatch/10                       1.05     11.2±0.10ms        ? ?/sec     1.00     10.7±0.10ms        ? ?/sec
dep_chain_dispatch/25                       1.04     26.8±0.48ms        ? ?/sec     1.00     25.7±0.41ms        ? ?/sec
dep_chain_dispatch/50                       1.02     53.1±0.94ms        ? ?/sec     1.00     52.2±1.02ms        ? ?/sec
dep_chain_submit/10                         1.15      3.4±0.18ms        ? ?/sec     1.00      3.0±0.09ms        ? ?/sec
dep_chain_submit/200                        1.03     78.1±3.25ms        ? ?/sec     1.00     76.1±3.10ms        ? ?/sec
dep_chain_submit/50                         1.07     17.4±0.81ms        ? ?/sec     1.00     16.4±0.84ms        ? ?/sec
dep_fan_in_dispatch/10                      1.07      7.3±0.11ms        ? ?/sec     1.00      6.9±0.17ms        ? ?/sec
dep_fan_in_dispatch/100                     1.00     52.4±0.78ms        ? ?/sec     1.02     53.2±4.01ms        ? ?/sec
dep_fan_in_dispatch/50                      1.04     27.3±0.41ms        ? ?/sec     1.00     26.3±0.91ms        ? ?/sec
dispatch_and_complete_1000                  1.01    489.6±8.28ms        ? ?/sec     1.00    483.7±5.91ms        ? ?/sec
dispatch_group_scaling/1                    1.00    408.1±5.35ms        ? ?/sec     1.01    411.2±5.07ms        ? ?/sec
dispatch_group_scaling/10                   1.00    416.1±5.94ms        ? ?/sec     1.00    415.9±6.22ms        ? ?/sec
dispatch_group_scaling/100                  1.01    415.2±6.50ms        ? ?/sec     1.00    413.0±5.62ms        ? ?/sec
dispatch_group_scaling/50                   1.00    416.7±6.21ms        ? ?/sec     1.00    414.8±4.94ms        ? ?/sec
dispatch_no_groups_500                      1.01    243.1±4.17ms        ? ?/sec     1.00    240.6±4.12ms        ? ?/sec
dispatch_one_group_500                      1.01    415.1±6.14ms        ? ?/sec     1.00    409.9±5.33ms        ? ?/sec
dispatch_permanent_failure_500              1.01    360.6±7.82ms        ? ?/sec     1.00    355.3±4.01ms        ? ?/sec
history_by_type/100                         1.00   881.7±24.09µs        ? ?/sec     1.01   893.1±23.95µs        ? ?/sec
history_by_type/1000                        1.00   906.5±29.60µs        ? ?/sec     1.01   912.9±29.47µs        ? ?/sec
history_by_type/5000                        1.00   933.2±36.33µs        ? ?/sec     1.01   943.6±20.15µs        ? ?/sec
history_query/100                           1.00   537.1±10.68µs        ? ?/sec     1.00   537.7±10.58µs        ? ?/sec
history_query/1000                          1.00    541.4±7.31µs        ? ?/sec     1.00    541.1±7.75µs        ? ?/sec
history_query/5000                          1.00    551.5±5.09µs        ? ?/sec     1.00    550.6±7.08µs        ? ?/sec
history_stats/100                           1.00    146.5±1.41µs        ? ?/sec     1.00    147.0±1.58µs        ? ?/sec
history_stats/1000                          1.00    353.2±1.89µs        ? ?/sec     1.05    370.1±0.91µs        ? ?/sec
history_stats/5000                          1.00   1270.1±5.24µs        ? ?/sec     1.06  1351.6±13.39µs        ? ?/sec
mixed_priority_dispatch_500                 1.02    270.3±5.47ms        ? ?/sec     1.00    265.3±6.34ms        ? ?/sec
peek_next/100                               1.02    121.1±3.55µs        ? ?/sec     1.00    119.3±3.38µs        ? ?/sec
peek_next/1000                              1.01    124.4±4.89µs        ? ?/sec     1.00    122.6±5.80µs        ? ?/sec
peek_next/5000                              1.03    123.3±3.53µs        ? ?/sec     1.00    119.5±2.74µs        ? ?/sec
query_by_tags/100                           1.00  1249.1±145.71µs        ? ?/sec    1.00  1254.3±154.40µs        ? ?/sec
query_by_tags/1000                          1.00      9.9±0.78ms        ? ?/sec     1.04     10.3±1.07ms        ? ?/sec
query_by_tags/5000                          1.08     60.1±7.96ms        ? ?/sec     1.00     55.8±4.98ms        ? ?/sec
retryable_dead_letter/constant              1.01    147.0±2.79ms        ? ?/sec     1.00    145.1±2.85ms        ? ?/sec
retryable_dead_letter/exponential           1.02    146.9±1.95ms        ? ?/sec     1.00    144.4±2.01ms        ? ?/sec
retryable_dead_letter/exponential_jitter    1.02    146.4±2.00ms        ? ?/sec     1.00    144.1±2.13ms        ? ?/sec
retryable_dead_letter/linear                1.02    146.9±2.31ms        ? ?/sec     1.00    143.9±1.89ms        ? ?/sec
submit_1000_tasks                           1.02    182.7±5.35ms        ? ?/sec     1.00    179.3±4.78ms        ? ?/sec
submit_dedup_hit_1000                       1.01    240.8±6.90ms        ? ?/sec     1.00    237.7±9.13ms        ? ?/sec
submit_with_tags/0                          1.00     90.4±3.59ms        ? ?/sec     1.00     90.5±4.50ms        ? ?/sec
submit_with_tags/10                         1.00   243.2±12.29ms        ? ?/sec     1.00    242.1±9.94ms        ? ?/sec
submit_with_tags/20                         1.00   391.3±16.12ms        ? ?/sec     1.01   396.0±19.83ms        ? ?/sec
submit_with_tags/5                          1.00    166.0±6.73ms        ? ?/sec     1.00    165.9±6.82ms        ? ?/sec
tag_values/100                              1.00    133.3±3.09µs        ? ?/sec     1.01    134.9±6.53µs        ? ?/sec
tag_values/1000                             1.00    194.4±2.98µs        ? ?/sec     1.00    195.2±3.47µs        ? ?/sec
tag_values/5000                             1.00    460.5±4.12µs        ? ?/sec     1.00    460.8±6.65µs        ? ?/sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant