evict broadcast entries after all consumers read by gene-bordegaray · Pull Request #726 · datafusion-contrib/datafusion-distributed

gene-bordegaray · 2026-09-14T00:44:46Z

broadcast wasn't dropping after all consumer read from the vec. now after the last consumer reads it will drop the entry. should help with memroy pressure

here are the memory changes from the broadcast micro bench:

Scenario	main peak reserved	Branch peak reserved	Reduction
`all_fast`	750.7 MiB	9.1 MiB	98.8%
`one_slow`	754.0 MiB	23.4 MiB	96.9%
`one_cancel`	747.8 MiB	9.1 MiB	98.8%
`unused_partition`	758.3 MiB	55.3 MiB	92.7%
`many_consumers`	12.4 MiB	0.2 MiB	98.8%

note the unused task is forced to retain the ful queue because we do not know whether its consumer will need to replay the record batches

gene-bordegaray · 2026-09-14T01:04:25Z

benchmarks run tpch/sf10

gabot-0 · 2026-09-14T01:04:30Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head ba1f21bcca51 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=859.6, new=860.0, diff=0.4 more (0.0%) (sum of per-query averages)
TOTAL: prev=15715 ms, new=15738 ms, diff=1.00 slower ✖

Show full query output

      q1: prev= 387 ms, new= 490 ms, diff=1.27 slower ✖, tasks: prev=23.6, new=24.0, diff=0.4 more (1.7%)
      q2: prev= 733 ms, new= 729 ms, diff=1.01 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
      q3: prev= 793 ms, new= 650 ms, diff=1.22 faster ✔, tasks: prev=33.0, new=33.0, diff=no change
      q4: prev= 384 ms, new= 348 ms, diff=1.10 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
      q5: prev= 886 ms, new= 837 ms, diff=1.06 faster ✔, tasks: prev=60.0, new=60.0, diff=no change
      q6: prev= 324 ms, new= 334 ms, diff=1.03 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev= 987 ms, new= 982 ms, diff=1.01 faster ✔, tasks: prev=68.0, new=68.0, diff=no change
      q8: prev=1208 ms, new=1159 ms, diff=1.04 faster ✔, tasks: prev=82.0, new=82.0, diff=no change
      q9: prev=1467 ms, new=1317 ms, diff=1.11 faster ✔, tasks: prev=82.0, new=82.0, diff=no change
     q10: prev=1136 ms, new=1346 ms, diff=1.18 slower ✖, tasks: prev=46.0, new=46.0, diff=no change
     q11: prev= 499 ms, new= 574 ms, diff=1.15 slower ✖, tasks: prev=20.0, new=20.0, diff=no change
     q12: prev= 435 ms, new= 441 ms, diff=1.01 slower ✖, tasks: prev=40.0, new=40.0, diff=no change
     q13: prev= 567 ms, new= 578 ms, diff=1.02 slower ✖, tasks: prev=13.0, new=13.0, diff=no change
     q14: prev= 558 ms, new= 525 ms, diff=1.06 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
     q15: prev= 601 ms, new= 597 ms, diff=1.01 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 372 ms, new= 378 ms, diff=1.02 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
     q17: prev= 705 ms, new= 853 ms, diff=1.21 slower ✖, tasks: prev=37.0, new=37.0, diff=no change
     q18: prev=1104 ms, new=1126 ms, diff=1.02 slower ✖, tasks: prev=45.0, new=45.0, diff=no change
     q19: prev= 434 ms, new= 421 ms, diff=1.03 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
     q20: prev= 679 ms, new= 701 ms, diff=1.03 slower ✖, tasks: prev=44.0, new=44.0, diff=no change
     q21: prev=1124 ms, new=1043 ms, diff=1.08 faster ✔, tasks: prev=66.0, new=66.0, diff=no change
     q22: prev= 332 ms, new= 309 ms, diff=1.07 faster ✔, tasks: prev=9.0, new=9.0, diff=no change

Verification and run details

Job 111 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`ba1f21bcca51e28a05aeeb344b35818e4740bf9c`

Phase	Base	PR head
Build and deployment	1m 40s	1m 27s
All benchmarks	1m 44s	1m 42s
Benchmark `tpch/sf10`	1m 44s	1m 42s

Workload: tpch/sf10 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.2xlarge nodes for both revisions

Other timings: Queue 0s · Dataset validation 0s · Total 6m 40s

gene-bordegaray · 2026-09-14T01:04:57Z

hello gabot

for reviewers: not expecting wall tie speedups just checking no wall time regression for some weird reason. main thing I am hopring to imporve is memory use. I will post RSS sampels from the microbench

gene-bordegaray · 2026-09-14T01:09:19Z

benchmarks run tpcds/sf1

gabot-0 · 2026-09-14T01:11:42Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head ba1f21bcca51 · View exact source diff

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1487.0, new=1487.0, diff=no change (sum of per-query averages)
TOTAL: prev=63205 ms, new=65077 ms, diff=1.03 slower ✖

Show full query output

      q1: prev= 310 ms, new= 315 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 697 ms, new= 831 ms, diff=1.19 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 212 ms, new= 350 ms, diff=1.65 slower ❌, tasks: prev=8.0, new=8.0, diff=no change
      q4: prev=3271 ms, new=3219 ms, diff=1.02 faster ✔, tasks: prev=32.0, new=32.0, diff=no change
      q5: prev= 505 ms, new= 507 ms, diff=1.00 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
      q6: prev=1324 ms, new=1301 ms, diff=1.02 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev= 345 ms, new= 346 ms, diff=1.00 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
      q8: prev= 284 ms, new= 277 ms, diff=1.03 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q9: prev= 275 ms, new= 388 ms, diff=1.41 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
     q10: prev= 645 ms, new= 727 ms, diff=1.13 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q11: prev=2367 ms, new=2163 ms, diff=1.09 faster ✔, tasks: prev=18.0, new=18.0, diff=no change
     q12: prev= 136 ms, new= 133 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 937 ms, new= 858 ms, diff=1.09 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q14: prev=1261 ms, new=1259 ms, diff=1.00 faster ✔, tasks: prev=82.0, new=82.0, diff=no change
     q15: prev= 295 ms, new= 277 ms, diff=1.06 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q16: prev= 375 ms, new= 364 ms, diff=1.03 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q17: prev= 408 ms, new= 383 ms, diff=1.07 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q18: prev= 770 ms, new= 730 ms, diff=1.05 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q19: prev= 544 ms, new= 530 ms, diff=1.03 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q20: prev= 215 ms, new= 189 ms, diff=1.14 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q21: prev= 147 ms, new= 150 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 580 ms, new= 596 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev=1060 ms, new=1078 ms, diff=1.02 slower ✖, tasks: prev=56.0, new=56.0, diff=no change
     q24: prev=1094 ms, new=1258 ms, diff=1.15 slower ✖, tasks: prev=31.0, new=31.0, diff=no change
     q25: prev= 438 ms, new= 582 ms, diff=1.33 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q26: prev= 294 ms, new= 305 ms, diff=1.04 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q27: prev= 454 ms, new= 457 ms, diff=1.01 slower ✖, tasks: prev=29.0, new=29.0, diff=no change
     q28: prev= 216 ms, new= 221 ms, diff=1.02 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
     q29: prev= 451 ms, new= 483 ms, diff=1.07 slower ✖, tasks: prev=17.0, new=17.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev=1007 ms, new=1054 ms, diff=1.05 slower ✖, tasks: prev=31.0, new=31.0, diff=no change
     q32: prev= 283 ms, new= 303 ms, diff=1.07 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q33: prev= 319 ms, new= 350 ms, diff=1.10 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q34: prev= 396 ms, new= 322 ms, diff=1.23 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q35: prev= 701 ms, new= 751 ms, diff=1.07 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q36: prev= 408 ms, new= 351 ms, diff=1.16 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q37: prev= 336 ms, new= 355 ms, diff=1.06 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q38: prev= 526 ms, new= 531 ms, diff=1.01 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q39: prev= 280 ms, new= 297 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 416 ms, new= 424 ms, diff=1.02 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
     q41: prev=  72 ms, new=  91 ms, diff=1.26 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 192 ms, new= 188 ms, diff=1.02 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q43: prev= 203 ms, new= 208 ms, diff=1.02 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q44: prev= 340 ms, new= 351 ms, diff=1.03 slower ✖, tasks: prev=18.0, new=18.0, diff=no change
     q45: prev= 264 ms, new= 289 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 574 ms, new= 567 ms, diff=1.01 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
     q47: prev= 929 ms, new= 999 ms, diff=1.08 slower ✖, tasks: prev=29.0, new=29.0, diff=no change
     q48: prev= 553 ms, new= 584 ms, diff=1.06 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q49: prev= 586 ms, new= 469 ms, diff=1.25 faster ✔, tasks: prev=20.0, new=20.0, diff=no change
     q50: prev= 521 ms, new= 518 ms, diff=1.01 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
     q51: prev= 495 ms, new= 479 ms, diff=1.03 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q52: prev= 206 ms, new= 210 ms, diff=1.02 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q53: prev= 219 ms, new= 228 ms, diff=1.04 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q54: prev= 609 ms, new= 561 ms, diff=1.09 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q55: prev= 159 ms, new= 223 ms, diff=1.40 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q56: prev= 325 ms, new= 364 ms, diff=1.12 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q57: prev= 701 ms, new= 716 ms, diff=1.02 slower ✖, tasks: prev=29.0, new=29.0, diff=no change
     q58: prev= 895 ms, new= 976 ms, diff=1.09 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q59: prev= 448 ms, new= 450 ms, diff=1.00 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q60: prev= 325 ms, new= 341 ms, diff=1.05 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q61: prev=1420 ms, new=1467 ms, diff=1.03 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
     q62: prev=1390 ms, new=1415 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 211 ms, new= 250 ms, diff=1.18 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q64: prev=2378 ms, new=3169 ms, diff=1.33 slower ✖, tasks: prev=80.0, new=80.0, diff=no change
     q65: prev= 530 ms, new= 502 ms, diff=1.06 faster ✔, tasks: prev=17.0, new=17.0, diff=no change
     q66: prev= 701 ms, new= 810 ms, diff=1.16 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q67: prev= 580 ms, new= 582 ms, diff=1.00 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q68: prev= 542 ms, new= 531 ms, diff=1.02 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
     q69: prev= 636 ms, new= 737 ms, diff=1.16 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q70: prev= 670 ms, new= 671 ms, diff=1.00 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q71: prev= 380 ms, new= 321 ms, diff=1.18 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q72: prev=7950 ms, new=8145 ms, diff=1.02 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
     q73: prev= 328 ms, new= 453 ms, diff=1.38 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q74: prev=1012 ms, new=1072 ms, diff=1.06 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q75: prev= 922 ms, new=1026 ms, diff=1.11 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q76: prev= 435 ms, new= 410 ms, diff=1.06 faster ✔, tasks: prev=18.0, new=18.0, diff=no change
     q77: prev= 599 ms, new= 634 ms, diff=1.06 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q78: prev= 766 ms, new= 801 ms, diff=1.05 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q79: prev= 354 ms, new= 348 ms, diff=1.02 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q80: prev= 831 ms, new= 712 ms, diff=1.17 faster ✔, tasks: prev=28.0, new=28.0, diff=no change
     q81: prev= 234 ms, new= 245 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 334 ms, new= 343 ms, diff=1.03 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q83: prev= 311 ms, new= 303 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 282 ms, new= 266 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 506 ms, new= 492 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 145 ms, new= 140 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 550 ms, new= 607 ms, diff=1.10 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q88: prev= 493 ms, new= 483 ms, diff=1.02 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q89: prev= 310 ms, new= 288 ms, diff=1.08 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q90: prev= 159 ms, new= 161 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 395 ms, new= 390 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 229 ms, new= 246 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 272 ms, new= 293 ms, diff=1.08 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q94: prev= 293 ms, new= 328 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 595 ms, new= 450 ms, diff=1.32 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 366 ms, new= 333 ms, diff=1.10 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q97: prev= 392 ms, new= 348 ms, diff=1.13 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q98: prev= 258 ms, new= 235 ms, diff=1.10 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q99: prev=1248 ms, new=1243 ms, diff=1.00 faster ✔, tasks: prev=11.0, new=11.0, diff=no change

Verification and run details

Job 112 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`ba1f21bcca51e28a05aeeb344b35818e4740bf9c`

Phase	Base	PR head
Build and deployment	1m 26s	1m 27s
All benchmarks	6m 40s	6m 49s
Benchmark `tpcds/sf1`	6m 40s	6m 49s

Workload: tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.2xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 0s · Total 16m 29s

gene-bordegaray · 2026-09-14T01:32:48Z

q3 does have a broadcast, but my microbenches are showing same latency perf 🤔

gene-bordegaray · 2026-09-14T01:51:47Z

benchmarks run tpch/sf10 tpcds/sf1 tpch/sf100

gabot-0 · 2026-09-14T01:51:53Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head ba1f21bcca51 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=859.2, new=860.0, diff=0.8 more (0.1%) (sum of per-query averages)
TOTAL: prev=14939 ms, new=15494 ms, diff=1.04 slower ✖

Show full query output

      q1: prev= 512 ms, new= 364 ms, diff=1.41 faster ✔, tasks: prev=23.2, new=24.0, diff=0.8 more (3.4%)
      q2: prev= 664 ms, new= 748 ms, diff=1.13 slower ✖, tasks: prev=25.0, new=25.0, diff=no change
      q3: prev= 685 ms, new= 741 ms, diff=1.08 slower ✖, tasks: prev=33.0, new=33.0, diff=no change
      q4: prev= 348 ms, new= 317 ms, diff=1.10 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
      q5: prev= 859 ms, new= 851 ms, diff=1.01 faster ✔, tasks: prev=60.0, new=60.0, diff=no change
      q6: prev= 224 ms, new= 247 ms, diff=1.10 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev= 970 ms, new= 933 ms, diff=1.04 faster ✔, tasks: prev=68.0, new=68.0, diff=no change
      q8: prev=1137 ms, new=1312 ms, diff=1.15 slower ✖, tasks: prev=82.0, new=82.0, diff=no change
      q9: prev=1291 ms, new=1362 ms, diff=1.05 slower ✖, tasks: prev=82.0, new=82.0, diff=no change
     q10: prev=1129 ms, new=1189 ms, diff=1.05 slower ✖, tasks: prev=46.0, new=46.0, diff=no change
     q11: prev= 488 ms, new= 514 ms, diff=1.05 slower ✖, tasks: prev=20.0, new=20.0, diff=no change
     q12: prev= 399 ms, new= 410 ms, diff=1.03 slower ✖, tasks: prev=40.0, new=40.0, diff=no change
     q13: prev= 578 ms, new= 554 ms, diff=1.04 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
     q14: prev= 414 ms, new= 472 ms, diff=1.14 slower ✖, tasks: prev=25.0, new=25.0, diff=no change
     q15: prev= 518 ms, new= 602 ms, diff=1.16 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 378 ms, new= 399 ms, diff=1.06 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
     q17: prev= 732 ms, new= 741 ms, diff=1.01 slower ✖, tasks: prev=37.0, new=37.0, diff=no change
     q18: prev=1126 ms, new=1088 ms, diff=1.03 faster ✔, tasks: prev=45.0, new=45.0, diff=no change
     q19: prev= 428 ms, new= 494 ms, diff=1.15 slower ✖, tasks: prev=25.0, new=25.0, diff=no change
     q20: prev= 704 ms, new= 770 ms, diff=1.09 slower ✖, tasks: prev=44.0, new=44.0, diff=no change
     q21: prev=1041 ms, new=1089 ms, diff=1.05 slower ✖, tasks: prev=66.0, new=66.0, diff=no change
     q22: prev= 314 ms, new= 297 ms, diff=1.06 faster ✔, tasks: prev=9.0, new=9.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1487.0, new=1487.0, diff=no change (sum of per-query averages)
TOTAL: prev=63009 ms, new=64216 ms, diff=1.02 slower ✖

Show full query output

      q1: prev= 293 ms, new= 289 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 776 ms, new= 732 ms, diff=1.06 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 360 ms, new= 332 ms, diff=1.08 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
      q4: prev=3009 ms, new=3257 ms, diff=1.08 slower ✖, tasks: prev=32.0, new=32.0, diff=no change
      q5: prev= 546 ms, new= 541 ms, diff=1.01 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
      q6: prev=1300 ms, new=1300 ms, diff=1.00 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev= 344 ms, new= 390 ms, diff=1.13 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
      q8: prev= 279 ms, new= 305 ms, diff=1.09 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q9: prev= 427 ms, new= 402 ms, diff=1.06 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q10: prev= 677 ms, new= 721 ms, diff=1.06 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q11: prev=2182 ms, new=2389 ms, diff=1.09 slower ✖, tasks: prev=18.0, new=18.0, diff=no change
     q12: prev= 169 ms, new= 139 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 959 ms, new= 872 ms, diff=1.10 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q14: prev=1299 ms, new=1250 ms, diff=1.04 faster ✔, tasks: prev=82.0, new=82.0, diff=no change
     q15: prev= 279 ms, new= 271 ms, diff=1.03 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q16: prev= 366 ms, new= 351 ms, diff=1.04 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q17: prev= 418 ms, new= 387 ms, diff=1.08 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q18: prev= 683 ms, new= 664 ms, diff=1.03 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q19: prev= 470 ms, new= 483 ms, diff=1.03 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q20: prev= 222 ms, new= 193 ms, diff=1.15 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q21: prev= 167 ms, new= 148 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 570 ms, new= 552 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev=1046 ms, new=1060 ms, diff=1.01 slower ✖, tasks: prev=56.0, new=56.0, diff=no change
     q24: prev=1126 ms, new=1044 ms, diff=1.08 faster ✔, tasks: prev=31.0, new=31.0, diff=no change
     q25: prev= 434 ms, new= 461 ms, diff=1.06 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q26: prev= 282 ms, new= 290 ms, diff=1.03 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q27: prev= 482 ms, new= 626 ms, diff=1.30 slower ✖, tasks: prev=29.0, new=29.0, diff=no change
     q28: prev= 232 ms, new= 250 ms, diff=1.08 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
     q29: prev= 453 ms, new= 515 ms, diff=1.14 slower ✖, tasks: prev=17.0, new=17.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 932 ms, new=1003 ms, diff=1.08 slower ✖, tasks: prev=31.0, new=31.0, diff=no change
     q32: prev= 286 ms, new= 309 ms, diff=1.08 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q33: prev= 317 ms, new= 320 ms, diff=1.01 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q34: prev= 336 ms, new= 319 ms, diff=1.05 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q35: prev= 673 ms, new= 676 ms, diff=1.00 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q36: prev= 342 ms, new= 424 ms, diff=1.24 slower ✖, tasks: prev=38.0, new=38.0, diff=no change
     q37: prev= 306 ms, new= 320 ms, diff=1.05 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q38: prev= 500 ms, new= 488 ms, diff=1.02 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
     q39: prev= 278 ms, new= 294 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 407 ms, new= 428 ms, diff=1.05 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
     q41: prev=  73 ms, new=  76 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 186 ms, new= 186 ms, diff=1.00 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q43: prev= 194 ms, new= 198 ms, diff=1.02 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q44: prev= 347 ms, new= 358 ms, diff=1.03 slower ✖, tasks: prev=18.0, new=18.0, diff=no change
     q45: prev= 260 ms, new= 279 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 544 ms, new= 640 ms, diff=1.18 slower ✖, tasks: prev=13.0, new=13.0, diff=no change
     q47: prev=1052 ms, new= 923 ms, diff=1.14 faster ✔, tasks: prev=29.0, new=29.0, diff=no change
     q48: prev= 558 ms, new= 581 ms, diff=1.04 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q49: prev= 492 ms, new= 414 ms, diff=1.19 faster ✔, tasks: prev=20.0, new=20.0, diff=no change
     q50: prev= 532 ms, new= 518 ms, diff=1.03 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
     q51: prev= 491 ms, new= 464 ms, diff=1.06 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q52: prev= 181 ms, new= 207 ms, diff=1.14 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q53: prev= 211 ms, new= 233 ms, diff=1.10 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q54: prev= 610 ms, new= 686 ms, diff=1.12 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q55: prev= 202 ms, new= 184 ms, diff=1.10 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q56: prev= 326 ms, new= 384 ms, diff=1.18 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q57: prev= 714 ms, new= 748 ms, diff=1.05 slower ✖, tasks: prev=29.0, new=29.0, diff=no change
     q58: prev= 860 ms, new= 823 ms, diff=1.04 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
     q59: prev= 417 ms, new= 441 ms, diff=1.06 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q60: prev= 330 ms, new= 361 ms, diff=1.09 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q61: prev=1468 ms, new=1474 ms, diff=1.00 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
     q62: prev=1384 ms, new=1409 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 232 ms, new= 229 ms, diff=1.01 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q64: prev=3067 ms, new=3321 ms, diff=1.08 slower ✖, tasks: prev=80.0, new=80.0, diff=no change
     q65: prev= 483 ms, new= 534 ms, diff=1.11 slower ✖, tasks: prev=17.0, new=17.0, diff=no change
     q66: prev= 677 ms, new= 698 ms, diff=1.03 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q67: prev= 567 ms, new= 545 ms, diff=1.04 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q68: prev= 527 ms, new= 525 ms, diff=1.00 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
     q69: prev= 622 ms, new= 632 ms, diff=1.02 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q70: prev= 689 ms, new= 641 ms, diff=1.07 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q71: prev= 300 ms, new= 282 ms, diff=1.06 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
     q72: prev=8029 ms, new=8065 ms, diff=1.00 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
     q73: prev= 323 ms, new= 334 ms, diff=1.03 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q74: prev=1006 ms, new=1058 ms, diff=1.05 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q75: prev= 988 ms, new=1022 ms, diff=1.03 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q76: prev= 447 ms, new= 406 ms, diff=1.10 faster ✔, tasks: prev=18.0, new=18.0, diff=no change
     q77: prev= 634 ms, new= 710 ms, diff=1.12 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q78: prev= 771 ms, new= 776 ms, diff=1.01 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q79: prev= 361 ms, new= 361 ms, diff=1.00 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q80: prev= 862 ms, new= 764 ms, diff=1.13 faster ✔, tasks: prev=28.0, new=28.0, diff=no change
     q81: prev= 243 ms, new= 242 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 325 ms, new= 324 ms, diff=1.00 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q83: prev= 292 ms, new= 295 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 266 ms, new= 273 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 475 ms, new= 468 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 130 ms, new= 137 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 438 ms, new= 514 ms, diff=1.17 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
     q88: prev= 504 ms, new= 485 ms, diff=1.04 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q89: prev= 248 ms, new= 289 ms, diff=1.17 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q90: prev= 159 ms, new= 167 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 382 ms, new= 381 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 234 ms, new= 244 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 279 ms, new= 259 ms, diff=1.08 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q94: prev= 270 ms, new= 289 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 405 ms, new= 402 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 300 ms, new= 284 ms, diff=1.06 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q97: prev= 337 ms, new= 340 ms, diff=1.01 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q98: prev= 206 ms, new= 229 ms, diff=1.11 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q99: prev=1272 ms, new=1309 ms, diff=1.03 slower ✖, tasks: prev=11.0, new=11.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1286.0, new=1286.0, diff=no change (sum of per-query averages)
TOTAL: prev=72508 ms, new=83849 ms, diff=1.16 slower ❌

Show full query output

      q1: prev=2781 ms, new=2750 ms, diff=1.01 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1744 ms, new=1735 ms, diff=1.01 faster ✔, tasks: prev=94.0, new=94.0, diff=no change
      q3: prev=2969 ms, new=11503 ms, diff=3.87 slower ❌, tasks: prev=56.0, new=56.0, diff=no change
      q4: prev=1418 ms, new=1312 ms, diff=1.08 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=4151 ms, new=4256 ms, diff=1.03 slower ✖, tasks: prev=83.0, new=83.0, diff=no change
      q6: prev=1421 ms, new=1519 ms, diff=1.07 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=4841 ms, new=4973 ms, diff=1.03 slower ✖, tasks: prev=83.0, new=83.0, diff=no change
      q8: prev=5333 ms, new=5492 ms, diff=1.03 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
      q9: prev=6552 ms, new=7296 ms, diff=1.11 slower ✖, tasks: prev=102.0, new=102.0, diff=no change
     q10: prev=5369 ms, new=6220 ms, diff=1.16 slower ✖, tasks: prev=69.0, new=69.0, diff=no change
     q11: prev=1126 ms, new=1223 ms, diff=1.09 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=2008 ms, new=2025 ms, diff=1.01 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1767 ms, new=1742 ms, diff=1.01 faster ✔, tasks: prev=44.0, new=44.0, diff=no change
     q14: prev=1899 ms, new=1945 ms, diff=1.02 slower ✖, tasks: prev=28.0, new=28.0, diff=no change
     q15: prev=4093 ms, new=4284 ms, diff=1.05 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 865 ms, new= 939 ms, diff=1.09 slower ✖, tasks: prev=53.0, new=53.0, diff=no change
     q17: prev=4959 ms, new=5008 ms, diff=1.01 slower ✖, tasks: prev=40.0, new=40.0, diff=no change
     q18: prev=5609 ms, new=5676 ms, diff=1.01 slower ✖, tasks: prev=68.0, new=68.0, diff=no change
     q19: prev=2175 ms, new=2133 ms, diff=1.02 faster ✔, tasks: prev=28.0, new=28.0, diff=no change
     q20: prev=2773 ms, new=2824 ms, diff=1.02 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q21: prev=7829 ms, new=8067 ms, diff=1.03 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 826 ms, new= 927 ms, diff=1.12 slower ✖, tasks: prev=40.0, new=40.0, diff=no change

Verification and run details

Job 113 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`ba1f21bcca51e28a05aeeb344b35818e4740bf9c`

Phase	Base	PR head
Build and deployment	1m 39s	1m 27s
All benchmarks	15m 52s	17m 43s
Benchmark `tpch/sf10`	1m 39s	1m 48s
Benchmark `tpcds/sf1`	6m 35s	6m 44s
Benchmark `tpch/sf100`	7m 39s	9m 11s

Workload: tpch/sf10, tpcds/sf1, tpch/sf100 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.2xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 36m 52s

gene-bordegaray · 2026-09-14T10:06:41Z

benchmarks run tpch/sf10 tpcds/sf1

gabot-0 · 2026-09-14T10:07:03Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head 387df12b81f0 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=14221 ms, new=14332 ms, diff=1.01 slower ✖

Show full query output

      q1: prev= 476 ms, new= 377 ms, diff=1.26 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 616 ms, new= 669 ms, diff=1.09 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 650 ms, new= 669 ms, diff=1.03 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 329 ms, new= 317 ms, diff=1.04 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 743 ms, new= 745 ms, diff=1.00 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 229 ms, new= 269 ms, diff=1.17 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 868 ms, new= 838 ms, diff=1.04 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1001 ms, new= 986 ms, diff=1.02 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1302 ms, new=1175 ms, diff=1.11 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1029 ms, new=1039 ms, diff=1.01 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 430 ms, new= 440 ms, diff=1.02 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 496 ms, new= 489 ms, diff=1.01 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 569 ms, new= 539 ms, diff=1.06 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 379 ms, new= 404 ms, diff=1.07 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 677 ms, new= 632 ms, diff=1.07 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 375 ms, new= 319 ms, diff=1.18 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 678 ms, new= 794 ms, diff=1.17 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1003 ms, new=1037 ms, diff=1.03 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 420 ms, new= 497 ms, diff=1.18 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 620 ms, new= 641 ms, diff=1.03 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1059 ms, new=1176 ms, diff=1.11 slower ✖, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 272 ms, new= 280 ms, diff=1.03 slower ✖, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=130.0, new=130.0, diff=no change (sum of per-query averages)
TOTAL: prev=61978 ms, new=63403 ms, diff=1.02 slower ✖

Show full query output

      q1: prev= 199 ms, new= 206 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 730 ms, new= 676 ms, diff=1.08 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 165 ms, new= 155 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2204 ms, new=2274 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 449 ms, new= 457 ms, diff=1.02 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4575 ms, new=4641 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 241 ms, new= 267 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 178 ms, new= 222 ms, diff=1.25 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 301 ms, new= 290 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 329 ms, new= 425 ms, diff=1.29 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1574 ms, new=1583 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 107 ms, new= 115 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 392 ms, new= 403 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 989 ms, new= 915 ms, diff=1.08 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 181 ms, new= 228 ms, diff=1.26 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 194 ms, new= 220 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 272 ms, new= 284 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 330 ms, new= 372 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 186 ms, new= 212 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 106 ms, new= 117 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 117 ms, new= 118 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 398 ms, new= 404 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 581 ms, new= 591 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 383 ms, new= 383 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 287 ms, new= 309 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 180 ms, new= 187 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 510 ms, new= 796 ms, diff=1.56 slower ❌, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 290 ms, new= 163 ms, diff=1.78 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 312 ms, new= 298 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev=1057 ms, new= 700 ms, diff=1.51 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 260 ms, new= 161 ms, diff=1.61 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 179 ms, new= 167 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 163 ms, new= 177 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 338 ms, new= 386 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 442 ms, new= 417 ms, diff=1.06 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 289 ms, new= 227 ms, diff=1.27 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 211 ms, new= 253 ms, diff=1.20 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 192 ms, new= 191 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 267 ms, new= 262 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  74 ms, new=  78 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 141 ms, new= 147 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 116 ms, new= 119 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 186 ms, new= 181 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 220 ms, new= 199 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 286 ms, new= 305 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 618 ms, new= 604 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 312 ms, new= 306 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 604 ms, new= 511 ms, diff=1.18 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 329 ms, new= 367 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 326 ms, new= 330 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 150 ms, new= 148 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 144 ms, new= 137 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 603 ms, new= 580 ms, diff=1.04 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 124 ms, new= 122 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 177 ms, new= 220 ms, diff=1.24 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 371 ms, new= 555 ms, diff=1.50 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 402 ms, new= 599 ms, diff=1.49 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 267 ms, new= 297 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 179 ms, new= 394 ms, diff=2.20 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev= 981 ms, new=1119 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1358 ms, new=1405 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 130 ms, new= 141 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev= 917 ms, new=1174 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 264 ms, new= 300 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 420 ms, new= 388 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 473 ms, new= 484 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 292 ms, new= 317 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 430 ms, new= 337 ms, diff=1.28 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 466 ms, new= 475 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 428 ms, new= 321 ms, diff=1.33 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7180 ms, new=7248 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 182 ms, new= 194 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 676 ms, new= 796 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev= 908 ms, new= 984 ms, diff=1.08 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 368 ms, new= 349 ms, diff=1.05 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 537 ms, new= 507 ms, diff=1.06 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 564 ms, new= 767 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 201 ms, new= 216 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 707 ms, new= 727 ms, diff=1.03 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 209 ms, new= 210 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 234 ms, new= 264 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 275 ms, new= 270 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 177 ms, new= 195 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 341 ms, new= 353 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 105 ms, new= 108 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 215 ms, new= 236 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 365 ms, new= 298 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 154 ms, new= 170 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 126 ms, new= 139 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 298 ms, new= 315 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 159 ms, new= 167 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 191 ms, new= 174 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 211 ms, new= 200 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 267 ms, new= 255 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 137 ms, new= 144 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 179 ms, new= 182 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 137 ms, new= 133 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14929 ms, new=14890 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 116 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`387df12b81f01feb6db99beb674a47bb73359887`

Phase	Base	PR head
Build and deployment	56s	1m 27s
All benchmarks	8m 6s	8m 23s
Benchmark `tpch/sf10`	1m 34s	1m 35s
Benchmark `tpcds/sf1`	6m 32s	6m 48s

Workload: tpch/sf10, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 19m 3s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-14T12:51:15Z

I think what is happening is now we are evicting on every update which is causing lock contention and small updates to the vec. We could set a threshold to clean up every say X times we add something to the queue. Or we could get fancy and create sepaate bucets that are immutable and lock free of some predetermined sixe then free them after all consumers havve passed.

gene-bordegaray · 2026-09-14T13:29:15Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gabot-0 · 2026-09-14T13:29:26Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head ca6822d24d50 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=15565 ms, new=14352 ms, diff=1.08 faster ✔

Show full query output

      q1: prev= 460 ms, new= 527 ms, diff=1.15 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 700 ms, new= 612 ms, diff=1.14 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 644 ms, new= 732 ms, diff=1.14 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 410 ms, new= 390 ms, diff=1.05 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 742 ms, new= 751 ms, diff=1.01 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 238 ms, new= 223 ms, diff=1.07 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev=1135 ms, new= 880 ms, diff=1.29 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1324 ms, new=1013 ms, diff=1.31 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1265 ms, new=1114 ms, diff=1.14 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1197 ms, new=1161 ms, diff=1.03 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 454 ms, new= 446 ms, diff=1.02 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 463 ms, new= 436 ms, diff=1.06 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 588 ms, new= 540 ms, diff=1.09 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 558 ms, new= 388 ms, diff=1.44 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 835 ms, new= 721 ms, diff=1.16 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 315 ms, new= 321 ms, diff=1.02 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 651 ms, new= 699 ms, diff=1.07 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1034 ms, new=1014 ms, diff=1.02 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 412 ms, new= 429 ms, diff=1.04 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 669 ms, new= 632 ms, diff=1.06 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev= 985 ms, new=1045 ms, diff=1.06 slower ✖, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 486 ms, new= 278 ms, diff=1.75 faster ✅, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=50946 ms, new=51629 ms, diff=1.01 slower ✖

Show full query output

      q1: prev=1703 ms, new=1873 ms, diff=1.10 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1495 ms, new=1555 ms, diff=1.04 slower ✖, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2382 ms, new=2299 ms, diff=1.04 faster ✔, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1102 ms, new= 992 ms, diff=1.11 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2858 ms, new=2869 ms, diff=1.00 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev= 927 ms, new= 905 ms, diff=1.02 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=3044 ms, new=3215 ms, diff=1.06 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3814 ms, new=3857 ms, diff=1.01 slower ✖, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4291 ms, new=4654 ms, diff=1.08 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4413 ms, new=4360 ms, diff=1.01 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev=1174 ms, new=1027 ms, diff=1.14 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1408 ms, new=1350 ms, diff=1.04 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1122 ms, new=1043 ms, diff=1.08 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1291 ms, new=1396 ms, diff=1.08 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2759 ms, new=2886 ms, diff=1.05 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 951 ms, new= 849 ms, diff=1.12 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3113 ms, new=3304 ms, diff=1.06 slower ✖, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3646 ms, new=3636 ms, diff=1.00 faster ✔, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1506 ms, new=1515 ms, diff=1.01 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=2180 ms, new=2208 ms, diff=1.01 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4979 ms, new=5077 ms, diff=1.02 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 788 ms, new= 759 ms, diff=1.04 faster ✔, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=130.0, new=130.0, diff=no change (sum of per-query averages)
TOTAL: prev=64596 ms, new=63742 ms, diff=1.01 faster ✔

Show full query output

      q1: prev= 191 ms, new= 213 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 575 ms, new= 621 ms, diff=1.08 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 162 ms, new= 155 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2178 ms, new=2352 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 552 ms, new= 569 ms, diff=1.03 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4621 ms, new=4787 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 241 ms, new= 250 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 204 ms, new= 189 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 297 ms, new= 310 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 379 ms, new= 449 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1577 ms, new=1681 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 121 ms, new= 119 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 391 ms, new= 427 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1024 ms, new= 975 ms, diff=1.05 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 230 ms, new= 188 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 216 ms, new= 206 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 285 ms, new= 284 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 365 ms, new= 327 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 255 ms, new= 173 ms, diff=1.47 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 118 ms, new= 101 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 108 ms, new= 111 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 392 ms, new= 454 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 597 ms, new= 588 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 423 ms, new= 421 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 298 ms, new= 317 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 189 ms, new= 203 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 747 ms, new= 532 ms, diff=1.40 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 280 ms, new= 166 ms, diff=1.69 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 484 ms, new= 324 ms, diff=1.49 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 877 ms, new= 645 ms, diff=1.36 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 142 ms, new= 137 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 220 ms, new= 165 ms, diff=1.33 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 398 ms, new= 175 ms, diff=2.27 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 376 ms, new= 362 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 456 ms, new= 558 ms, diff=1.22 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 244 ms, new= 228 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 267 ms, new= 249 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 204 ms, new= 211 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 274 ms, new= 284 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  78 ms, new=  79 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 148 ms, new= 160 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 160 ms, new= 123 ms, diff=1.30 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 201 ms, new= 192 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 212 ms, new= 217 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 343 ms, new= 314 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 594 ms, new= 622 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 306 ms, new= 339 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 585 ms, new= 514 ms, diff=1.14 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 321 ms, new= 330 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 310 ms, new= 328 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 154 ms, new= 144 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 142 ms, new= 138 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 598 ms, new= 601 ms, diff=1.01 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 122 ms, new= 117 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 362 ms, new= 261 ms, diff=1.39 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 435 ms, new= 392 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 607 ms, new= 430 ms, diff=1.41 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 374 ms, new= 286 ms, diff=1.31 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 418 ms, new= 175 ms, diff=2.39 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1045 ms, new=1354 ms, diff=1.30 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1361 ms, new=1439 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 146 ms, new= 138 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1100 ms, new=1016 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 289 ms, new= 294 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 398 ms, new= 398 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 480 ms, new= 483 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 303 ms, new= 291 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 351 ms, new= 355 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 463 ms, new= 537 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 320 ms, new= 394 ms, diff=1.23 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7092 ms, new=7558 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 166 ms, new= 181 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 672 ms, new= 775 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1095 ms, new=1035 ms, diff=1.06 faster ✔, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 354 ms, new= 348 ms, diff=1.02 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 574 ms, new= 514 ms, diff=1.12 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 619 ms, new= 573 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 228 ms, new= 227 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 729 ms, new= 762 ms, diff=1.05 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 219 ms, new= 212 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 232 ms, new= 241 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 236 ms, new= 243 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 205 ms, new= 193 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 396 ms, new= 377 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 114 ms, new= 110 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 347 ms, new= 243 ms, diff=1.43 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 298 ms, new= 300 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 162 ms, new= 165 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 129 ms, new= 128 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 286 ms, new= 290 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 174 ms, new= 195 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 251 ms, new= 193 ms, diff=1.30 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 376 ms, new= 223 ms, diff=1.69 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 448 ms, new= 288 ms, diff=1.56 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 155 ms, new= 139 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 204 ms, new= 184 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 140 ms, new= 142 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=15011 ms, new=14936 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 117 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`ca6822d24d505979fd69d60799aab121dfbf067e`

Phase	Base	PR head
Build and deployment	1m 30s	1m 28s
All benchmarks	14m 7s	13m 49s
Benchmark `tpch/sf10`	1m 45s	1m 37s
Benchmark `tpch/sf100`	5m 30s	5m 29s
Benchmark `tpcds/sf1`	6m 52s	6m 43s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 31m 7s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-14T14:02:33Z

ok more promising now. I am going to play around with immutable buckets later

gene-bordegaray · 2026-09-15T10:06:33Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gabot-0 · 2026-09-15T10:06:53Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head 668299e60d3c · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=14974 ms, new=14038 ms, diff=1.07 faster ✔

Show full query output

      q1: prev= 431 ms, new= 487 ms, diff=1.13 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 731 ms, new= 623 ms, diff=1.17 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 781 ms, new= 769 ms, diff=1.02 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 458 ms, new= 332 ms, diff=1.38 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 738 ms, new= 730 ms, diff=1.01 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 238 ms, new= 252 ms, diff=1.06 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 871 ms, new= 800 ms, diff=1.09 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1123 ms, new=1018 ms, diff=1.10 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1265 ms, new=1075 ms, diff=1.18 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1048 ms, new=1050 ms, diff=1.00 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 459 ms, new= 428 ms, diff=1.07 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 434 ms, new= 410 ms, diff=1.06 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 542 ms, new= 534 ms, diff=1.01 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 408 ms, new= 360 ms, diff=1.13 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 657 ms, new= 601 ms, diff=1.09 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 335 ms, new= 318 ms, diff=1.05 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 722 ms, new= 637 ms, diff=1.13 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1019 ms, new=1048 ms, diff=1.03 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 463 ms, new= 487 ms, diff=1.05 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 706 ms, new= 748 ms, diff=1.06 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1259 ms, new=1051 ms, diff=1.20 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 286 ms, new= 280 ms, diff=1.02 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=51740 ms, new=50927 ms, diff=1.02 faster ✔

Show full query output

      q1: prev=1770 ms, new=1734 ms, diff=1.02 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1706 ms, new=1527 ms, diff=1.12 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2631 ms, new=2222 ms, diff=1.18 faster ✔, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1154 ms, new=1116 ms, diff=1.03 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2719 ms, new=2735 ms, diff=1.01 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev= 915 ms, new= 936 ms, diff=1.02 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=3022 ms, new=3089 ms, diff=1.02 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3614 ms, new=3588 ms, diff=1.01 faster ✔, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4527 ms, new=4411 ms, diff=1.03 faster ✔, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4716 ms, new=4385 ms, diff=1.08 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev=1073 ms, new= 963 ms, diff=1.11 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1456 ms, new=1719 ms, diff=1.18 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1013 ms, new= 997 ms, diff=1.02 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1337 ms, new=1293 ms, diff=1.03 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2758 ms, new=2762 ms, diff=1.00 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 956 ms, new= 836 ms, diff=1.14 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3177 ms, new=3153 ms, diff=1.01 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3649 ms, new=3632 ms, diff=1.00 faster ✔, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1655 ms, new=1449 ms, diff=1.14 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=2199 ms, new=2391 ms, diff=1.09 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4915 ms, new=5059 ms, diff=1.03 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 778 ms, new= 930 ms, diff=1.20 slower ✖, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=130.0, new=130.0, diff=no change (sum of per-query averages)
TOTAL: prev=66838 ms, new=63514 ms, diff=1.05 faster ✔

Show full query output

      q1: prev= 224 ms, new= 217 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 801 ms, new= 767 ms, diff=1.04 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 158 ms, new= 171 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2445 ms, new=2150 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 517 ms, new= 615 ms, diff=1.19 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4726 ms, new=4549 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 273 ms, new= 243 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 250 ms, new= 203 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 286 ms, new= 290 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 408 ms, new= 363 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1640 ms, new=1624 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 181 ms, new= 141 ms, diff=1.28 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 645 ms, new= 472 ms, diff=1.37 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1144 ms, new= 948 ms, diff=1.21 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 249 ms, new= 229 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 251 ms, new= 286 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 369 ms, new= 337 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 587 ms, new= 416 ms, diff=1.41 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 216 ms, new= 205 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 119 ms, new= 109 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 121 ms, new= 110 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 404 ms, new= 389 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 608 ms, new= 581 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 529 ms, new= 381 ms, diff=1.39 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 347 ms, new= 300 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 204 ms, new= 179 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 838 ms, new= 847 ms, diff=1.01 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 287 ms, new= 279 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 463 ms, new= 341 ms, diff=1.36 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 759 ms, new= 677 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 135 ms, new= 172 ms, diff=1.27 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 184 ms, new= 188 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 193 ms, new= 188 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 362 ms, new= 391 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 399 ms, new= 438 ms, diff=1.10 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 227 ms, new= 236 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 236 ms, new= 249 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 203 ms, new= 203 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 253 ms, new= 274 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  83 ms, new=  86 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 159 ms, new= 154 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 126 ms, new= 116 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 180 ms, new= 180 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 229 ms, new= 227 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 421 ms, new= 312 ms, diff=1.35 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 602 ms, new= 623 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 354 ms, new= 319 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 509 ms, new= 638 ms, diff=1.25 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 349 ms, new= 309 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 328 ms, new= 319 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 146 ms, new= 171 ms, diff=1.17 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 125 ms, new= 143 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 600 ms, new= 634 ms, diff=1.06 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 117 ms, new= 132 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 188 ms, new= 206 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 382 ms, new= 423 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 414 ms, new= 430 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 249 ms, new= 281 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 197 ms, new= 351 ms, diff=1.78 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1068 ms, new=1204 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1405 ms, new=1364 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 138 ms, new= 136 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1134 ms, new= 982 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 289 ms, new= 310 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 413 ms, new= 398 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 509 ms, new= 478 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 334 ms, new= 330 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 458 ms, new= 353 ms, diff=1.30 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 493 ms, new= 490 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 379 ms, new= 322 ms, diff=1.18 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7341 ms, new=7193 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 365 ms, new= 240 ms, diff=1.52 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 984 ms, new= 744 ms, diff=1.32 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1042 ms, new=1005 ms, diff=1.04 faster ✔, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 453 ms, new= 395 ms, diff=1.15 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 539 ms, new= 513 ms, diff=1.05 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 648 ms, new= 561 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 229 ms, new= 239 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 910 ms, new= 749 ms, diff=1.21 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 236 ms, new= 212 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 248 ms, new= 243 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 258 ms, new= 250 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 206 ms, new= 196 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 424 ms, new= 366 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 107 ms, new= 108 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 279 ms, new= 220 ms, diff=1.27 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 408 ms, new= 291 ms, diff=1.40 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 181 ms, new= 160 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 156 ms, new= 125 ms, diff=1.25 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 666 ms, new= 293 ms, diff=2.27 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 385 ms, new= 166 ms, diff=2.32 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 203 ms, new= 181 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 267 ms, new= 194 ms, diff=1.38 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 334 ms, new= 290 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 155 ms, new= 160 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 178 ms, new= 181 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 156 ms, new= 144 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14861 ms, new=14916 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 134 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`668299e60d3c05d658146085b223814cc4bbc8aa`

Phase	Base	PR head
Build and deployment	1m 48s	1m 27s
All benchmarks	14m 16s	13m 41s
Benchmark `tpch/sf10`	1m 39s	1m 33s
Benchmark `tpch/sf100`	5m 32s	5m 28s
Benchmark `tpcds/sf1`	7m 5s	6m 40s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 31m 25s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-15T20:55:04Z

I tried a lockless design and the allocation overhead outwighs the mutex contention. The contention is pretty limited to small regions so I am not worried

gene-bordegaray · 2026-09-16T00:14:33Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gabot-0 · 2026-09-16T00:14:48Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head 948ab77a2c9b · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=15174 ms, new=16152 ms, diff=1.06 slower ✖

Show full query output

      q1: prev= 518 ms, new= 548 ms, diff=1.06 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 586 ms, new= 694 ms, diff=1.18 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 640 ms, new= 695 ms, diff=1.09 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 376 ms, new= 379 ms, diff=1.01 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 733 ms, new= 866 ms, diff=1.18 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 231 ms, new= 285 ms, diff=1.23 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 868 ms, new= 955 ms, diff=1.10 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev= 996 ms, new=1337 ms, diff=1.34 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1225 ms, new=1142 ms, diff=1.07 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1138 ms, new=1153 ms, diff=1.01 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 452 ms, new= 484 ms, diff=1.07 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 469 ms, new= 561 ms, diff=1.20 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 550 ms, new= 574 ms, diff=1.04 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 551 ms, new= 626 ms, diff=1.14 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev=1021 ms, new= 703 ms, diff=1.45 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 313 ms, new= 334 ms, diff=1.07 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 752 ms, new= 709 ms, diff=1.06 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1047 ms, new=1163 ms, diff=1.11 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 553 ms, new= 555 ms, diff=1.00 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 782 ms, new= 707 ms, diff=1.11 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1073 ms, new=1390 ms, diff=1.30 slower ✖, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 300 ms, new= 292 ms, diff=1.03 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=54735 ms, new=53347 ms, diff=1.03 faster ✔

Show full query output

      q1: prev=1782 ms, new=1897 ms, diff=1.06 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1681 ms, new=1638 ms, diff=1.03 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2345 ms, new=2870 ms, diff=1.22 slower ✖, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1164 ms, new=1190 ms, diff=1.02 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2825 ms, new=2916 ms, diff=1.03 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev=1055 ms, new= 972 ms, diff=1.09 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=3215 ms, new=3292 ms, diff=1.02 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3626 ms, new=3817 ms, diff=1.05 slower ✖, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4531 ms, new=4887 ms, diff=1.08 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4438 ms, new=4573 ms, diff=1.03 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev=1196 ms, new=1080 ms, diff=1.11 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1579 ms, new=1505 ms, diff=1.05 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1208 ms, new=1089 ms, diff=1.11 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1363 ms, new=1434 ms, diff=1.05 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=3168 ms, new=2722 ms, diff=1.16 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 950 ms, new= 876 ms, diff=1.08 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3242 ms, new=3163 ms, diff=1.02 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3651 ms, new=3864 ms, diff=1.06 slower ✖, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1513 ms, new=1554 ms, diff=1.03 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=3938 ms, new=2170 ms, diff=1.81 faster ✅, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=5420 ms, new=5118 ms, diff=1.06 faster ✔, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 845 ms, new= 720 ms, diff=1.17 faster ✔, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=130.0, new=130.0, diff=no change (sum of per-query averages)
TOTAL: prev=66400 ms, new=64953 ms, diff=1.02 faster ✔

Show full query output

      q1: prev= 217 ms, new= 206 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 690 ms, new= 670 ms, diff=1.03 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 169 ms, new= 196 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2218 ms, new=2436 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 505 ms, new= 514 ms, diff=1.02 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4596 ms, new=4789 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 260 ms, new= 424 ms, diff=1.63 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 210 ms, new= 287 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 287 ms, new= 296 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 447 ms, new= 348 ms, diff=1.28 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1593 ms, new=1661 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 111 ms, new= 115 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 416 ms, new= 386 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 941 ms, new= 884 ms, diff=1.06 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 207 ms, new= 248 ms, diff=1.20 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 225 ms, new= 215 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 291 ms, new= 290 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 353 ms, new= 361 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 213 ms, new= 188 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 109 ms, new= 103 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 127 ms, new= 108 ms, diff=1.18 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 407 ms, new= 383 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 595 ms, new= 590 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 665 ms, new= 477 ms, diff=1.39 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 329 ms, new= 334 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 191 ms, new= 186 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 768 ms, new= 892 ms, diff=1.16 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 200 ms, new= 266 ms, diff=1.33 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 343 ms, new= 336 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 741 ms, new= 854 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 202 ms, new= 153 ms, diff=1.32 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 250 ms, new= 205 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 182 ms, new= 404 ms, diff=2.22 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 513 ms, new= 394 ms, diff=1.30 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 482 ms, new= 425 ms, diff=1.13 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 261 ms, new= 232 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 270 ms, new= 230 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 200 ms, new= 199 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 273 ms, new= 261 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  88 ms, new=  75 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 153 ms, new= 144 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 124 ms, new= 121 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 205 ms, new= 188 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 234 ms, new= 215 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 555 ms, new= 439 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 610 ms, new= 582 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 331 ms, new= 340 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 605 ms, new= 550 ms, diff=1.10 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 402 ms, new= 317 ms, diff=1.27 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 359 ms, new= 323 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 167 ms, new= 152 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 150 ms, new= 143 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 715 ms, new= 642 ms, diff=1.11 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 237 ms, new= 115 ms, diff=2.06 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 384 ms, new= 258 ms, diff=1.49 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 407 ms, new= 385 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 485 ms, new= 434 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 288 ms, new= 272 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 258 ms, new= 371 ms, diff=1.44 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1069 ms, new=1151 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1390 ms, new=1395 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 167 ms, new= 160 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1718 ms, new=1235 ms, diff=1.39 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 348 ms, new= 428 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 445 ms, new= 430 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 484 ms, new= 488 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 516 ms, new= 372 ms, diff=1.39 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 471 ms, new= 402 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 505 ms, new= 484 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 325 ms, new= 301 ms, diff=1.08 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7336 ms, new=7257 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 222 ms, new= 168 ms, diff=1.32 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 912 ms, new= 719 ms, diff=1.27 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev= 976 ms, new=1002 ms, diff=1.03 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 291 ms, new= 363 ms, diff=1.25 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 619 ms, new= 565 ms, diff=1.10 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 639 ms, new= 579 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 226 ms, new= 236 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 759 ms, new= 875 ms, diff=1.15 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 270 ms, new= 215 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 287 ms, new= 245 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 259 ms, new= 249 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 245 ms, new= 241 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 440 ms, new= 358 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 142 ms, new= 100 ms, diff=1.42 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 273 ms, new= 230 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 301 ms, new= 301 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 167 ms, new= 161 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 134 ms, new= 132 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 321 ms, new= 297 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 171 ms, new= 167 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 203 ms, new= 185 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 248 ms, new= 220 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 270 ms, new= 278 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 160 ms, new= 146 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 203 ms, new= 181 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 131 ms, new= 140 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14943 ms, new=14885 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 140 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`948ab77a2c9b28f57dc0a3fc9f3b717343c96b54`

Phase	Base	PR head
Build and deployment	1m 27s	1m 28s
All benchmarks	14m 44s	14m 27s
Benchmark `tpch/sf10`	1m 43s	1m 48s
Benchmark `tpch/sf100`	5m 56s	5m 48s
Benchmark `tpcds/sf1`	7m 5s	6m 51s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 32m 16s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-16T00:48:29Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gabot-0 · 2026-09-16T00:48:38Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head 948ab77a2c9b · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=258.0, new=258.0, diff=no change (sum of per-query averages)
TOTAL: prev=6757 ms, new=6675 ms, diff=1.01 faster ✔

Show full query output

      q1: prev= 453 ms, new= 400 ms, diff=1.13 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 670 ms, new= 652 ms, diff=1.03 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 681 ms, new= 857 ms, diff=1.26 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 574 ms, new= 411 ms, diff=1.40 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 805 ms, new= 809 ms, diff=1.00 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 264 ms, new= 239 ms, diff=1.10 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 877 ms, new= 972 ms, diff=1.11 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1100 ms, new=1085 ms, diff=1.01 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1333 ms, new=1250 ms, diff=1.07 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
q10: Previously succeeded, but now failed ❌
q11: Previously succeeded, but now failed ❌
q12: Previously succeeded, but now failed ❌
q13: Previously succeeded, but now failed ❌
q14: Previously succeeded, but now failed ❌
q15: Previously succeeded, but now failed ❌
q16: Previously succeeded, but now failed ❌
q17: Previously succeeded, but now failed ❌
q18: Previously succeeded, but now failed ❌
q19: Previously succeeded, but now failed ❌
q20: Previously succeeded, but now failed ❌
q21: Previously succeeded, but now failed ❌
q22: Previously succeeded, but now failed ❌

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1202.4, diff=31.6 fewer (2.6%) (sum of per-query averages)
TOTAL: prev=51280 ms, new=52980 ms, diff=1.03 slower ✖

Show full query output

      q1: prev=1864 ms, new=1930 ms, diff=1.04 slower ✖, tasks: prev=24.0, new=20.4, diff=3.6 fewer (15.0%)
      q2: prev=1607 ms, new=1438 ms, diff=1.12 faster ✔, tasks: prev=92.0, new=73.8, diff=18.2 fewer (19.8%)
      q3: prev=2209 ms, new=2546 ms, diff=1.15 slower ✖, tasks: prev=52.0, new=45.6, diff=6.4 fewer (12.3%)
      q4: prev=1235 ms, new=1162 ms, diff=1.06 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2818 ms, new=2966 ms, diff=1.05 slower ✖, tasks: prev=79.0, new=77.8, diff=1.2 fewer (1.5%)
      q6: prev= 918 ms, new=1023 ms, diff=1.11 slower ✖, tasks: prev=12.0, new=11.0, diff=1.0 fewer (8.3%)
      q7: prev=3090 ms, new=3111 ms, diff=1.01 slower ✖, tasks: prev=79.0, new=77.8, diff=1.2 fewer (1.5%)
      q8: prev=3589 ms, new=3687 ms, diff=1.03 slower ✖, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4381 ms, new=4548 ms, diff=1.04 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4670 ms, new=4852 ms, diff=1.04 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev=1099 ms, new=1085 ms, diff=1.01 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1420 ms, new=1588 ms, diff=1.12 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1107 ms, new=1212 ms, diff=1.09 slower ✖, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1260 ms, new=1537 ms, diff=1.22 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2743 ms, new=2986 ms, diff=1.09 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev=1124 ms, new= 899 ms, diff=1.25 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3154 ms, new=3211 ms, diff=1.02 slower ✖, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3635 ms, new=3669 ms, diff=1.01 slower ✖, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1432 ms, new=1469 ms, diff=1.03 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=2216 ms, new=2304 ms, diff=1.04 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4924 ms, new=4942 ms, diff=1.00 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 785 ms, new= 815 ms, diff=1.04 slower ✖, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=83.0, new=83.0, diff=no change (sum of per-query averages)
TOTAL: prev=32351 ms, new=36262 ms, diff=1.12 slower ❌

Show full query output

      q1: prev= 204 ms, new= 298 ms, diff=1.46 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 748 ms, new= 675 ms, diff=1.11 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 153 ms, new= 209 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2184 ms, new=2466 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 604 ms, new= 642 ms, diff=1.06 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4534 ms, new=5015 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 238 ms, new= 321 ms, diff=1.35 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 220 ms, new= 382 ms, diff=1.74 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 296 ms, new= 360 ms, diff=1.22 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 389 ms, new= 418 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1545 ms, new=1750 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 115 ms, new= 211 ms, diff=1.83 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 454 ms, new= 535 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 918 ms, new= 976 ms, diff=1.06 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 205 ms, new= 255 ms, diff=1.24 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 216 ms, new= 340 ms, diff=1.57 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 291 ms, new= 487 ms, diff=1.67 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 353 ms, new= 433 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 189 ms, new= 198 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 102 ms, new= 107 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 120 ms, new= 124 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 387 ms, new= 403 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 592 ms, new= 619 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 417 ms, new= 469 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 297 ms, new= 495 ms, diff=1.67 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 194 ms, new= 244 ms, diff=1.26 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 786 ms, new= 853 ms, diff=1.09 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 163 ms, new= 195 ms, diff=1.20 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 316 ms, new= 433 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 630 ms, new= 969 ms, diff=1.54 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 129 ms, new= 307 ms, diff=2.38 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 172 ms, new= 373 ms, diff=2.17 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 187 ms, new= 254 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 364 ms, new= 422 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 407 ms, new= 489 ms, diff=1.20 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 238 ms, new= 238 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 240 ms, new= 278 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 197 ms, new= 200 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 264 ms, new= 277 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  73 ms, new=  78 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 146 ms, new= 148 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 125 ms, new= 122 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 220 ms, new= 191 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 227 ms, new= 269 ms, diff=1.19 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 529 ms, new= 362 ms, diff=1.46 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 666 ms, new= 673 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 345 ms, new= 351 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 684 ms, new= 692 ms, diff=1.01 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 303 ms, new= 320 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 314 ms, new= 349 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 163 ms, new= 160 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 137 ms, new= 143 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 635 ms, new= 616 ms, diff=1.03 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 116 ms, new= 121 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 196 ms, new= 271 ms, diff=1.38 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 380 ms, new= 406 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 442 ms, new= 601 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 238 ms, new= 259 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 199 ms, new= 242 ms, diff=1.22 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev= 987 ms, new=1124 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1378 ms, new=1412 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 134 ms, new= 148 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1086 ms, new=1039 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 431 ms, new= 272 ms, diff=1.58 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 404 ms, new= 417 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 487 ms, new= 478 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 404 ms, new= 368 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 382 ms, new= 401 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 462 ms, new= 509 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q71: Previously failed, but now succeeded 🟠
q72: Previously failed, but now succeeded 🟠
q73: Previously failed, but now succeeded 🟠
q74: Previously failed, but now succeeded 🟠
q75: Previously failed, but now succeeded 🟠
q76: Previously failed, but now succeeded 🟠
q77: Previously failed, but now succeeded 🟠
q78: Previously failed, but now succeeded 🟠
q79: Previously failed, but now succeeded 🟠
q80: Previously failed, but now succeeded 🟠
q81: Previously failed, but now succeeded 🟠
q82: Previously failed, but now succeeded 🟠
q83: Previously failed, but now succeeded 🟠
q84: Previously failed, but now succeeded 🟠
q85: Previously failed, but now succeeded 🟠
q86: Previously failed, but now succeeded 🟠
q87: Previously failed, but now succeeded 🟠
q88: Previously failed, but now succeeded 🟠
q89: Previously failed, but now succeeded 🟠
q90: Previously failed, but now succeeded 🟠
q91: Previously failed, but now succeeded 🟠
q92: Previously failed, but now succeeded 🟠
q93: Previously failed, but now succeeded 🟠
q94: Previously failed, but now succeeded 🟠
q95: Previously failed, but now succeeded 🟠
q96: Previously failed, but now succeeded 🟠
q97: Previously failed, but now succeeded 🟠
q98: Previously failed, but now succeeded 🟠
q99: Previously failed, but now succeeded 🟠

Verification and run details

Job 141 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`948ab77a2c9b28f57dc0a3fc9f3b717343c96b54`

Phase	Base	PR head
Build and deployment	1m 28s	2m 5s
All benchmarks	10m 44s	13m 34s
Benchmark `tpch/sf10`	1m 42s	47s
Benchmark `tpch/sf100`	5m 33s	5m 39s
Benchmark `tpcds/sf1`	3m 29s	7m 9s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 28m 2s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-16T00:50:39Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1 --nodes 50

gabot-0 · 2026-09-16T01:17:13Z

Requested by this comment.

Benchmark results

Compared: PR base 22e333ca96a8 → PR head 948ab77a2c9b · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=17093 ms, new=17187 ms, diff=1.01 slower ✖

Show full query output

      q1: prev= 517 ms, new= 519 ms, diff=1.00 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 807 ms, new= 896 ms, diff=1.11 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 903 ms, new= 906 ms, diff=1.00 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 448 ms, new= 537 ms, diff=1.20 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev=1064 ms, new= 989 ms, diff=1.08 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 258 ms, new= 307 ms, diff=1.19 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 993 ms, new=1127 ms, diff=1.13 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1312 ms, new=1416 ms, diff=1.08 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1522 ms, new=1357 ms, diff=1.12 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1213 ms, new=1260 ms, diff=1.04 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 624 ms, new= 578 ms, diff=1.08 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 489 ms, new= 524 ms, diff=1.07 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 596 ms, new= 541 ms, diff=1.10 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 565 ms, new= 452 ms, diff=1.25 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 739 ms, new= 651 ms, diff=1.14 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 338 ms, new= 385 ms, diff=1.14 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 631 ms, new= 771 ms, diff=1.22 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1117 ms, new=1035 ms, diff=1.08 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 606 ms, new= 472 ms, diff=1.28 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 876 ms, new= 944 ms, diff=1.08 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1206 ms, new=1187 ms, diff=1.02 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 269 ms, new= 333 ms, diff=1.24 slower ✖, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1986.0, new=1986.0, diff=no change (sum of per-query averages)
TOTAL: prev=41342 ms, new=40121 ms, diff=1.03 faster ✔

Show full query output

      q1: prev=1442 ms, new=1431 ms, diff=1.01 faster ✔, tasks: prev=100.0, new=100.0, diff=no change
      q2: prev=1957 ms, new=1958 ms, diff=1.00 slower ✖, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=3081 ms, new=2929 ms, diff=1.05 faster ✔, tasks: prev=140.0, new=140.0, diff=no change
      q4: prev=2506 ms, new=2513 ms, diff=1.00 slower ✖, tasks: prev=168.0, new=168.0, diff=no change
      q5: prev=4336 ms, new=4130 ms, diff=1.05 faster ✔, tasks: prev=243.0, new=243.0, diff=no change
      q6: prev= 743 ms, new= 858 ms, diff=1.15 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q7: prev=5892 ms, new=5221 ms, diff=1.13 faster ✔, tasks: prev=275.0, new=275.0, diff=no change
      q8: prev=6350 ms, new=6374 ms, diff=1.00 slower ✖, tasks: prev=328.0, new=328.0, diff=no change
      q9: prev=7032 ms, new=6931 ms, diff=1.01 faster ✔, tasks: prev=334.0, new=334.0, diff=no change
     q10: prev=6951 ms, new=6650 ms, diff=1.05 faster ✔, tasks: prev=191.0, new=191.0, diff=no change
     q11: prev=1052 ms, new=1126 ms, diff=1.07 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
q12: Previously failed, and now also failed ❌
q13: Previously failed, and now also failed ❌
q14: Previously failed, and now also failed ❌
q15: Previously failed, and now also failed ❌
q16: Previously failed, and now also failed ❌
q17: Previously failed, and now also failed ❌
q18: Previously failed, and now also failed ❌
q19: Previously failed, and now also failed ❌
q20: Previously failed, and now also failed ❌
q21: Previously failed, and now also failed ❌
q22: Previously succeeded, but now failed ❌

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=130.0, new=130.0, diff=no change (sum of per-query averages)
TOTAL: prev=66488 ms, new=65965 ms, diff=1.01 faster ✔

Show full query output

      q1: prev= 196 ms, new= 206 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 792 ms, new= 855 ms, diff=1.08 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q3: prev= 155 ms, new= 162 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2190 ms, new=2284 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 652 ms, new= 604 ms, diff=1.08 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q6: prev=4534 ms, new=4676 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 248 ms, new= 252 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 190 ms, new= 195 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 296 ms, new= 327 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 374 ms, new= 674 ms, diff=1.80 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1547 ms, new=1720 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 103 ms, new= 132 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 408 ms, new= 507 ms, diff=1.24 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 987 ms, new=1096 ms, diff=1.11 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 252 ms, new= 203 ms, diff=1.24 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 215 ms, new= 225 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 310 ms, new= 296 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 409 ms, new= 414 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 197 ms, new= 190 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 108 ms, new= 110 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 114 ms, new= 111 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 413 ms, new= 399 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 628 ms, new= 602 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 433 ms, new= 432 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 410 ms, new= 314 ms, diff=1.31 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 197 ms, new= 191 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 970 ms, new=1021 ms, diff=1.05 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 268 ms, new= 164 ms, diff=1.63 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 362 ms, new= 326 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌
     q31: prev= 756 ms, new= 625 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 176 ms, new= 129 ms, diff=1.36 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 422 ms, new= 179 ms, diff=2.36 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 346 ms, new= 173 ms, diff=2.00 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 577 ms, new= 374 ms, diff=1.54 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 647 ms, new= 610 ms, diff=1.06 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 355 ms, new= 240 ms, diff=1.48 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 240 ms, new= 244 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 203 ms, new= 189 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 272 ms, new= 277 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  79 ms, new=  73 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 146 ms, new= 149 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 121 ms, new= 122 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 298 ms, new= 184 ms, diff=1.62 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 231 ms, new= 265 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 482 ms, new= 333 ms, diff=1.45 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 605 ms, new= 577 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 360 ms, new= 327 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 759 ms, new= 649 ms, diff=1.17 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 339 ms, new= 319 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 307 ms, new= 351 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 151 ms, new= 175 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 130 ms, new= 136 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 773 ms, new= 760 ms, diff=1.02 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
     q55: prev= 115 ms, new= 163 ms, diff=1.42 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 208 ms, new= 193 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 354 ms, new= 367 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 383 ms, new= 385 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 257 ms, new= 262 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 185 ms, new= 212 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1007 ms, new=1111 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1363 ms, new=1402 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 136 ms, new= 135 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1107 ms, new=1133 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 305 ms, new= 308 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 395 ms, new= 400 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 487 ms, new= 480 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 372 ms, new= 321 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 381 ms, new= 384 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 525 ms, new= 460 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 428 ms, new= 350 ms, diff=1.22 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7053 ms, new=7204 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 225 ms, new= 170 ms, diff=1.32 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 736 ms, new= 669 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1466 ms, new=1333 ms, diff=1.10 faster ✔, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 586 ms, new= 494 ms, diff=1.19 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 697 ms, new= 782 ms, diff=1.12 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 659 ms, new= 571 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 227 ms, new= 193 ms, diff=1.18 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 839 ms, new= 890 ms, diff=1.06 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 533 ms, new= 203 ms, diff=2.63 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 257 ms, new= 233 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 288 ms, new= 247 ms, diff=1.17 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 281 ms, new= 197 ms, diff=1.43 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 475 ms, new= 423 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 108 ms, new= 111 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 253 ms, new= 248 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 322 ms, new= 298 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 180 ms, new= 179 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 131 ms, new= 145 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 335 ms, new= 321 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 183 ms, new= 164 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 201 ms, new= 233 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 227 ms, new= 318 ms, diff=1.40 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 333 ms, new= 501 ms, diff=1.50 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 149 ms, new= 345 ms, diff=2.32 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 186 ms, new= 285 ms, diff=1.53 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 144 ms, new= 139 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14673 ms, new=14855 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 142 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-benchmarks --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`22e333ca96a8ada7f70855cedd4949fbc8c1990e`	`948ab77a2c9b28f57dc0a3fc9f3b717343c96b54`

Phase	Base	PR head
Build and deployment	2m 22s	1m 42s
All benchmarks	14m 0s	13m 39s
Benchmark `tpch/sf10`	1m 52s	1m 52s
Benchmark `tpch/sf100`	5m 12s	4m 53s
Benchmark `tpcds/sf1`	6m 56s	6m 54s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query

Capacity: 50 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 31m 55s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection and iteration overrides are not supported; every query uses 1 warmup and 5 measured iterations for both revisions.

gabotechs · 2026-09-16T07:03:03Z

+#[derive(Debug)]
+struct QueueState<T> {
+    entries: VecDeque<T>,
+    base_sequence: usize,
+    tail_sequence: usize,
+    readers: Box<[ReaderSlot]>,


Even if this is really cool, I also see it's unreasonably complex...

I remember we explored this a while back, but have you consider a simple solution like having one mpsc per reader? the current solution is so complex that I'd be fine losing a bit of performance for the sake of simplicity.

My diagrams may make it look more intimidating than it is. I am also open to a channel per producer consumer pair, it would duplicate data sent but could help lock contention

It should not be too bad, as cloning a RecordBatch does not clone the data, everything is hidden behind an atomic reference, so clones are as cheap as cloning Arcs.

I'd at least consider it. If after considering it you still think it's worth to keep this implementation, then let's go with this 👍.

ok sounds good. I can test it ut and hoepfully it performs just as well 😄

gene-bordegaray · 2026-09-23T18:32:11Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1 --nodes 50

gabot-0 · 2026-09-23T18:32:31Z

Requested by this comment.

Benchmark job 190 failed for tpch/sf10, tpch/sf100, tpcds/sf1. Full details are available in the controller journal.

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-23T18:35:37Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1 --nodes 50

gabot-0 · 2026-09-23T18:35:42Z

Requested by this comment.

Benchmark job 191 failed for tpch/sf10, tpch/sf100, tpcds/sf1. Full details are available in the controller journal.

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-23T18:36:48Z

benchmarks run tpch/sf10 --nodes 50

gabot-0 · 2026-09-24T17:37:49Z

Requested by this comment.

Benchmark results

Compared: PR base d7e27c9607d4 → PR head 3c6115c196ac · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=15206 ms, new=15035 ms, diff=1.01 faster ✔

Show full query output

      q1: prev= 448 ms, new= 433 ms, diff=1.03 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 625 ms, new= 767 ms, diff=1.23 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 721 ms, new= 741 ms, diff=1.03 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 447 ms, new= 480 ms, diff=1.07 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 867 ms, new= 778 ms, diff=1.11 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 290 ms, new= 273 ms, diff=1.06 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 878 ms, new= 880 ms, diff=1.00 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1075 ms, new=1063 ms, diff=1.01 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1263 ms, new=1182 ms, diff=1.07 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1044 ms, new=1111 ms, diff=1.06 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 455 ms, new= 448 ms, diff=1.02 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 518 ms, new= 493 ms, diff=1.05 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 547 ms, new= 571 ms, diff=1.04 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 505 ms, new= 461 ms, diff=1.10 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 617 ms, new= 780 ms, diff=1.26 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 334 ms, new= 324 ms, diff=1.03 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 745 ms, new= 632 ms, diff=1.18 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1019 ms, new=1043 ms, diff=1.02 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 496 ms, new= 458 ms, diff=1.08 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 791 ms, new= 757 ms, diff=1.04 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1253 ms, new=1093 ms, diff=1.15 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 268 ms, new= 267 ms, diff=1.00 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=49785 ms, new=49260 ms, diff=1.01 faster ✔

Show full query output

      q1: prev=1943 ms, new=1799 ms, diff=1.08 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1570 ms, new=1365 ms, diff=1.15 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2181 ms, new=2068 ms, diff=1.05 faster ✔, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1181 ms, new=1022 ms, diff=1.16 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2757 ms, new=2657 ms, diff=1.04 faster ✔, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev=1010 ms, new= 977 ms, diff=1.03 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=3003 ms, new=3320 ms, diff=1.11 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3579 ms, new=3271 ms, diff=1.09 faster ✔, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4227 ms, new=4372 ms, diff=1.03 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4326 ms, new=4107 ms, diff=1.05 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev= 794 ms, new= 814 ms, diff=1.03 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1373 ms, new=1314 ms, diff=1.04 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev=1019 ms, new= 956 ms, diff=1.07 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1660 ms, new=1359 ms, diff=1.22 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2642 ms, new=2737 ms, diff=1.04 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 723 ms, new= 671 ms, diff=1.08 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3254 ms, new=3160 ms, diff=1.03 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3481 ms, new=3583 ms, diff=1.03 slower ✖, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1488 ms, new=1554 ms, diff=1.04 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=2073 ms, new=2355 ms, diff=1.14 slower ✖, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4658 ms, new=5017 ms, diff=1.08 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 843 ms, new= 782 ms, diff=1.08 faster ✔, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=64121 ms, new=67007 ms, diff=1.05 slower ✖

Show full query output

      q1: prev= 222 ms, new= 268 ms, diff=1.21 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 218 ms, new= 238 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 198 ms, new= 231 ms, diff=1.17 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2266 ms, new=2236 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 561 ms, new= 635 ms, diff=1.13 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4665 ms, new=4645 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 285 ms, new= 271 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 205 ms, new= 186 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 296 ms, new= 302 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 375 ms, new= 361 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1619 ms, new=1638 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 115 ms, new= 126 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 437 ms, new= 387 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 936 ms, new= 936 ms, diff=1.00 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 212 ms, new= 239 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 227 ms, new= 230 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 315 ms, new= 299 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 341 ms, new= 420 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 200 ms, new= 213 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 120 ms, new= 124 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 140 ms, new= 118 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 431 ms, new= 405 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 613 ms, new= 731 ms, diff=1.19 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 382 ms, new= 794 ms, diff=2.08 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 341 ms, new= 566 ms, diff=1.66 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 176 ms, new= 249 ms, diff=1.41 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 726 ms, new= 897 ms, diff=1.24 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 273 ms, new= 279 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 334 ms, new= 358 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev= 661 ms, new= 774 ms, diff=1.17 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 158 ms, new= 155 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 196 ms, new= 305 ms, diff=1.56 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 191 ms, new= 216 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 360 ms, new= 435 ms, diff=1.21 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 423 ms, new= 441 ms, diff=1.04 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 230 ms, new= 461 ms, diff=2.00 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 529 ms, new= 330 ms, diff=1.60 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 237 ms, new= 220 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 284 ms, new= 314 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  78 ms, new= 106 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 164 ms, new= 191 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 120 ms, new= 147 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 184 ms, new= 236 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 241 ms, new= 241 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 360 ms, new= 394 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 663 ms, new= 637 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 362 ms, new= 355 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 636 ms, new= 593 ms, diff=1.07 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 312 ms, new= 368 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 330 ms, new= 355 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 158 ms, new= 239 ms, diff=1.51 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 160 ms, new= 132 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 361 ms, new= 616 ms, diff=1.71 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 173 ms, new= 149 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 202 ms, new= 344 ms, diff=1.70 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 406 ms, new= 389 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 422 ms, new= 434 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 257 ms, new= 340 ms, diff=1.32 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 194 ms, new= 237 ms, diff=1.22 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1069 ms, new=1140 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1360 ms, new=1378 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 139 ms, new= 203 ms, diff=1.46 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1001 ms, new=1222 ms, diff=1.22 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 289 ms, new= 340 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 390 ms, new= 419 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 499 ms, new= 479 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 340 ms, new= 314 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 368 ms, new= 377 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 494 ms, new= 483 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 414 ms, new= 305 ms, diff=1.36 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7254 ms, new=7460 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 207 ms, new= 191 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 735 ms, new= 917 ms, diff=1.25 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1006 ms, new=1148 ms, diff=1.14 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 352 ms, new= 377 ms, diff=1.07 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 587 ms, new= 707 ms, diff=1.20 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 603 ms, new= 836 ms, diff=1.39 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 245 ms, new= 257 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 817 ms, new= 735 ms, diff=1.11 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 205 ms, new= 326 ms, diff=1.59 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 243 ms, new= 408 ms, diff=1.68 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 277 ms, new= 295 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 243 ms, new= 246 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 372 ms, new= 413 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 112 ms, new= 110 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 264 ms, new= 299 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 421 ms, new= 308 ms, diff=1.37 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 325 ms, new= 180 ms, diff=1.81 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 143 ms, new= 148 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 668 ms, new= 336 ms, diff=1.99 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 414 ms, new= 255 ms, diff=1.62 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 381 ms, new= 202 ms, diff=1.89 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 259 ms, new= 287 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 437 ms, new= 368 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 156 ms, new= 148 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 204 ms, new= 206 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 155 ms, new= 156 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14392 ms, new=14492 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 206 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`d7e27c9607d4f53c0d77daf1431bd2bffd99f076`	`3c6115c196ac7423f0272403ebfdf789ada4ed8c`

Phase	Base	PR head
Build and deployment	1m 39s	1m 35s
All benchmarks	13m 57s	14m 3s
Benchmark `tpch/sf10`	1m 46s	1m 43s
Benchmark `tpch/sf100`	5m 25s	5m 15s
Benchmark `tpcds/sf1`	6m 47s	7m 6s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 2s · Dataset validation 1s · Total 31m 28s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gabot-0 · 2026-09-24T17:37:50Z

Requested by this comment.

Benchmark results

Compared: PR base d7e27c9607d4 → PR head 3c6115c196ac · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=18021 ms, new=19063 ms, diff=1.06 slower ✖

Show full query output

      q1: prev= 574 ms, new= 617 ms, diff=1.07 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 893 ms, new= 984 ms, diff=1.10 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 758 ms, new= 811 ms, diff=1.07 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 583 ms, new= 750 ms, diff=1.29 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 944 ms, new=1022 ms, diff=1.08 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 288 ms, new= 359 ms, diff=1.25 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev=1031 ms, new=1048 ms, diff=1.02 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1174 ms, new=1190 ms, diff=1.01 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1463 ms, new=1448 ms, diff=1.01 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1188 ms, new=1335 ms, diff=1.12 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 721 ms, new= 791 ms, diff=1.10 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 699 ms, new= 564 ms, diff=1.24 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 563 ms, new= 611 ms, diff=1.09 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 568 ms, new= 580 ms, diff=1.02 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 909 ms, new= 934 ms, diff=1.03 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 487 ms, new= 420 ms, diff=1.16 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 835 ms, new= 846 ms, diff=1.01 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1152 ms, new=1278 ms, diff=1.11 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 600 ms, new= 539 ms, diff=1.11 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 897 ms, new=1168 ms, diff=1.30 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1390 ms, new=1492 ms, diff=1.07 slower ✖, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 304 ms, new= 276 ms, diff=1.10 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=3552.0, new=3552.0, diff=no change (sum of per-query averages)
TOTAL: prev=45511 ms, new=59958 ms, diff=1.32 slower ❌

Show full query output

      q1: prev= 924 ms, new=1086 ms, diff=1.18 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
      q2: prev=1540 ms, new=1689 ms, diff=1.10 slower ✖, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2370 ms, new=2157 ms, diff=1.10 faster ✔, tasks: prev=140.0, new=140.0, diff=no change
      q4: prev=1324 ms, new=1448 ms, diff=1.09 slower ✖, tasks: prev=168.0, new=168.0, diff=no change
      q5: prev=2273 ms, new=2499 ms, diff=1.10 slower ✖, tasks: prev=243.0, new=243.0, diff=no change
      q6: prev= 737 ms, new= 965 ms, diff=1.31 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q7: prev=2381 ms, new=2513 ms, diff=1.06 slower ✖, tasks: prev=275.0, new=275.0, diff=no change
      q8: prev=3850 ms, new=4191 ms, diff=1.09 slower ✖, tasks: prev=328.0, new=328.0, diff=no change
      q9: prev=4074 ms, new=4900 ms, diff=1.20 slower ✖, tasks: prev=334.0, new=334.0, diff=no change
     q10: prev=5180 ms, new=6468 ms, diff=1.25 slower ✖, tasks: prev=191.0, new=191.0, diff=no change
     q11: prev=1461 ms, new=1155 ms, diff=1.26 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1583 ms, new=2104 ms, diff=1.33 slower ✖, tasks: prev=168.0, new=168.0, diff=no change
     q13: prev=1030 ms, new=1986 ms, diff=1.93 slower ❌, tasks: prev=58.0, new=58.0, diff=no change
     q14: prev=1312 ms, new=2130 ms, diff=1.62 slower ❌, tasks: prev=102.0, new=102.0, diff=no change
     q15: prev=2316 ms, new=8693 ms, diff=3.75 slower ❌, tasks: prev=202.0, new=202.0, diff=no change
     q16: prev= 714 ms, new= 835 ms, diff=1.17 slower ✖, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=1966 ms, new=4519 ms, diff=2.30 slower ❌, tasks: prev=152.0, new=152.0, diff=no change
     q18: prev=2585 ms, new=2906 ms, diff=1.12 slower ✖, tasks: prev=190.0, new=190.0, diff=no change
     q19: prev=1402 ms, new=1421 ms, diff=1.01 slower ✖, tasks: prev=102.0, new=102.0, diff=no change
     q20: prev=2141 ms, new=2220 ms, diff=1.04 slower ✖, tasks: prev=177.0, new=177.0, diff=no change
     q21: prev=3297 ms, new=3209 ms, diff=1.03 faster ✔, tasks: prev=320.0, new=320.0, diff=no change
     q22: prev=1051 ms, new= 864 ms, diff=1.22 faster ✔, tasks: prev=44.0, new=44.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=69743 ms, new=66882 ms, diff=1.04 faster ✔

Show full query output

      q1: prev= 242 ms, new= 308 ms, diff=1.27 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 252 ms, new= 223 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 174 ms, new= 188 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2273 ms, new=2343 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 730 ms, new= 761 ms, diff=1.04 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4659 ms, new=4695 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 287 ms, new= 258 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 187 ms, new= 321 ms, diff=1.72 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 300 ms, new= 297 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 421 ms, new= 573 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1688 ms, new=1714 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 141 ms, new= 166 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 428 ms, new= 425 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1105 ms, new=1181 ms, diff=1.07 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 316 ms, new= 237 ms, diff=1.33 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 460 ms, new= 262 ms, diff=1.76 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 561 ms, new= 438 ms, diff=1.28 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 495 ms, new= 469 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 247 ms, new= 215 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 117 ms, new= 115 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 121 ms, new= 137 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 419 ms, new= 416 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 620 ms, new= 667 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 485 ms, new= 445 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 375 ms, new= 327 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 194 ms, new= 194 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev=1046 ms, new= 981 ms, diff=1.07 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 276 ms, new= 288 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 420 ms, new= 394 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev=1015 ms, new= 903 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 218 ms, new= 188 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 253 ms, new= 238 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 218 ms, new= 209 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 403 ms, new= 403 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 667 ms, new= 523 ms, diff=1.28 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 257 ms, new= 240 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 318 ms, new= 262 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 209 ms, new= 212 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 290 ms, new= 287 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  86 ms, new=  91 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 168 ms, new= 162 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 136 ms, new= 120 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 215 ms, new= 219 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 257 ms, new= 238 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 341 ms, new= 311 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 652 ms, new= 658 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 369 ms, new= 365 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 728 ms, new= 744 ms, diff=1.02 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 346 ms, new= 363 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 384 ms, new= 493 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 174 ms, new= 157 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 157 ms, new= 152 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 474 ms, new= 351 ms, diff=1.35 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 137 ms, new= 122 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 245 ms, new= 198 ms, diff=1.24 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 405 ms, new= 393 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 412 ms, new= 459 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 284 ms, new= 264 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 230 ms, new= 200 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1478 ms, new=1098 ms, diff=1.35 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1414 ms, new=1428 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 172 ms, new= 137 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1203 ms, new=1194 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 305 ms, new= 315 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 430 ms, new= 412 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 478 ms, new= 517 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 348 ms, new= 310 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 395 ms, new= 340 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 494 ms, new= 497 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 478 ms, new= 468 ms, diff=1.02 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7858 ms, new=7498 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 382 ms, new= 210 ms, diff=1.82 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 845 ms, new= 788 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1773 ms, new=1421 ms, diff=1.25 faster ✔, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 563 ms, new= 489 ms, diff=1.15 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 857 ms, new= 704 ms, diff=1.22 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 691 ms, new= 617 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 283 ms, new= 230 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev=1028 ms, new= 961 ms, diff=1.07 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 279 ms, new= 227 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 357 ms, new= 236 ms, diff=1.51 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 425 ms, new= 294 ms, diff=1.45 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 287 ms, new= 199 ms, diff=1.44 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 420 ms, new= 448 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 117 ms, new= 115 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 298 ms, new= 292 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 377 ms, new= 308 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 171 ms, new= 176 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 140 ms, new= 146 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 325 ms, new= 312 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 189 ms, new= 199 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 260 ms, new= 236 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 257 ms, new= 239 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 327 ms, new= 321 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 143 ms, new= 147 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 201 ms, new= 195 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 150 ms, new= 154 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14458 ms, new=14441 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 207 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`d7e27c9607d4f53c0d77daf1431bd2bffd99f076`	`3c6115c196ac7423f0272403ebfdf789ada4ed8c`

Phase	Base	PR head
Build and deployment	2m 5s	1m 38s
All benchmarks	14m 31s	15m 50s
Benchmark `tpch/sf10`	1m 56s	2m 3s
Benchmark `tpch/sf100`	5m 10s	6m 43s
Benchmark `tpcds/sf1`	7m 25s	7m 4s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 50 c5n.4xlarge nodes for both revisions

Other timings: Queue 31m 30s · Dataset validation 1s · Total 34m 18s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-24T20:09:51Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gene-bordegaray · 2026-09-24T20:10:05Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1 --nodes 50

gabot-0 · 2026-09-24T20:10:16Z

Requested by this comment.

Benchmark results

Compared: PR base c24d1612738b → PR head b7f42b6c6a44 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=17044 ms, new=15842 ms, diff=1.08 faster ✔

Show full query output

      q1: prev= 541 ms, new= 520 ms, diff=1.04 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 717 ms, new= 650 ms, diff=1.10 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 797 ms, new= 771 ms, diff=1.03 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 389 ms, new= 433 ms, diff=1.11 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 831 ms, new= 825 ms, diff=1.01 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 348 ms, new= 283 ms, diff=1.23 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev=1132 ms, new= 860 ms, diff=1.32 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1438 ms, new=1169 ms, diff=1.23 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1293 ms, new=1227 ms, diff=1.05 faster ✔, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1233 ms, new=1086 ms, diff=1.14 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 575 ms, new= 477 ms, diff=1.21 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 694 ms, new= 431 ms, diff=1.61 faster ✅, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 659 ms, new= 553 ms, diff=1.19 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 430 ms, new= 587 ms, diff=1.37 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 959 ms, new= 763 ms, diff=1.26 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 366 ms, new= 350 ms, diff=1.05 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 756 ms, new= 790 ms, diff=1.04 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1183 ms, new=1071 ms, diff=1.10 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 444 ms, new= 628 ms, diff=1.41 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 750 ms, new= 932 ms, diff=1.24 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1238 ms, new=1177 ms, diff=1.05 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 271 ms, new= 259 ms, diff=1.05 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=50023 ms, new=50727 ms, diff=1.01 slower ✖

Show full query output

      q1: prev=1729 ms, new=1724 ms, diff=1.00 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1345 ms, new=1263 ms, diff=1.06 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2323 ms, new=2326 ms, diff=1.00 slower ✖, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1055 ms, new=1211 ms, diff=1.15 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2663 ms, new=2767 ms, diff=1.04 slower ✖, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev= 977 ms, new= 927 ms, diff=1.05 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=3072 ms, new=2931 ms, diff=1.05 faster ✔, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3541 ms, new=3339 ms, diff=1.06 faster ✔, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4240 ms, new=4246 ms, diff=1.00 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4777 ms, new=4319 ms, diff=1.11 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev= 831 ms, new=1060 ms, diff=1.28 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1503 ms, new=1460 ms, diff=1.03 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev= 965 ms, new=1064 ms, diff=1.10 slower ✖, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1440 ms, new=1497 ms, diff=1.04 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2784 ms, new=4271 ms, diff=1.53 slower ❌, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 779 ms, new= 711 ms, diff=1.10 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3221 ms, new=3185 ms, diff=1.01 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3674 ms, new=3433 ms, diff=1.07 faster ✔, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1558 ms, new=1464 ms, diff=1.06 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=1976 ms, new=1965 ms, diff=1.01 faster ✔, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4868 ms, new=4827 ms, diff=1.01 faster ✔, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 702 ms, new= 737 ms, diff=1.05 slower ✖, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=65220 ms, new=64981 ms, diff=1.00 faster ✔

Show full query output

      q1: prev= 349 ms, new= 255 ms, diff=1.37 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 394 ms, new= 261 ms, diff=1.51 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 188 ms, new= 168 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2288 ms, new=2210 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 573 ms, new= 625 ms, diff=1.09 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4578 ms, new=4574 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 254 ms, new= 233 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 221 ms, new= 255 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 314 ms, new= 291 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 419 ms, new= 576 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1621 ms, new=1704 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 112 ms, new= 135 ms, diff=1.21 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 407 ms, new= 426 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev= 930 ms, new=1276 ms, diff=1.37 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 260 ms, new= 545 ms, diff=2.10 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 250 ms, new= 576 ms, diff=2.30 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 369 ms, new= 518 ms, diff=1.40 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 384 ms, new= 741 ms, diff=1.93 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 219 ms, new= 236 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 104 ms, new= 111 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 115 ms, new= 130 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 428 ms, new= 391 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 691 ms, new= 587 ms, diff=1.18 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 541 ms, new= 456 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 480 ms, new= 332 ms, diff=1.45 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 210 ms, new= 217 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 841 ms, new= 841 ms, diff=1.00 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 267 ms, new= 321 ms, diff=1.20 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 362 ms, new= 411 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev= 716 ms, new= 724 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 190 ms, new= 170 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 175 ms, new= 211 ms, diff=1.21 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 195 ms, new= 181 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 431 ms, new= 360 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 458 ms, new= 430 ms, diff=1.07 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 260 ms, new= 273 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 284 ms, new= 283 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 257 ms, new= 223 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 273 ms, new= 268 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  73 ms, new=  76 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 151 ms, new= 149 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 144 ms, new= 120 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 212 ms, new= 202 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 269 ms, new= 236 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 394 ms, new= 285 ms, diff=1.38 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 644 ms, new= 606 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 367 ms, new= 338 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 637 ms, new= 543 ms, diff=1.17 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 373 ms, new= 325 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 354 ms, new= 325 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 160 ms, new= 150 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 139 ms, new= 148 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 387 ms, new= 340 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 157 ms, new= 132 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 213 ms, new= 188 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 390 ms, new= 384 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 437 ms, new= 399 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 353 ms, new= 262 ms, diff=1.35 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 226 ms, new= 180 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1207 ms, new=1052 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1434 ms, new=1381 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 178 ms, new= 148 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1022 ms, new= 944 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 280 ms, new= 272 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 374 ms, new= 375 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 488 ms, new= 478 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 311 ms, new= 331 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 371 ms, new= 350 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 473 ms, new= 477 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 324 ms, new= 422 ms, diff=1.30 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7165 ms, new=7245 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 191 ms, new= 217 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 736 ms, new= 782 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1091 ms, new= 945 ms, diff=1.15 faster ✔, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 461 ms, new= 352 ms, diff=1.31 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 538 ms, new= 529 ms, diff=1.02 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 609 ms, new= 890 ms, diff=1.46 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 250 ms, new= 342 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 794 ms, new= 666 ms, diff=1.19 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 247 ms, new= 400 ms, diff=1.62 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 277 ms, new= 272 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 266 ms, new= 269 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 239 ms, new= 212 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 508 ms, new= 407 ms, diff=1.25 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 237 ms, new= 125 ms, diff=1.90 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 512 ms, new= 248 ms, diff=2.06 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 390 ms, new= 324 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 198 ms, new= 184 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 156 ms, new= 156 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 391 ms, new= 363 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 193 ms, new= 211 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 215 ms, new= 242 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 243 ms, new= 228 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 412 ms, new= 271 ms, diff=1.52 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 151 ms, new= 153 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 190 ms, new= 185 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 147 ms, new= 161 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14463 ms, new=14459 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 210 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`c24d1612738bb279257d7767e464b9332eecd7af`	`b7f42b6c6a444e104a52df23d663ae1956e8ecc4`

Phase	Base	PR head
Build and deployment	5m 24s	1m 36s
All benchmarks	14m 2s	13m 58s
Benchmark `tpch/sf10`	1m 55s	1m 48s
Benchmark `tpch/sf100`	5m 14s	5m 21s
Benchmark `tpcds/sf1`	6m 54s	6m 49s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 2s · Dataset validation 1s · Total 35m 13s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gabot-0 · 2026-09-24T20:10:17Z

Requested by this comment.

Benchmark results

Compared: PR base c24d1612738b → PR head b7f42b6c6a44 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=16676 ms, new=17397 ms, diff=1.04 slower ✖

Show full query output

      q1: prev= 639 ms, new= 598 ms, diff=1.07 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 855 ms, new= 872 ms, diff=1.02 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 832 ms, new= 798 ms, diff=1.04 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 589 ms, new= 560 ms, diff=1.05 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev=1000 ms, new= 956 ms, diff=1.05 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 327 ms, new= 329 ms, diff=1.01 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 928 ms, new= 932 ms, diff=1.00 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1186 ms, new=1249 ms, diff=1.05 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1223 ms, new=1480 ms, diff=1.21 slower ✖, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1191 ms, new=1219 ms, diff=1.02 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 662 ms, new= 601 ms, diff=1.10 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 542 ms, new= 533 ms, diff=1.02 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 537 ms, new= 622 ms, diff=1.16 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 391 ms, new= 520 ms, diff=1.33 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 731 ms, new= 807 ms, diff=1.10 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 416 ms, new= 463 ms, diff=1.11 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 764 ms, new= 750 ms, diff=1.02 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1061 ms, new=1025 ms, diff=1.04 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 435 ms, new= 455 ms, diff=1.05 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 774 ms, new= 774 ms, diff=1.00 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1307 ms, new=1541 ms, diff=1.18 slower ✖, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 286 ms, new= 313 ms, diff=1.09 slower ✖, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=3552.0, new=3552.0, diff=no change (sum of per-query averages)
TOTAL: prev=40282 ms, new=43662 ms, diff=1.08 slower ✖

Show full query output

      q1: prev= 867 ms, new=1113 ms, diff=1.28 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
      q2: prev=2210 ms, new=1462 ms, diff=1.51 faster ✅, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=1995 ms, new=2060 ms, diff=1.03 slower ✖, tasks: prev=140.0, new=140.0, diff=no change
      q4: prev=1269 ms, new=1185 ms, diff=1.07 faster ✔, tasks: prev=168.0, new=168.0, diff=no change
      q5: prev=2085 ms, new=1986 ms, diff=1.05 faster ✔, tasks: prev=243.0, new=243.0, diff=no change
      q6: prev= 778 ms, new=1079 ms, diff=1.39 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q7: prev=2206 ms, new=2372 ms, diff=1.08 slower ✖, tasks: prev=275.0, new=275.0, diff=no change
      q8: prev=2955 ms, new=2819 ms, diff=1.05 faster ✔, tasks: prev=328.0, new=328.0, diff=no change
      q9: prev=3060 ms, new=3172 ms, diff=1.04 slower ✖, tasks: prev=334.0, new=334.0, diff=no change
     q10: prev=4526 ms, new=5080 ms, diff=1.12 slower ✖, tasks: prev=191.0, new=191.0, diff=no change
     q11: prev= 905 ms, new=1040 ms, diff=1.15 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1413 ms, new=1420 ms, diff=1.00 slower ✖, tasks: prev=168.0, new=168.0, diff=no change
     q13: prev=1178 ms, new=1091 ms, diff=1.08 faster ✔, tasks: prev=58.0, new=58.0, diff=no change
     q14: prev=1445 ms, new=1641 ms, diff=1.14 slower ✖, tasks: prev=102.0, new=102.0, diff=no change
     q15: prev=1967 ms, new=3778 ms, diff=1.92 slower ❌, tasks: prev=202.0, new=202.0, diff=no change
     q16: prev= 882 ms, new= 686 ms, diff=1.29 faster ✔, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=1684 ms, new=1661 ms, diff=1.01 faster ✔, tasks: prev=152.0, new=152.0, diff=no change
     q18: prev=2290 ms, new=2347 ms, diff=1.02 slower ✖, tasks: prev=190.0, new=190.0, diff=no change
     q19: prev=1127 ms, new=1276 ms, diff=1.13 slower ✖, tasks: prev=102.0, new=102.0, diff=no change
     q20: prev=1867 ms, new=2250 ms, diff=1.21 slower ✖, tasks: prev=177.0, new=177.0, diff=no change
     q21: prev=2871 ms, new=3284 ms, diff=1.14 slower ✖, tasks: prev=320.0, new=320.0, diff=no change
     q22: prev= 702 ms, new= 860 ms, diff=1.23 slower ✖, tasks: prev=44.0, new=44.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=65831 ms, new=65651 ms, diff=1.00 faster ✔

Show full query output

      q1: prev= 230 ms, new= 242 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 221 ms, new= 250 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 154 ms, new= 166 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2263 ms, new=2258 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 635 ms, new= 842 ms, diff=1.33 slower ✖, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4630 ms, new=4647 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 238 ms, new= 253 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 202 ms, new= 187 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 297 ms, new= 303 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 360 ms, new= 370 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1620 ms, new=1581 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 111 ms, new= 121 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 396 ms, new= 413 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1033 ms, new= 971 ms, diff=1.06 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 329 ms, new= 214 ms, diff=1.54 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 384 ms, new= 220 ms, diff=1.75 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 626 ms, new= 313 ms, diff=2.00 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 592 ms, new= 377 ms, diff=1.57 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 236 ms, new= 214 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 147 ms, new= 102 ms, diff=1.44 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 170 ms, new= 114 ms, diff=1.49 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 538 ms, new= 390 ms, diff=1.38 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 615 ms, new= 626 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 456 ms, new= 452 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 348 ms, new= 366 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 209 ms, new= 210 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 867 ms, new= 974 ms, diff=1.12 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 167 ms, new= 155 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 345 ms, new= 334 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev= 660 ms, new= 654 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 163 ms, new= 200 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 178 ms, new= 192 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 179 ms, new= 189 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 353 ms, new= 381 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 672 ms, new= 751 ms, diff=1.12 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 229 ms, new= 490 ms, diff=2.14 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 272 ms, new= 456 ms, diff=1.68 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 229 ms, new= 294 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 355 ms, new= 327 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  80 ms, new= 103 ms, diff=1.29 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 154 ms, new= 154 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 142 ms, new= 127 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 188 ms, new= 202 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 247 ms, new= 253 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 324 ms, new= 334 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 617 ms, new= 639 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 320 ms, new= 347 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 720 ms, new= 750 ms, diff=1.04 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 341 ms, new= 333 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 326 ms, new= 334 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 149 ms, new= 152 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 133 ms, new= 141 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 332 ms, new= 357 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 116 ms, new= 121 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 184 ms, new= 214 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 385 ms, new= 376 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 405 ms, new= 420 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 261 ms, new= 249 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 174 ms, new= 189 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1148 ms, new=1046 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1430 ms, new=1354 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 152 ms, new= 194 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1173 ms, new=1180 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 302 ms, new= 296 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 428 ms, new= 401 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 555 ms, new= 478 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 399 ms, new= 333 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 377 ms, new= 364 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 504 ms, new= 454 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 495 ms, new= 442 ms, diff=1.12 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7722 ms, new=7130 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 183 ms, new= 244 ms, diff=1.33 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 754 ms, new= 813 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1274 ms, new=1575 ms, diff=1.24 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 507 ms, new= 484 ms, diff=1.05 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 786 ms, new= 787 ms, diff=1.00 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 576 ms, new= 659 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 203 ms, new= 249 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 887 ms, new= 949 ms, diff=1.07 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 172 ms, new= 218 ms, diff=1.27 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 252 ms, new= 239 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 262 ms, new= 291 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 197 ms, new= 208 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 361 ms, new= 374 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 118 ms, new= 120 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 239 ms, new= 263 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 406 ms, new= 396 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 168 ms, new= 165 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 133 ms, new= 165 ms, diff=1.24 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 374 ms, new= 428 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 187 ms, new= 300 ms, diff=1.60 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 243 ms, new= 198 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 217 ms, new= 232 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 287 ms, new= 297 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 158 ms, new= 147 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 211 ms, new= 189 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 143 ms, new= 157 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14541 ms, new=14438 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 211 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`c24d1612738bb279257d7767e464b9332eecd7af`	`b7f42b6c6a444e104a52df23d663ae1956e8ecc4`

Phase	Base	PR head
Build and deployment	2m 54s	1m 41s
All benchmarks	13m 21s	13m 54s
Benchmark `tpch/sf10`	1m 52s	1m 55s
Benchmark `tpch/sf100`	4m 33s	5m 4s
Benchmark `tpcds/sf1`	6m 56s	6m 55s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 50 c5n.4xlarge nodes for both revisions

Other timings: Queue 35m 15s · Dataset validation 1s · Total 32m 2s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-24T21:21:00Z

its hard to tell what i real regression and not. It seems like tpch q15 genuinely is

gene-bordegaray · 2026-09-25T00:33:03Z

ok I think it was regressing because it was retaining memory again on each tail squence incrememnt which acquired a lock. Back to pretty much my initial design. I think it is quite nice, but lets see how this goes

gene-bordegaray · 2026-09-25T00:34:17Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1

gabot-0 · 2026-09-25T00:34:24Z

Requested by this comment.

Benchmark results

Compared: PR base 588a07f78b9d → PR head f05ad4351d93 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=518.2, diff=0.8 fewer (0.2%) (sum of per-query averages)
TOTAL: prev=15970 ms, new=16200 ms, diff=1.01 slower ✖

Show full query output

      q1: prev= 425 ms, new= 464 ms, diff=1.09 slower ✖, tasks: prev=14.0, new=13.2, diff=0.8 fewer (5.7%)
      q2: prev= 767 ms, new= 636 ms, diff=1.21 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 850 ms, new= 723 ms, diff=1.18 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 459 ms, new= 588 ms, diff=1.28 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 994 ms, new= 822 ms, diff=1.21 faster ✔, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 399 ms, new= 282 ms, diff=1.41 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev=1172 ms, new= 885 ms, diff=1.32 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1090 ms, new=1665 ms, diff=1.53 slower ❌, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1221 ms, new=1514 ms, diff=1.24 slower ✖, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1054 ms, new=1240 ms, diff=1.18 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 477 ms, new= 514 ms, diff=1.08 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 423 ms, new= 478 ms, diff=1.13 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 555 ms, new= 580 ms, diff=1.05 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 383 ms, new= 448 ms, diff=1.17 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 739 ms, new= 560 ms, diff=1.32 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 327 ms, new= 437 ms, diff=1.34 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 807 ms, new= 698 ms, diff=1.16 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1083 ms, new=1155 ms, diff=1.07 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 513 ms, new= 467 ms, diff=1.10 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 696 ms, new= 697 ms, diff=1.00 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1239 ms, new=1085 ms, diff=1.14 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 297 ms, new= 262 ms, diff=1.13 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=1234.0, new=1234.0, diff=no change (sum of per-query averages)
TOTAL: prev=48459 ms, new=47181 ms, diff=1.03 faster ✔

Show full query output

      q1: prev=1833 ms, new=1825 ms, diff=1.00 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
      q2: prev=1489 ms, new=1355 ms, diff=1.10 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2096 ms, new=2054 ms, diff=1.02 faster ✔, tasks: prev=52.0, new=52.0, diff=no change
      q4: prev=1043 ms, new=1032 ms, diff=1.01 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
      q5: prev=2755 ms, new=2607 ms, diff=1.06 faster ✔, tasks: prev=79.0, new=79.0, diff=no change
      q6: prev= 931 ms, new= 934 ms, diff=1.00 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
      q7: prev=2960 ms, new=2841 ms, diff=1.04 faster ✔, tasks: prev=79.0, new=79.0, diff=no change
      q8: prev=3311 ms, new=3319 ms, diff=1.00 slower ✖, tasks: prev=94.0, new=94.0, diff=no change
      q9: prev=4292 ms, new=4464 ms, diff=1.04 slower ✖, tasks: prev=100.0, new=100.0, diff=no change
     q10: prev=4416 ms, new=4266 ms, diff=1.04 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q11: prev= 861 ms, new= 791 ms, diff=1.09 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1303 ms, new=1235 ms, diff=1.06 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
     q13: prev= 969 ms, new= 920 ms, diff=1.05 faster ✔, tasks: prev=40.0, new=40.0, diff=no change
     q14: prev=1174 ms, new=1193 ms, diff=1.02 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q15: prev=2706 ms, new=2755 ms, diff=1.02 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
     q16: prev= 663 ms, new= 762 ms, diff=1.15 slower ✖, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=3228 ms, new=3119 ms, diff=1.03 faster ✔, tasks: prev=38.0, new=38.0, diff=no change
     q18: prev=3813 ms, new=3327 ms, diff=1.15 faster ✔, tasks: prev=64.0, new=64.0, diff=no change
     q19: prev=1394 ms, new=1453 ms, diff=1.04 slower ✖, tasks: prev=26.0, new=26.0, diff=no change
     q20: prev=1933 ms, new=1828 ms, diff=1.06 faster ✔, tasks: prev=63.0, new=63.0, diff=no change
     q21: prev=4556 ms, new=4488 ms, diff=1.02 faster ✔, tasks: prev=86.0, new=86.0, diff=no change
     q22: prev= 733 ms, new= 613 ms, diff=1.20 faster ✔, tasks: prev=32.0, new=32.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=69142 ms, new=67967 ms, diff=1.02 faster ✔

Show full query output

      q1: prev= 338 ms, new= 219 ms, diff=1.54 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 306 ms, new= 208 ms, diff=1.47 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 174 ms, new= 156 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2193 ms, new=2260 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 671 ms, new= 573 ms, diff=1.17 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4739 ms, new=4796 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 248 ms, new= 240 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 401 ms, new= 235 ms, diff=1.71 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 328 ms, new= 300 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev= 756 ms, new= 414 ms, diff=1.83 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1586 ms, new=1601 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 302 ms, new= 108 ms, diff=2.80 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 782 ms, new= 440 ms, diff=1.78 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1033 ms, new= 860 ms, diff=1.20 faster ✔, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 317 ms, new= 205 ms, diff=1.55 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 473 ms, new= 219 ms, diff=2.16 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 315 ms, new= 289 ms, diff=1.09 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 437 ms, new= 432 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 250 ms, new= 175 ms, diff=1.43 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 109 ms, new= 105 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 151 ms, new= 123 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 384 ms, new= 398 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 611 ms, new= 592 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 640 ms, new= 612 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 339 ms, new= 323 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 206 ms, new= 195 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev= 829 ms, new= 790 ms, diff=1.05 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 193 ms, new= 282 ms, diff=1.46 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 410 ms, new= 326 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev= 985 ms, new= 722 ms, diff=1.36 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 216 ms, new= 139 ms, diff=1.55 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 403 ms, new= 215 ms, diff=1.87 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 236 ms, new= 176 ms, diff=1.34 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 546 ms, new= 446 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 405 ms, new= 398 ms, diff=1.02 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 244 ms, new= 239 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 240 ms, new= 252 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 236 ms, new= 232 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 263 ms, new= 280 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  78 ms, new=  86 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 167 ms, new= 158 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 131 ms, new= 123 ms, diff=1.07 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 200 ms, new= 185 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 379 ms, new= 374 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 527 ms, new= 547 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 596 ms, new= 633 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 404 ms, new= 615 ms, diff=1.52 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 609 ms, new= 576 ms, diff=1.06 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 324 ms, new= 328 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 325 ms, new= 347 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 177 ms, new= 206 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 135 ms, new= 152 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 427 ms, new= 460 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 127 ms, new= 138 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 311 ms, new= 294 ms, diff=1.06 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 369 ms, new= 388 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 408 ms, new= 481 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 250 ms, new= 275 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 223 ms, new= 331 ms, diff=1.48 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1106 ms, new=1228 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1344 ms, new=1402 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 150 ms, new= 149 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1347 ms, new=1730 ms, diff=1.28 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 309 ms, new= 424 ms, diff=1.37 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 412 ms, new= 458 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 489 ms, new= 641 ms, diff=1.31 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 629 ms, new= 685 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 535 ms, new= 563 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 489 ms, new= 529 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 370 ms, new= 316 ms, diff=1.17 faster ✔, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7245 ms, new=7286 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 278 ms, new= 181 ms, diff=1.54 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 780 ms, new= 898 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1113 ms, new=1291 ms, diff=1.16 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 337 ms, new= 368 ms, diff=1.09 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 594 ms, new= 594 ms, diff=1.00 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 839 ms, new= 615 ms, diff=1.36 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 396 ms, new= 224 ms, diff=1.77 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev= 838 ms, new= 748 ms, diff=1.12 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 781 ms, new= 436 ms, diff=1.79 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 712 ms, new= 274 ms, diff=2.60 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 281 ms, new= 271 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 377 ms, new= 408 ms, diff=1.08 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 524 ms, new= 585 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 156 ms, new= 131 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 261 ms, new= 303 ms, diff=1.16 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 294 ms, new= 390 ms, diff=1.33 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 172 ms, new= 198 ms, diff=1.15 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 136 ms, new= 211 ms, diff=1.55 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 379 ms, new= 643 ms, diff=1.70 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 215 ms, new= 217 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 200 ms, new= 248 ms, diff=1.24 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 363 ms, new= 491 ms, diff=1.35 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 326 ms, new= 450 ms, diff=1.38 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 152 ms, new= 154 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 184 ms, new= 201 ms, diff=1.09 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 168 ms, new= 147 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14399 ms, new=14607 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 212 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`588a07f78b9d59fbf6c93d4fbcacefebed209f1e`	`f05ad4351d93609a1289ce29e6214a38b123441b`

Phase	Base	PR head
Build and deployment	1m 52s	1m 34s
All benchmarks	14m 12s	13m 48s
Benchmark `tpch/sf10`	1m 46s	1m 46s
Benchmark `tpch/sf100`	5m 9s	4m 58s
Benchmark `tpcds/sf1`	7m 17s	7m 3s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 12 c5n.4xlarge nodes for both revisions

Other timings: Queue 1s · Dataset validation 1s · Total 31m 39s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gene-bordegaray · 2026-09-25T00:34:26Z

benchmarks run tpch/sf10 tpch/sf100 tpcds/sf1 --nodes 50

gene-bordegaray · 2026-09-25T00:35:14Z

the diagrams might be whats scary, I can work on those

gabot-0 · 2026-09-25T01:06:36Z

Requested by this comment.

Benchmark results

Compared: PR base 588a07f78b9d → PR head f05ad4351d93 · View exact source diff

=== Comparing tpch/sf10 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=519.0, new=519.0, diff=no change (sum of per-query averages)
TOTAL: prev=17431 ms, new=17217 ms, diff=1.01 faster ✔

Show full query output

      q1: prev= 592 ms, new= 483 ms, diff=1.23 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q2: prev= 813 ms, new= 755 ms, diff=1.08 faster ✔, tasks: prev=19.0, new=19.0, diff=no change
      q3: prev= 882 ms, new=1025 ms, diff=1.16 slower ✖, tasks: prev=19.0, new=19.0, diff=no change
      q4: prev= 537 ms, new= 570 ms, diff=1.06 slower ✖, tasks: prev=23.0, new=23.0, diff=no change
      q5: prev= 939 ms, new= 962 ms, diff=1.02 slower ✖, tasks: prev=36.0, new=36.0, diff=no change
      q6: prev= 269 ms, new= 247 ms, diff=1.09 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
      q7: prev= 970 ms, new= 861 ms, diff=1.13 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
      q8: prev=1106 ms, new=1387 ms, diff=1.25 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
      q9: prev=1318 ms, new=1432 ms, diff=1.09 slower ✖, tasks: prev=49.0, new=49.0, diff=no change
     q10: prev=1229 ms, new=1222 ms, diff=1.01 faster ✔, tasks: prev=27.0, new=27.0, diff=no change
     q11: prev= 661 ms, new= 687 ms, diff=1.04 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q12: prev= 657 ms, new= 609 ms, diff=1.08 faster ✔, tasks: prev=23.0, new=23.0, diff=no change
     q13: prev= 601 ms, new= 607 ms, diff=1.01 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
     q14: prev= 474 ms, new= 444 ms, diff=1.07 faster ✔, tasks: prev=15.0, new=15.0, diff=no change
     q15: prev= 755 ms, new= 756 ms, diff=1.00 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
     q16: prev= 395 ms, new= 446 ms, diff=1.13 slower ✖, tasks: prev=10.0, new=10.0, diff=no change
     q17: prev= 910 ms, new= 756 ms, diff=1.20 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
     q18: prev=1164 ms, new=1043 ms, diff=1.12 faster ✔, tasks: prev=26.0, new=26.0, diff=no change
     q19: prev= 513 ms, new= 560 ms, diff=1.09 slower ✖, tasks: prev=15.0, new=15.0, diff=no change
     q20: prev= 781 ms, new= 870 ms, diff=1.11 slower ✖, tasks: prev=27.0, new=27.0, diff=no change
     q21: prev=1485 ms, new=1209 ms, diff=1.23 faster ✔, tasks: prev=39.0, new=39.0, diff=no change
     q22: prev= 380 ms, new= 286 ms, diff=1.33 faster ✔, tasks: prev=5.0, new=5.0, diff=no change

=== Comparing tpch/sf100 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=3552.0, new=3552.0, diff=no change (sum of per-query averages)
TOTAL: prev=42159 ms, new=39533 ms, diff=1.07 faster ✔

Show full query output

      q1: prev=1118 ms, new= 945 ms, diff=1.18 faster ✔, tasks: prev=100.0, new=100.0, diff=no change
      q2: prev=1639 ms, new=1534 ms, diff=1.07 faster ✔, tasks: prev=92.0, new=92.0, diff=no change
      q3: prev=2263 ms, new=1990 ms, diff=1.14 faster ✔, tasks: prev=140.0, new=140.0, diff=no change
      q4: prev=1261 ms, new=1061 ms, diff=1.19 faster ✔, tasks: prev=168.0, new=168.0, diff=no change
      q5: prev=2019 ms, new=2030 ms, diff=1.01 slower ✖, tasks: prev=243.0, new=243.0, diff=no change
      q6: prev= 739 ms, new= 610 ms, diff=1.21 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
      q7: prev=2443 ms, new=2187 ms, diff=1.12 faster ✔, tasks: prev=275.0, new=275.0, diff=no change
      q8: prev=3004 ms, new=2847 ms, diff=1.06 faster ✔, tasks: prev=328.0, new=328.0, diff=no change
      q9: prev=3282 ms, new=2994 ms, diff=1.10 faster ✔, tasks: prev=334.0, new=334.0, diff=no change
     q10: prev=4900 ms, new=4471 ms, diff=1.10 faster ✔, tasks: prev=191.0, new=191.0, diff=no change
     q11: prev= 919 ms, new=1001 ms, diff=1.09 slower ✖, tasks: prev=65.0, new=65.0, diff=no change
     q12: prev=1381 ms, new=1257 ms, diff=1.10 faster ✔, tasks: prev=168.0, new=168.0, diff=no change
     q13: prev= 903 ms, new= 909 ms, diff=1.01 slower ✖, tasks: prev=58.0, new=58.0, diff=no change
     q14: prev=1174 ms, new=1294 ms, diff=1.10 slower ✖, tasks: prev=102.0, new=102.0, diff=no change
     q15: prev=2792 ms, new=2137 ms, diff=1.31 faster ✔, tasks: prev=202.0, new=202.0, diff=no change
     q16: prev= 691 ms, new= 792 ms, diff=1.15 slower ✖, tasks: prev=51.0, new=51.0, diff=no change
     q17: prev=1804 ms, new=1768 ms, diff=1.02 faster ✔, tasks: prev=152.0, new=152.0, diff=no change
     q18: prev=2368 ms, new=2369 ms, diff=1.00 slower ✖, tasks: prev=190.0, new=190.0, diff=no change
     q19: prev=1370 ms, new=1217 ms, diff=1.13 faster ✔, tasks: prev=102.0, new=102.0, diff=no change
     q20: prev=2221 ms, new=1921 ms, diff=1.16 faster ✔, tasks: prev=177.0, new=177.0, diff=no change
     q21: prev=3091 ms, new=3223 ms, diff=1.04 slower ✖, tasks: prev=320.0, new=320.0, diff=no change
     q22: prev= 777 ms, new= 976 ms, diff=1.26 slower ✖, tasks: prev=44.0, new=44.0, diff=no change

=== Comparing tpcds/sf1 results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] ===
TASKS: prev=90.0, new=90.0, diff=no change (sum of per-query averages)
TOTAL: prev=72972 ms, new=69429 ms, diff=1.05 faster ✔

Show full query output

      q1: prev= 225 ms, new= 469 ms, diff=2.08 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
      q2: prev= 225 ms, new= 252 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q3: prev= 162 ms, new= 182 ms, diff=1.12 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q4: prev=2251 ms, new=2339 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q5: prev= 924 ms, new= 636 ms, diff=1.45 faster ✔, tasks: prev=14.0, new=14.0, diff=no change
      q6: prev=4915 ms, new=4858 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q7: prev= 278 ms, new= 318 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
      q8: prev= 552 ms, new= 381 ms, diff=1.45 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
      q9: prev= 305 ms, new= 290 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q10: prev=1117 ms, new= 530 ms, diff=2.11 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q11: prev=1573 ms, new=1658 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q12: prev= 155 ms, new= 120 ms, diff=1.29 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q13: prev= 903 ms, new= 411 ms, diff=2.20 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q14: prev=1055 ms, new=1061 ms, diff=1.01 slower ✖, tasks: prev=9.0, new=9.0, diff=no change
     q15: prev= 417 ms, new= 272 ms, diff=1.53 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q16: prev= 419 ms, new= 298 ms, diff=1.41 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q17: prev= 331 ms, new= 322 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q18: prev= 705 ms, new= 446 ms, diff=1.58 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q19: prev= 411 ms, new= 212 ms, diff=1.94 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q20: prev= 114 ms, new= 115 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q21: prev= 138 ms, new= 120 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q22: prev= 399 ms, new= 405 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q23: prev= 596 ms, new= 587 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q24: prev= 962 ms, new= 753 ms, diff=1.28 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q25: prev= 364 ms, new= 318 ms, diff=1.14 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q26: prev= 387 ms, new= 191 ms, diff=2.03 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q27: prev=1025 ms, new=1018 ms, diff=1.01 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
     q28: prev= 272 ms, new= 284 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q29: prev= 316 ms, new= 544 ms, diff=1.72 slower ❌, tasks: prev=0.0, new=0.0, diff=no change
q30: Previously failed, and now also failed ❌: Query failed: 500 Schema error: No field named c_last_review_date_sk. Did you mean 'customer.c_last_review_date'? Valid fields are ctr1.ctr_customer_sk, ctr1.ctr_state, ctr1.ctr_total_return, customer_address.ca_address_sk, customer_address.ca_address_id, customer_address.ca_street_number, customer…
     q31: prev= 890 ms, new= 772 ms, diff=1.15 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q32: prev= 157 ms, new= 186 ms, diff=1.18 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q33: prev= 323 ms, new= 368 ms, diff=1.14 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q34: prev= 200 ms, new= 240 ms, diff=1.20 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q35: prev= 658 ms, new= 704 ms, diff=1.07 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q36: prev= 671 ms, new= 639 ms, diff=1.05 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
     q37: prev= 254 ms, new= 263 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q38: prev= 271 ms, new= 239 ms, diff=1.13 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q39: prev= 208 ms, new= 230 ms, diff=1.11 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q40: prev= 287 ms, new= 354 ms, diff=1.23 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q41: prev=  81 ms, new=  77 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q42: prev= 175 ms, new= 182 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q43: prev= 120 ms, new= 122 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q44: prev= 205 ms, new= 196 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q45: prev= 408 ms, new= 398 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q46: prev= 789 ms, new= 558 ms, diff=1.41 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q47: prev= 620 ms, new= 638 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q48: prev= 428 ms, new= 499 ms, diff=1.17 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q49: prev= 706 ms, new= 710 ms, diff=1.01 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q50: prev= 345 ms, new= 320 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q51: prev= 622 ms, new= 331 ms, diff=1.88 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q52: prev= 190 ms, new= 171 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q53: prev= 188 ms, new= 151 ms, diff=1.25 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q54: prev= 751 ms, new= 517 ms, diff=1.45 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q55: prev= 137 ms, new= 122 ms, diff=1.12 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q56: prev= 416 ms, new= 327 ms, diff=1.27 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q57: prev= 392 ms, new= 385 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q58: prev= 434 ms, new= 432 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q59: prev= 290 ms, new= 277 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q60: prev= 410 ms, new= 335 ms, diff=1.22 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q61: prev=1182 ms, new=1180 ms, diff=1.00 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q62: prev=1349 ms, new=1372 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q63: prev= 158 ms, new= 150 ms, diff=1.05 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q64: prev=1359 ms, new=1344 ms, diff=1.01 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q65: prev= 275 ms, new= 311 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q66: prev= 414 ms, new= 561 ms, diff=1.36 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q67: prev= 477 ms, new= 469 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q68: prev= 575 ms, new= 461 ms, diff=1.25 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q69: prev= 511 ms, new= 414 ms, diff=1.23 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q70: prev= 467 ms, new= 514 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q71: prev= 338 ms, new= 430 ms, diff=1.27 slower ✖, tasks: prev=8.0, new=8.0, diff=no change
     q72: prev=7088 ms, new=7475 ms, diff=1.05 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q73: prev= 183 ms, new= 178 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q74: prev= 688 ms, new= 731 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q75: prev=1319 ms, new=1572 ms, diff=1.19 slower ✖, tasks: prev=21.0, new=21.0, diff=no change
     q76: prev= 412 ms, new= 512 ms, diff=1.24 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q77: prev= 651 ms, new= 746 ms, diff=1.15 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q78: prev= 648 ms, new= 630 ms, diff=1.03 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q79: prev= 195 ms, new= 233 ms, diff=1.19 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q80: prev=1059 ms, new=1061 ms, diff=1.00 slower ✖, tasks: prev=6.0, new=6.0, diff=no change
     q81: prev= 553 ms, new= 244 ms, diff=2.27 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q82: prev= 286 ms, new= 240 ms, diff=1.19 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q83: prev= 248 ms, new= 273 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q84: prev= 594 ms, new= 267 ms, diff=2.22 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q85: prev= 839 ms, new= 469 ms, diff=1.79 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q86: prev= 137 ms, new= 114 ms, diff=1.20 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q87: prev= 292 ms, new= 241 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q88: prev= 376 ms, new= 311 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q89: prev= 163 ms, new= 168 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q90: prev= 132 ms, new= 135 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q91: prev= 470 ms, new= 376 ms, diff=1.25 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q92: prev= 209 ms, new= 190 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q93: prev= 203 ms, new= 199 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q94: prev= 350 ms, new= 397 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q95: prev= 598 ms, new= 368 ms, diff=1.63 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q96: prev= 153 ms, new= 142 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
     q97: prev= 310 ms, new= 179 ms, diff=1.73 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
     q98: prev= 145 ms, new= 153 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
     q99: prev=14444 ms, new=14486 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change

Verification and run details

Job 213 captured both immutable revisions when the request was queued. The bot fetched and checked out each full commit SHA in detached HEAD, then built and deployed the datafusion-distributed-remote-worker --bin worker target from that checkout.

Identity	PR base	PR head
Source commit	`588a07f78b9d59fbf6c93d4fbcacefebed209f1e`	`f05ad4351d93609a1289ce29e6214a38b123441b`

Phase	Base	PR head
Build and deployment	2m 0s	1m 38s
All benchmarks	14m 19s	13m 34s
Benchmark `tpch/sf10`	1m 54s	1m 53s
Benchmark `tpch/sf100`	4m 46s	4m 28s
Benchmark `tpcds/sf1`	7m 39s	7m 13s

Workload: tpch/sf10, tpch/sf100, tpcds/sf1 · all queries · 1 warmup + 5 measured iterations per query for both revisions

Capacity: 50 c5n.4xlarge nodes for both revisions

Other timings: Queue 5s · Dataset validation 1s · Total 31m 44s

How to use the benchmark bot

Post a comment whose first non-empty line is:

benchmarks run <suite>/<variant>... [--instance-type <type>] [--nodes <count>] [--iterations <count>] [--base main] [--config <key=value>]...

For example:

benchmarks run tpch/sf10 tpch/sf100 --instance-type m5.2xlarge --nodes 24 --iterations 20 --base main --config distributed.collect_dynamic_filters=false

Currently available datasets: clickbench/0-100, tpcds/sf1, tpch/sf1, tpch/sf10, tpch/sf100. Request one or more, without duplicates. The bot validates availability before provisioning and runs every query in each dataset.

Option	Default	Supported values and behavior
`--instance-type <type>`	`c5n.4xlarge`	One of the supported instance types listed below, subject to availability within the cluster's availability zones and quota.
`--nodes <count>`	`12`	An integer from 1 to 60. The deployment uses one benchmark worker per node.
`--iterations <count>`	`5`	A positive safe integer. Measured iterations per query for both revisions; warmup is excluded.
`--base main`	PR base	Compare against a snapshot of `main`; no other explicit base is supported.
`--config <key=value>`	none	Apply a safe DataFusion session setting to the PR head only. Repeat for distinct keys; spaces and shell syntax are not supported.

Supported instances and per-worker limits:

Instance type	EC2 capacity	Worker requests and limits
`c5n.2xlarge`	8 vCPU, 21 GiB	7 vCPU, 17Gi
`c5n.4xlarge`	16 vCPU, 42 GiB	15 vCPU, 38Gi
`m5.2xlarge`	8 vCPU, 32 GiB	7 vCPU, 28Gi
`m5.4xlarge`	16 vCPU, 64 GiB	15 vCPU, 60Gi
`r5.2xlarge`	8 vCPU, 64 GiB	7 vCPU, 60Gi
`r5.4xlarge`	16 vCPU, 128 GiB	15 vCPU, 124Gi

Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark.

Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions.

gabotechs

As the complexity is scoped to broadcast.rs, I trust you on whatever you consider it's a good final design, so leaving a +1 here so that you can pull this in at your own pace.

I've not made an in-depth review, but I like what I see, it looks like a thoughtful and very well documented design, so I'm in.

If you need me to give an in-depth look at any part of the changes, let me know, otherwise, happy to go with this 👍

gene-bordegaray · 2026-09-25T18:40:27Z

@gabotechs I think this is the best design from everything I have iterated and have made the docs as best I can. Could you just review the diagrams lightly to ensure others are able to comprehend them well. Thank you 🙇

gabotechs · 2026-09-28T12:26:30Z

-#[derive(Debug, Clone, Copy)]
-struct BroadcastState {
-    len: usize,
+/// Represents the queue for a single real partition, which multiple virtual partitions may share.


🤔 Reading this first, I do not really now what a "real" partition is. Aren't all partitions real?

This explanation starts by making the distinction between "real" and "virtual" partitions. I bet that if I keep reading I'll figure out what those are, but that'll force me to come back here to understand this.

If you see an option to explain this sooner that'll make it easier to navigate.

gabotechs · 2026-09-28T12:31:34Z

+///
+/// ```text
+///
+///              base_sequence             tail_sequence


In this explanation, *_sequence makes me thing that it's reference to a sequence (a concatenation of elements), however, both base_sequence and tail_sequence are referring to individual elements in a sequence.

Probably "cursor" is a more accurate word for this? unless I'm misinterpreting it.

gabotechs · 2026-09-28T12:35:26Z

+/// `e1`, so `e1` is reclaimable but remains temporarily retained until an append reaches the next
+/// `RECLAIM_INTERVAL` boundary. At that boundary, the queue removes entries before the smallest


Here, it's a bit unclear what a "next RECLAIM_INTERVAL boundary" is. I instinctively go the drawings but I don't see, so probably a preparatory explanation of what that is could help readers understand.

I finished reading the docs and I still don't know what RECLAIM_INTERVAL, we probably should explain that here.

gabotechs · 2026-09-28T12:38:14Z

+/// ```text
+/// ┌────────────────┐
+/// │                │───────────────────────┐                      ┌─────────────────────────┐
+/// │   Consumer 0   │─────────────┐         │                      │BroadcastShared          │       ┌────────────close ───────┐
+/// │                │◀───────┐    │         │    release reader    │ ┌─────────────────────┐ │       │       (acquires lock)   │
+/// └────────────────┘        │    │         └────(acquires lock)───┼▶│  Mutex(QueueState)  │◀┼───────┘                         │
+///                           │    │                                │ └─────────────────────┘ │             ┌────────────────┐  │
+///                           │    │                                │            │ notify on  │             │                │  │
+/// ┌────────────────┐        │    │                                │            ▼   close    │     ┌──────▶│    Producer    │──┘
+/// │                │        │    │   ┌───────────────┐    notify  │ ┌─────────────────────┐ │     │       │                │
+/// │   Consumer 1   │◀────notify──┼───│ Watch Channel │◀──signals──┼─│       Sender        │ │  cancel     └────────────────┘
+/// │                │     signal  │   └───────────────┘            │ └─────────────────────┘ │  signal
+/// └────────────────┘        │    │                                │ ┌─────────────────────┐ │     │
+///                           │    └────────────cancel──────────────┼▶│  CancellationToken  │─┼─────┘
+///       ...                 │                                     │ └─────────────────────┘ │
+///                           │                                     └─────────────────────────┘
+/// ┌────────────────┐        │
+/// │                │        │
+/// │   Consumer N   │◀───────┘
+/// │                │
+/// └────────────────┘
+/// ```


It's a bit weird how in this drawing the consumers are at the left, however, on the drawing above the consumers are on the right.

Unless I'm missing something, I think the mirror effect does not really contribute to the explanation, and it might be better for both drawings to follow the same left-to-right order.

gabotechs · 2026-09-28T12:41:19Z

 use tokio_stream::wrappers::WatchStream;
+use tokio_util::sync::CancellationToken;
+
+const RECLAIM_INTERVAL: usize = 8;


This 8 here looks like a magic number. Why 8? what are the units? probably some comment above the const can be helpful.

gene-bordegaray marked this pull request as ready for review September 15, 2026 20:33

gabotechs reviewed Sep 16, 2026

View reviewed changes

gene-bordegaray force-pushed the gene.bordegaray/2026/09/evict_brodcast_entries branch from 3c6115c to b7f42b6 Compare September 24, 2026 20:09

gene-bordegaray force-pushed the gene.bordegaray/2026/09/evict_brodcast_entries branch from e7113b5 to aa9a994 Compare September 24, 2026 21:25

gene-bordegaray and others added 8 commits September 24, 2026 20:18

evict broadcast entries after all consumers read

7c24cfd

get rid of the verbose RSS mode and just use a summary

f5d2eb9

take the non cleanup path out the lock

5baba46

only reclaim memory periodically

e2ce4a3

separte criterion and memory bench paths using peak recroding pool

3718cab

simplify the amount of state

bc88d3b

add some sick diagrams

3b92112

put the drop outside the lock

73dcdaa

gene-bordegaray force-pushed the gene.bordegaray/2026/09/evict_brodcast_entries branch from aa9a994 to f05ad43 Compare September 25, 2026 00:32

gabotechs approved these changes Sep 25, 2026

View reviewed changes

clean up bench

6fed81b

gene-bordegaray force-pushed the gene.bordegaray/2026/09/evict_brodcast_entries branch from f05ad43 to 6fed81b Compare September 25, 2026 18:29

clean up tests

89d3022

gabotechs reviewed Sep 28, 2026

View reviewed changes

		/// `e1`, so `e1` is reclaimable but remains temporarily retained until an append reaches the next
		/// `RECLAIM_INTERVAL` boundary. At that boundary, the queue removes entries before the smallest

Conversation

gene-bordegaray commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gabot-0 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gabot-0 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gabot-0 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gabot-0 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gabot-0 commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 14, 2026

Uh oh!

gene-bordegaray commented Sep 15, 2026

Uh oh!

gabot-0 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 15, 2026

Uh oh!

gene-bordegaray commented Sep 16, 2026

Uh oh!

gabot-0 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 16, 2026

Uh oh!

gabot-0 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gene-bordegaray commented Sep 16, 2026

Uh oh!

gabot-0 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

gabotechs Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Sep 16, 2026

gene-bordegaray commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 14, 2026 •

edited

Loading

gene-bordegaray commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 14, 2026 •

edited

Loading

gene-bordegaray commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 14, 2026 •

edited

Loading

gabot-0 commented Sep 15, 2026 •

edited

Loading

gabot-0 commented Sep 16, 2026 •

edited

Loading

gabot-0 commented Sep 16, 2026 •

edited

Loading

gabot-0 commented Sep 16, 2026 •

edited

Loading

gabot-0 commented Sep 23, 2026 •

edited

Loading

gabot-0 commented Sep 23, 2026 •

edited

Loading

gabot-0 commented Sep 24, 2026 •

edited

Loading

gabot-0 commented Sep 24, 2026 •

edited

Loading

gabot-0 commented Sep 24, 2026 •

edited

Loading

gabot-0 commented Sep 24, 2026 •

edited

Loading

gene-bordegaray commented Sep 24, 2026 •

edited

Loading

gene-bordegaray commented Sep 25, 2026 •

edited

Loading

gabot-0 commented Sep 25, 2026 •

edited

Loading

gabot-0 commented Sep 25, 2026 •

edited

Loading