TPC-H SF1000 results for Ballista main (DataFusion 55.1.0): 0.72x of Spark 3.5, 1.45x of Spark + Comet, 15% of time in planning (#2497) #2499

I ran the full TPC-H suite at SF1000 against the latest main (32ceaf8, "fix(scheduler): declare distributed EXPLAIN output columns not-null (#2494)"), which now uses DataFusion 55.1.0. I compared it with an earlier Ballista run from Sep 8 (DataFusion 55.0.0), and with plain Spark 3.5 and Spark 3.5 + DataFusion Comet main (bc4be39) on the same cluster and data.

Summary:

• Ballista main is about 6% faster than the Sep 8 build overall (696.1s → 652.5s). Most of that comes from cheaper job planning; execution time is roughly unchanged. q05 and q18 (and to a lesser degree q07) are slower in execution.
• Compared with plain Spark 3.5 (909.6s), Ballista is 28% faster overall (0.72x) and faster on 16 of 22 queries. Spark is faster on q05, q06, q07, q10, q14 and q20.
• Spark + Comet completes the suite in 450.0s, so Ballista is currently about 1.45x slower overall. Ballista is faster on q08, q09 and q17.
• Job planning is 15% of Ballista's total time (100.2s of 652.5s), 3–8s per query for anything that scans lineitem or orders. This is Scheduler re-reads every Parquet footer on every job, so planning takes seconds on large tables #2497: the scheduler re-reads every Parquet footer on every job. Leaving out planning, Ballista's execution time is 546.3s, 1.21x Spark + Comet.

Setup

All runs use TPC-H SF1000, Parquet (zstd), read from the same S3 location, on the same Kubernetes cluster (linux/amd64), with 2 iterations per query and the mean reported.

Ballista

• 1 scheduler, 32 executors, 16 concurrent tasks per executor
• Executor pods: 8 CPU request / 16 CPU limit, 32 GiB memory
• Upstream tpch benchmark binary from benchmarks/, built from the same commit as the scheduler/executor

Spark 3.5 (plain)

• Spark 3.5.8, same image and executor layout as the Comet run, with Comet disabled
• 32 executors, spark.executor.cores=16
• Executor pods: 8 CPU request / 8 CPU limit, 64 GiB heap

Spark 3.5 + Comet

• 32 executors, spark.executor.cores=16
• Executor pods: 8 CPU request / 8 CPU limit, 64 GiB heap + 32 GiB off-heap (Comet native memory) + 10 GiB overhead
• Comet native execution and Comet columnar shuffle enabled; otherwise default Comet settings

The layouts aren't identical. Ballista executors can burst to 16 CPUs but have less memory; Spark executors are capped at 8 CPUs but have much more memory.

Results (seconds, mean of 2 iterations)

• "Ballista change" is 32ceaf8 vs Sep 8 (< 1 means faster).
• "of which planning" is the part of the 32ceaf8 time spent between the job being queued and starting, from the scheduler event log (started_at - queued_at).
• "Ballista / Spark" and "Ballista / Comet" are Ballista 32ceaf8 vs plain Spark and vs Spark + Comet (< 1 means Ballista is faster).

Finding: job planning re-reads every Parquet footer (#2497)

While looking at these numbers I found that planning alone takes 3–8s for every query that scans lineitem or orders. The three queries that don't (q02, q11, q16) plan in 0.2–0.3s. Details and a local repro are in #2497. In short, the scheduler builds a fresh RuntimeEnv for every job, so the file statistics cache starts empty each time, and every Parquet footer is fetched from S3 again.

It matters most for the short queries. Planning is over 40% of the time for q01, q06, q12 and q14, and more than half of q06's gap to Spark + Comet (9.3s vs 0.8s) is planning. Across the suite, fixing it would bring Ballista from 1.45x to roughly 1.21x of Spark + Comet.

Changes since Sep 8

The Sep 8 run's event log only covers q01–q20, with one iteration each, so this breakdown is rough. For those 20 queries:

• Planning dropped from 136.3s to 89.4s, which accounts for most of the overall improvement.
• Execution was roughly flat (472.7s → 465.5s, 0.98x).

The slower queries are slower in execution, not planning:

• q05: execution 54.9s → 69.7s (71.2s / 68.2s this run), so this looks real.
• q18: execution 43.2s → 58.6s. Noisy (69.2s / 48.0s), but both iterations are slower than Sep 8.
• q07: execution 43.7s → 50.8s (52.3s / 49.3s).
• q21: 65.7s → 81.8s end to end. It's missing from the Sep 8 event log, so I can't split it.

I plan to rerun these with more iterations to confirm. I'm opening this to share the numbers and to ask whether anyone has ideas about the q05/q07/q18 execution slowdowns, or about the largest execution gaps vs Spark + Comet (q06, q07, q14, q20).