feat(benchmarks): add dict_encoding_sweep harness

[Rich-T-kid]

Which issue does this PR close?

• Closes #.

Rationale for this change

Several recent PRs that touch dictionary / Utf8View operator paths have had
to answer the same question: "at what cardinality does dictionary encoding
stop being a win over Utf8View?" There is no in-tree harness that answers
this reproducibly — contributors have been spinning up ad-hoc scripts, each
measuring slightly different things.

This PR adds dict_encoding_sweep: a focused benchmark driver that generates
two in-memory tables sharing identical values (one Utf8View, one
Dictionary<Int32, Utf8>), runs the same SQL against each, and reports
median/min/p25/p75 runtimes to CSV. One invocation sweeps across an NDV
grid for a chosen operator group so the crossover point is directly visible.

What changes are included in this PR?

• benchmarks/src/dict_encoding_sweep.rs — harness library (deterministic
  data gen, interleaved warmup/timing loop, per-group query catalog, CSV
  writer).
• benchmarks/src/bin/dict_encoding_sweep.rs — CLI driver; one invocation
  handles one (group, rows, width, nulls, batch_size) combination and
  sweeps across the provided --ndvs.
• benchmarks/README.md — new Dictionary vs Utf8View Encoding Sweep
  section documenting operator groups, flags, and CSV schema.

Nothing under datafusion/ is touched.

What is the testing strategy for this PR?

The sweep binary is itself the artifact — a smoke run (--rows 100000 --ndvs 10,1000 --iters 3 --warmups 1) completes end-to-end and writes a valid CSV.
No new unit tests are added; the harness is a self-contained reproduction
tool rather than a code path exercised by production queries. cargo fmt --all --check and cargo clippy -p datafusion-benchmarks --lib --bins --all-features -- -D warnings both pass on the new files.

Are there any user-facing changes?

No. New files under benchmarks/.

Results

Numbers below come from running the harness on this branch, 2026-10-12, with
--iters 15 --warmups 3 --dict-chunk 100000 --ndvs 10,100,500,1000,2000,3000,4000,8192. Cells are dict median / view median — values < 1.00 mean dictionary is faster, > 1.00 means
Utf8View wins. Variance on this machine is roughly ±20 %; treat
differences inside that band as noise. Full CSVs are reproducible from the
commands in benchmarks/README.md.

filter — 1,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
filter_1pct_group	0.43	0.54	0.24	0.29	0.37	0.53	0.53	1.05
filter_eq	0.29	0.49	0.63	0.72	0.97	1.22	1.45	2.41
filter_gt	0.36	0.65	0.79	0.92	1.22	1.51	1.82	3.05
filter_in10	0.09	0.10	0.18	0.25	0.38	0.50	0.62	1.13
filter_like_prefix	0.26	0.30	0.30	0.35	0.43	0.50	0.60	0.95
filter_ne	0.30	0.49	0.62	0.74	0.97	1.21	1.41	2.37

filter — 5,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
filter_1pct_group	0.41	0.46	0.24	0.28	0.36	0.51	0.53	1.01
filter_eq	0.28	0.39	0.57	0.73	0.99	1.23	1.41	2.32
filter_gt	0.34	0.54	0.75	0.94	1.17	1.47	1.75	3.01
filter_in10	0.09	0.10	0.18	0.25	0.37	0.49	0.62	1.11
filter_like_prefix	0.25	0.31	0.30	0.35	0.44	0.53	0.59	0.93
filter_ne	0.29	0.41	0.60	0.73	0.99	1.23	1.40	2.36

aggregation — 1,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
agg_count_distinct_k	0.70	0.78	0.71	0.71	0.72	0.77	0.79	0.91
agg_group_count_k	0.75	0.81	0.75	0.74	0.74	0.78	0.79	0.94
agg_group_count_kw	0.70	0.72	0.69	0.71	0.72	0.77	0.82	0.90
agg_group_sum_v_k	0.76	0.83	0.75	0.74	0.75	0.77	0.80	0.92
agg_having_k	0.75	0.81	0.76	0.74	0.74	0.78	0.79	0.94

aggregation — 5,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
agg_count_distinct_k	0.70	0.79	0.71	0.71	0.71	0.74	0.77	0.89
agg_group_count_k	0.75	0.81	0.73	0.74	0.74	0.78	0.78	0.92
agg_group_count_kw	0.70	0.72	0.69	0.71	0.72	0.78	0.80	0.86
agg_group_sum_v_k	0.75	0.82	0.75	0.74	0.75	0.78	0.77	0.91
agg_having_k	0.75	0.81	0.74	0.73	0.74	0.78	0.78	0.92

sort — 1,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
sort_k	0.50	0.58	0.61	0.64	0.74	0.81	0.90	1.36
sort_k_desc	0.50	0.59	0.65	0.67	0.76	0.83	0.95	1.41
sort_k_limit_1000	0.35	0.65	0.80	1.12	1.61	1.34	1.46	1.91
sort_kw	1.02	1.75	1.71	1.65	1.60	1.58	1.60	1.66
sort_v_by_k	0.54	0.50	0.71	0.73	0.83	0.90	1.11	1.87

sort — 5,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
sort_k	0.51	0.74	0.72	0.74	0.83	0.96	1.11	1.86
sort_k_desc	0.50	0.74	0.73	0.75	0.87	1.03	1.15	1.89
sort_k_limit_1000	0.31	0.57	0.76	0.90	1.21	1.47	1.79	3.68
sort_kw	1.02	3.16	2.73	2.63	2.56	2.53	2.60	2.60
sort_v_by_k	0.56	1.15	0.99	0.95	1.11	1.21	1.50	2.81

project — 1,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
proj_case_like	0.40	0.46	0.46	0.51	0.62	0.70	0.73	0.93
proj_concat	0.91	0.92	0.92	0.91	0.88	0.90	0.92	0.94
proj_length	0.03	0.03	0.04	0.06	0.09	0.13	0.15	0.34
proj_substr	1.71	1.73	1.71	1.75	1.72	1.71	1.75	1.74
proj_upper	0.01	0.01	0.01	0.02	0.02	0.02	0.04	0.09

project — 5,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
proj_case_like	0.42	0.47	0.47	0.52	0.62	0.71	0.77	0.99
proj_concat	0.92	0.92	0.92	0.90	0.87	0.88	0.92	0.92
proj_length	0.03	0.03	0.04	0.07	0.10	0.14	0.17	0.37
proj_substr	1.75	1.74	1.74	1.74	1.74	1.73	1.74	1.75
proj_upper	0.01	0.01	0.01	0.01	0.02	0.03	0.04	0.08

join — 1,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
join_anti_half	0.98	1.00	0.97	0.97	0.97	0.98	1.00	1.06
join_inner	0.94	1.05	1.00	0.99	1.01	1.01	1.05	1.24
join_inner_project	0.94	0.98	1.00	0.96	0.99	1.01	1.04	1.23
join_left	0.95	1.02	0.95	0.95	0.98	1.01	1.05	1.24
join_semi	0.94	1.00	0.98	0.96	1.00	1.01	1.06	1.21

join — 5,000,000 rows

query \ ndv	10	100	500	1000	2000	3000	4000	8192
join_anti_half	0.98	1.01	0.96	0.98	0.95	0.97	0.99	1.05
join_inner	0.97	1.04	1.00	0.96	0.97	1.01	1.06	1.26
join_inner_project	0.93	1.01	0.96	0.96	0.97	1.01	1.05	1.21
join_left	0.98	1.00	0.98	0.96	0.98	1.01	1.06	1.24
join_semi	0.97	1.00	0.99	0.99	0.99	1.01	1.05	1.21

Headline takeaways consistent across both row counts:

• Filter: eq / ne / gt all cross 1.00 right around NDV 2000 at both
  row counts; in10, like_prefix, and 1pct_group stay dict-favourable
  well past 3000.
• Sort: sort_k / sort_k_desc are still dict-favourable at 2k–3k
  (0.74–1.03) and cross around NDV 4000; sort_k_limit_1000 and
  sort_v_by_k cross earlier (~1k–2k); sort_kw is a persistent dict
  regression (ties-breaker column).
• Aggregation: dict wins (0.70–0.82) across the entire 10–3000 range;
  only at 4k–8k does it drift to 0.78–0.94 (still dict-favourable).
• Projection: proj_length and proj_upper are dict runaways
  (0.01–0.14) through 3000; proj_substr is a persistent Utf8View win.
• Join: nearly a wash at NDV ≤ 3000 (0.95–1.02); mild Utf8View edge
  (~1.05–1.26×) past 4000.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 590 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.72%. Comparing base (9551051) to head (2b497c8).

Files with missing lines	Patch %	Lines
benchmarks/src/dict_encoding_sweep.rs	0.00%	396 Missing ⚠️
benchmarks/src/bin/dict_encoding_sweep.rs	0.00%	194 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26250      +/-   ##
==========================================
- Coverage   82.83%   82.72%   -0.12%     
==========================================
  Files        1149     1151       +2     
  Lines      452724   453314     +590     
  Branches   452724   453314     +590     
==========================================
- Hits       375022   375009      -13     
- Misses      54881    55481     +600     
- Partials    22821    22824       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rich-T-kid]

will keep this PR in draft mode and wait for the next arrow-rs bump as apache/arrow-rs#11407 should close the gap for the sort operator, and #26186 & #25874 will remove the un-needed cast in the project operator. Good results overall, once the parquet scan speed up is in I expect dictionarys to be faster than utf8View for low to medium queries