generate_interactive_graphs.py defaults its delta-chart baseline to a single Java
series, and auto-selection of a fallback is restricted to single-driver sweeps. A
multi-driver sweep with no spring-data-valkey-glide series therefore keeps a
baseline that matches nothing, and the chart is still emitted — titled, axes drawn,
no data — so the report looks broken rather than saying the comparison is
unavailable.
Repro
python scripts/generate_interactive_graphs.py <results-dir-from-a-node-sweep> --output /tmp/g
# stderr: Warning: reference 'spring-data-valkey-glide' not found in data. Delta charts will be empty.
Resulting HTML:
Plotly.newPlot("delta-total", [], {"title": {"text": "spring-data-valkey-glide Advantage — Total Workload Throughput" ...
Trace counts for every chart in that report — only the delta chart is empty:
scalability-total traces=3 ['ioredis', 'iovalkey', 'valkey-glide-node']
delta-total traces=0 [] <-- blank
latency-get-p50 traces=3 [...]
... (all 11 other charts: 3 traces each)
Cause
# scripts/generate_interactive_graphs.py:1165
REFERENCE_DRIVER = "spring-data-valkey-glide"
# build_delta_traces(), :1279
ref_points = data.get(REFERENCE_DRIVER, [])
if not ref_points:
return traces # empty; caller appends the chart anyway at :1568
Auto-selection exists but is deliberately scoped to single-driver sweeps, so a
multi-driver sweep never reaches it and keeps the Java default:
# main(), :1986-1993
driver_names = set(v.get("driver_name", "") for v in manifest["variants"].values())
if len(driver_names) == 1:
REFERENCE_DRIVER = first_label
Same path affects cpu-efficiency-delta (:1431, :1696).
Affected matrices
Series labels come from the driver-config basename (load_all_data:
driver = ndjson_file.stem), so the baseline matches only when a config named
spring-data-valkey-glide.json is in the sweep:
| Matrix |
Series |
Distinct drivers |
Delta charts |
driver-comparison-defaults |
9 |
9 |
populated (baseline present) |
driver-comparison-high-tps |
9 |
9 |
populated (baseline present) |
lettuce-pool-sweep |
3 |
1 |
populated (auto-selection fires) |
valkey-glide-thread-sweep |
5 |
1 |
populated (auto-selection fires) |
valkey-glide-basic-multiclients (AWS default) |
1 |
1 |
blank — unavoidable |
node-driver-comparison |
3 |
3 |
blank — this bug |
Two distinct blank cases, only one of which is a real defect:
node-driver-comparison — multiple drivers, none of them the Java baseline.
Auto-selection is skipped by the len(driver_names) == 1 guard, so the Java
default survives and matches nothing. This is the bug.
valkey-glide-basic-multiclients — a single series has nothing to compare
against, so a delta chart is meaningless by definition. Auto-selection does fire
and picks that series, but then build_delta_traces skips it as the reference and
returns nothing. Correct outcome, wrong presentation: it should be omitted rather
than drawn blank (fix 2 below).
The single-driver multi-variant sweeps (lettuce-pool-sweep,
valkey-glide-thread-sweep) are fine — auto-selection picks the first variant and
the remaining variants plot against it.
Workaround (works today)
--reference already exists and fixes it:
python scripts/generate_interactive_graphs.py <results-dir> --output /tmp/g --reference ioredis
# delta-total traces=2 ['iovalkey (0.4.0)', 'valkey-glide-node (2.5.2)']
But nothing in the automation passes it: make benchmark-matrix-graphs (Makefile)
and infra/aws/run-remote.sh:240 both invoke the script with only
results_dir + --output. So every automated non-Java sweep silently publishes a
blank chart to S3.
Proposed fix
- Extend auto-selection to multi-driver sweeps. Drop the
len(driver_names) == 1
restriction and pick a deterministic baseline when the configured reference is
absent — preferring a GLIDE series, since the charts are framed as GLIDE
advantage, else the first series by label. Log which one was chosen.
- Omit the chart instead of emitting it blank when no baseline can be
resolved (e.g. a genuine single-series sweep, where a delta is meaningless).
- Surface it in the report, not just stderr. The warning currently goes to
stderr, which the AWS runner buries in logs/cloud-init-output.log; a reader of
report.html gets no explanation. Either note the chosen baseline in the
report's footer, or state that the delta comparison was unavailable.
Optionally, plumb --reference through make benchmark-matrix-graphs so a matrix
can pin its own baseline.
Scope: scripts/generate_interactive_graphs.py only (plus one Makefile line if the
option is plumbed). No engine or config changes.
Notes
Pre-existing — the hardcoded default dates to 091dd75 ("Add Python benchmark
orchestrator, CPU monitoring, and interactive graphs"). Surfaced while reviewing the
Node engine's first full AWS sweep, whose report was otherwise complete.
generate_interactive_graphs.pydefaults its delta-chart baseline to a single Javaseries, and auto-selection of a fallback is restricted to single-driver sweeps. A
multi-driver sweep with no
spring-data-valkey-glideseries therefore keeps abaseline that matches nothing, and the chart is still emitted — titled, axes drawn,
no data — so the report looks broken rather than saying the comparison is
unavailable.
Repro
Resulting HTML:
Trace counts for every chart in that report — only the delta chart is empty:
Cause
Auto-selection exists but is deliberately scoped to single-driver sweeps, so a
multi-driver sweep never reaches it and keeps the Java default:
Same path affects
cpu-efficiency-delta(:1431, :1696).Affected matrices
Series labels come from the driver-config basename (
load_all_data:driver = ndjson_file.stem), so the baseline matches only when a config namedspring-data-valkey-glide.jsonis in the sweep:driver-comparison-defaultsdriver-comparison-high-tpslettuce-pool-sweepvalkey-glide-thread-sweepvalkey-glide-basic-multiclients(AWS default)node-driver-comparisonTwo distinct blank cases, only one of which is a real defect:
node-driver-comparison— multiple drivers, none of them the Java baseline.Auto-selection is skipped by the
len(driver_names) == 1guard, so the Javadefault survives and matches nothing. This is the bug.
valkey-glide-basic-multiclients— a single series has nothing to compareagainst, so a delta chart is meaningless by definition. Auto-selection does fire
and picks that series, but then
build_delta_tracesskips it as the reference andreturns nothing. Correct outcome, wrong presentation: it should be omitted rather
than drawn blank (fix 2 below).
The single-driver multi-variant sweeps (
lettuce-pool-sweep,valkey-glide-thread-sweep) are fine — auto-selection picks the first variant andthe remaining variants plot against it.
Workaround (works today)
--referencealready exists and fixes it:But nothing in the automation passes it:
make benchmark-matrix-graphs(Makefile)and
infra/aws/run-remote.sh:240both invoke the script with onlyresults_dir+--output. So every automated non-Java sweep silently publishes ablank chart to S3.
Proposed fix
len(driver_names) == 1restriction and pick a deterministic baseline when the configured reference is
absent — preferring a GLIDE series, since the charts are framed as GLIDE
advantage, else the first series by label. Log which one was chosen.
resolved (e.g. a genuine single-series sweep, where a delta is meaningless).
stderr, which the AWS runner buries in
logs/cloud-init-output.log; a reader ofreport.htmlgets no explanation. Either note the chosen baseline in thereport's footer, or state that the delta comparison was unavailable.
Optionally, plumb
--referencethroughmake benchmark-matrix-graphsso a matrixcan pin its own baseline.
Scope:
scripts/generate_interactive_graphs.pyonly (plus one Makefile line if theoption is plumbed). No engine or config changes.
Notes
Pre-existing — the hardcoded default dates to
091dd75("Add Python benchmarkorchestrator, CPU monitoring, and interactive graphs"). Surfaced while reviewing the
Node engine's first full AWS sweep, whose report was otherwise complete.