Skip to content

Delta charts render blank for any sweep without a spring-data-valkey-glide series #26

Description

@jamesx-improving

generate_interactive_graphs.py defaults its delta-chart baseline to a single Java
series, and auto-selection of a fallback is restricted to single-driver sweeps. A
multi-driver sweep with no spring-data-valkey-glide series therefore keeps a
baseline that matches nothing, and the chart is still emitted — titled, axes drawn,
no data — so the report looks broken rather than saying the comparison is
unavailable.

Repro

python scripts/generate_interactive_graphs.py <results-dir-from-a-node-sweep> --output /tmp/g
# stderr: Warning: reference 'spring-data-valkey-glide' not found in data. Delta charts will be empty.

Resulting HTML:

Plotly.newPlot("delta-total", [], {"title": {"text": "spring-data-valkey-glide Advantage — Total Workload Throughput" ...

Trace counts for every chart in that report — only the delta chart is empty:

scalability-total      traces=3   ['ioredis', 'iovalkey', 'valkey-glide-node']
delta-total            traces=0   []            <-- blank
latency-get-p50        traces=3   [...]
... (all 11 other charts: 3 traces each)

Cause

# scripts/generate_interactive_graphs.py:1165
REFERENCE_DRIVER = "spring-data-valkey-glide"
# build_delta_traces(), :1279
ref_points = data.get(REFERENCE_DRIVER, [])
if not ref_points:
    return traces          # empty; caller appends the chart anyway at :1568

Auto-selection exists but is deliberately scoped to single-driver sweeps, so a
multi-driver sweep never reaches it and keeps the Java default:

# main(), :1986-1993
driver_names = set(v.get("driver_name", "") for v in manifest["variants"].values())
if len(driver_names) == 1:
    REFERENCE_DRIVER = first_label

Same path affects cpu-efficiency-delta (:1431, :1696).

Affected matrices

Series labels come from the driver-config basename (load_all_data:
driver = ndjson_file.stem), so the baseline matches only when a config named
spring-data-valkey-glide.json is in the sweep:

Matrix Series Distinct drivers Delta charts
driver-comparison-defaults 9 9 populated (baseline present)
driver-comparison-high-tps 9 9 populated (baseline present)
lettuce-pool-sweep 3 1 populated (auto-selection fires)
valkey-glide-thread-sweep 5 1 populated (auto-selection fires)
valkey-glide-basic-multiclients (AWS default) 1 1 blank — unavoidable
node-driver-comparison 3 3 blank — this bug

Two distinct blank cases, only one of which is a real defect:

  • node-driver-comparison — multiple drivers, none of them the Java baseline.
    Auto-selection is skipped by the len(driver_names) == 1 guard, so the Java
    default survives and matches nothing. This is the bug.
  • valkey-glide-basic-multiclients — a single series has nothing to compare
    against, so a delta chart is meaningless by definition. Auto-selection does fire
    and picks that series, but then build_delta_traces skips it as the reference and
    returns nothing. Correct outcome, wrong presentation: it should be omitted rather
    than drawn blank (fix 2 below).

The single-driver multi-variant sweeps (lettuce-pool-sweep,
valkey-glide-thread-sweep) are fine — auto-selection picks the first variant and
the remaining variants plot against it.

Workaround (works today)

--reference already exists and fixes it:

python scripts/generate_interactive_graphs.py <results-dir> --output /tmp/g --reference ioredis
# delta-total traces=2 ['iovalkey (0.4.0)', 'valkey-glide-node (2.5.2)']

But nothing in the automation passes it: make benchmark-matrix-graphs (Makefile)
and infra/aws/run-remote.sh:240 both invoke the script with only
results_dir + --output. So every automated non-Java sweep silently publishes a
blank chart to S3.

Proposed fix

  1. Extend auto-selection to multi-driver sweeps. Drop the len(driver_names) == 1
    restriction and pick a deterministic baseline when the configured reference is
    absent — preferring a GLIDE series, since the charts are framed as GLIDE
    advantage, else the first series by label. Log which one was chosen.
  2. Omit the chart instead of emitting it blank when no baseline can be
    resolved (e.g. a genuine single-series sweep, where a delta is meaningless).
  3. Surface it in the report, not just stderr. The warning currently goes to
    stderr, which the AWS runner buries in logs/cloud-init-output.log; a reader of
    report.html gets no explanation. Either note the chosen baseline in the
    report's footer, or state that the delta comparison was unavailable.

Optionally, plumb --reference through make benchmark-matrix-graphs so a matrix
can pin its own baseline.

Scope: scripts/generate_interactive_graphs.py only (plus one Makefile line if the
option is plumbed). No engine or config changes.

Notes

Pre-existing — the hardcoded default dates to 091dd75 ("Add Python benchmark
orchestrator, CPU monitoring, and interactive graphs"). Surfaced while reviewing the
Node engine's first full AWS sweep, whose report was otherwise complete.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions