Skip to content

plot_correlations() misaligns the heatmap with non-default metrics order #1210

Description

@nikosbosse

Description

plot_correlations() produces a misaligned heatmap when the correlations object was created with get_correlations(scores, metrics = ...) using a non-default metric order (e.g. reversed or a subset in a different order). Cell values are attached to the wrong metric pairs, including a diagonal that is not 1.

Reproduction (verified)

library(scoringutils)
scores <- example_quantile |> as_forecast_quantile() |> score()
m <- rev(get_metrics(scores))
correlations <- get_correlations(summarise_scores(scores), metrics = m)
p <- plot_correlations(correlations, digits = 2)
pd <- data.table::as.data.table(p$data)
pd[as.character(metric) == as.character(variable)]$value

Observed diagonal values in the plot data (should all be 1):

interval_coverage_90 / interval_coverage_90 = 0.20
interval_coverage_50 / interval_coverage_50 = 0.16
bias / bias                                 = 0.15
dispersion / dispersion                     = 0.15
underprediction / underprediction           = 0.16
overprediction / overprediction             = 0.20

Only ae_median and wis show 1.00 — coincidentally, because the reversal maps them onto each other's rows and cor(wis, ae_median) rounds to 1. Comparing against the true correlation matrix, 30 of 36 plotted cells are wrong (e.g. plotted interval_coverage_90 x dispersion = 0.92 vs true -0.178).

Cause

  • R/get-correlations.R line 33: df <- scores[, .SD, .SDcols = names(scores) %in% metrics] uses a logical .SDcols, so the returned correlation table's rows/columns are in data-column order while the stored metrics attribute keeps the user-supplied order.
  • R/get-correlations.R line 80 (plot_correlations()): correlations[, .SD, .SDcols = metrics] (character vector) reorders the columns to attribute order while the rows stay in data order; line 98 then stamps colnames onto rownames and line 101 labels rows with metrics, yielding a doubly misaligned matrix.

get_correlations() output itself is internally consistent (for every row, correlations[metric == m][[m]] == 1); only the plot (and the attribute-vs-row-order mismatch) is wrong. With the default metric order everything aligns, which is why existing tests and the vdiffr snapshot never caught it.

Intended fix (maintainer approved)

  1. get_correlations(): select columns with a character .SDcols = metrics, so rows, columns, the metric column, and the metrics attribute all share the requested order (the originally intended behaviour).
  2. plot_correlations(): align rows explicitly via the metric column instead of relying on positional order (rownames(cor_mat) <- correlations$metric; cor_mat[metrics, , drop = FALSE]), moving the validation ahead of matrix construction. This also fixes plots of objects produced by the pre-fix get_correlations().

Part of the bug audit in #1189.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions