Description
plot_correlations() produces a misaligned heatmap when the correlations object was created with get_correlations(scores, metrics = ...) using a non-default metric order (e.g. reversed or a subset in a different order). Cell values are attached to the wrong metric pairs, including a diagonal that is not 1.
Reproduction (verified)
library(scoringutils)
scores <- example_quantile |> as_forecast_quantile() |> score()
m <- rev(get_metrics(scores))
correlations <- get_correlations(summarise_scores(scores), metrics = m)
p <- plot_correlations(correlations, digits = 2)
pd <- data.table::as.data.table(p$data)
pd[as.character(metric) == as.character(variable)]$value
Observed diagonal values in the plot data (should all be 1):
interval_coverage_90 / interval_coverage_90 = 0.20
interval_coverage_50 / interval_coverage_50 = 0.16
bias / bias = 0.15
dispersion / dispersion = 0.15
underprediction / underprediction = 0.16
overprediction / overprediction = 0.20
Only ae_median and wis show 1.00 — coincidentally, because the reversal maps them onto each other's rows and cor(wis, ae_median) rounds to 1. Comparing against the true correlation matrix, 30 of 36 plotted cells are wrong (e.g. plotted interval_coverage_90 x dispersion = 0.92 vs true -0.178).
Cause
R/get-correlations.R line 33: df <- scores[, .SD, .SDcols = names(scores) %in% metrics] uses a logical .SDcols, so the returned correlation table's rows/columns are in data-column order while the stored metrics attribute keeps the user-supplied order.
R/get-correlations.R line 80 (plot_correlations()): correlations[, .SD, .SDcols = metrics] (character vector) reorders the columns to attribute order while the rows stay in data order; line 98 then stamps colnames onto rownames and line 101 labels rows with metrics, yielding a doubly misaligned matrix.
get_correlations() output itself is internally consistent (for every row, correlations[metric == m][[m]] == 1); only the plot (and the attribute-vs-row-order mismatch) is wrong. With the default metric order everything aligns, which is why existing tests and the vdiffr snapshot never caught it.
Intended fix (maintainer approved)
get_correlations(): select columns with a character .SDcols = metrics, so rows, columns, the metric column, and the metrics attribute all share the requested order (the originally intended behaviour).
plot_correlations(): align rows explicitly via the metric column instead of relying on positional order (rownames(cor_mat) <- correlations$metric; cor_mat[metrics, , drop = FALSE]), moving the validation ahead of matrix construction. This also fixes plots of objects produced by the pre-fix get_correlations().
Part of the bug audit in #1189.
Description
plot_correlations()produces a misaligned heatmap when thecorrelationsobject was created withget_correlations(scores, metrics = ...)using a non-default metric order (e.g. reversed or a subset in a different order). Cell values are attached to the wrong metric pairs, including a diagonal that is not 1.Reproduction (verified)
Observed diagonal values in the plot data (should all be 1):
Only
ae_medianandwisshow 1.00 — coincidentally, because the reversal maps them onto each other's rows andcor(wis, ae_median)rounds to 1. Comparing against the true correlation matrix, 30 of 36 plotted cells are wrong (e.g. plottedinterval_coverage_90xdispersion= 0.92 vs true -0.178).Cause
R/get-correlations.Rline 33:df <- scores[, .SD, .SDcols = names(scores) %in% metrics]uses a logical.SDcols, so the returned correlation table's rows/columns are in data-column order while the storedmetricsattribute keeps the user-supplied order.R/get-correlations.Rline 80 (plot_correlations()):correlations[, .SD, .SDcols = metrics](character vector) reorders the columns to attribute order while the rows stay in data order; line 98 then stamps colnames onto rownames and line 101 labels rows withmetrics, yielding a doubly misaligned matrix.get_correlations()output itself is internally consistent (for every row,correlations[metric == m][[m]] == 1); only the plot (and the attribute-vs-row-order mismatch) is wrong. With the default metric order everything aligns, which is why existing tests and the vdiffr snapshot never caught it.Intended fix (maintainer approved)
get_correlations(): select columns with a character.SDcols = metrics, so rows, columns, themetriccolumn, and themetricsattribute all share the requested order (the originally intended behaviour).plot_correlations(): align rows explicitly via themetriccolumn instead of relying on positional order (rownames(cor_mat) <- correlations$metric; cor_mat[metrics, , drop = FALSE]), moving the validation ahead of matrix construction. This also fixes plots of objects produced by the pre-fixget_correlations().Part of the bug audit in #1189.