Skip to content

get_pairwise_comparisons() aborts entirely when any subgroup has too few models #1209

Description

@nikosbosse

get_pairwise_comparisons() (and therefore add_relative_skill()) aborts entirely when any subgroup defined by by has fewer than two comparators, even if other subgroups could be compared just fine. The error message also does not say which subgroup caused the problem.

Reproduction

library(scoringutils)
scores <- score(example_quantile)
# keep only one model for Deaths; Cases still has 3 models
scores_sub <- scores[!(target_type == "Deaths" & model != "EuroCOVIDhub-ensemble")]
scores_sub[, .(n_models = length(unique(model))), by = target_type]
#>    target_type n_models
#> 1:       Cases        3
#> 2:      Deaths        1

get_pairwise_comparisons(scores_sub, by = "target_type")
#> Error in pairwise_comparison_one_group(scores = scores, metric = metric,  :
#>   ! There are not enough comparators to do any comparison

The Cases group alone would return 9 perfectly valid rows:

nrow(get_pairwise_comparisons(scores[target_type == "Cases"], by = "target_type"))
#> [1] 9

Cause

pairwise_comparison_one_group() calls cli_abort() when a group has fewer than two comparators (R/pairwise-comparisons.R:273-277). get_pairwise_comparisons() lapply()s it over split(scores, by = by) (R/pairwise-comparisons.R:221-234) with no per-group handling, so a single small subgroup aborts the whole computation.

Intended fix

  • In get_pairwise_comparisons(), subgroups with fewer than two comparators are skipped with a warning that names the offending subgroup(s) (formatted as col=value, e.g. "target_type=Deaths"); results are returned for the remaining groups.
  • add_relative_skill() consequently fills NA for the skipped groups (its merge already uses all.x = TRUE).
  • If all subgroups have fewer than two comparators (including by = NULL with a single model), the existing error is kept unchanged.
  • The safeguard inside pairwise_comparison_one_group() stays untouched for direct calls.

Part of the bug audit in #1189.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions