Skip to content

interval_coverage() floating-point matching error and NaN scores for quantile levels 0 and 1 #1202

Description

@nikosbosse

Two related edge-case bugs in the quantile-based metrics.

Bug 1: interval_coverage() fails on seq()-generated quantile levels (floating point)

interval_coverage() checks for the required quantile levels with an exact floating-point comparison (R/metrics-quantile.R, line 338: !all(necessary_quantiles %in% quantile_level)). Quantile levels generated with seq() are not exactly representable, so the check fails even though the required quantiles are present:

ql <- seq(0.05, 0.95, 0.05)
0.35 %in% ql
#> [1] FALSE   # element is 0.35000000000000003

interval_coverage(observed, predicted, ql, interval_range = 30)
#> Error in `interval_coverage()`:
#> ! To compute the interval coverage for an interval range of "30%", the
#>   0.35 and 0.65 quantiles are required

interval_coverage(observed, predicted, ql, interval_range = 50)
#> Error in `interval_coverage()`:
#> ! To compute the interval coverage for an interval range of "50%", the
#>   0.25 and 0.75 quantiles are required

The downstream row filter (line 351) is already safe because get_range_from_quantile() rounds ranges to 10 decimal places (R/helper-quantile-interval-range.R).

Fix: round both sides to 10 decimal places before matching, consistent with the package's existing convention (round(x, 10) in R/class-forecast-quantile.R and R/class-forecast-sample.R).

Bug 2: wis() / interval_score() / quantile_score(weigh = FALSE) return NaN for quantile levels 0 and 1

Quantile levels 0 and 1 are explicitly permitted by assert_input_quantile() (bounds lower = 0, upper = 1) and occur in real forecast hub data ("min"/"max" quantiles). But the 0/1 pair forms a 100% interval with alpha = 0, and interval_score() (R/metrics-interval-range.R, lines 168-171) computes 2 / alpha * (lower - observed) * indicator, which is Inf * 0 = NaN even when the observation lies inside the interval:

wis(5, matrix(c(1, 3, 5, 7, 9), nrow = 1), c(0, 0.25, 0.5, 0.75, 1))
#> [1] NaN
quantile_score(5, matrix(c(1, 3, 5, 7, 9), nrow = 1), c(0, 0.25, 0.5, 0.75, 1))
#> [1] 0.4

This breaks the documented identity WIS = mean of quantile scores. quantile_score(weigh = FALSE) has the same alpha = 0 division (R/metrics-quantile.R, line 692, 2 * score / alpha -> 0/0 = NaN).

Fix (per maintainer decision): apply the indicator before the 2/alpha factor and cancel 2/alpha * alpha/2 analytically in the weighted case, so quantile levels 0/1 give finite, correct scores when the observation is inside the interval. Unweighted interval_score(weigh = FALSE) returns Inf when the observation falls outside a 100% interval, which is mathematically correct. The sibling quantile_score(weigh = FALSE) NaN is fixed in the same way (0-valued weighted scores stay 0 instead of becoming 0/0). Verified numerically that outputs for all other interval ranges are unchanged and that the WIS = mean quantile score identity is restored to within 1e-14 with levels 0/1 included.

Part of the bug audit in #1189.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions