Skip to content

1116: Extend multivariate scoring vignette with energy vs variogram comparison#1193

Open
kathsherratt wants to merge 8 commits into
mainfrom
multivariate-vignette
Open

1116: Extend multivariate scoring vignette with energy vs variogram comparison#1193
kathsherratt wants to merge 8 commits into
mainfrom
multivariate-vignette

Conversation

@kathsherratt

@kathsherratt kathsherratt commented Jul 18, 2026

Copy link
Copy Markdown

Description

The aim of this PR is to lightly expand the multivariate scoring vignette and add a comparison between the energy score and the variogram score on some example data. For me, the problem of autocorrelation by forecast horizon is the most obvious and intuitive/compelling use case for multivariate scores, so wanted to include something on that (as well as the existing vignette with spatial dependence).

This PR relates to #1116 (now closed) and various others related to enenrgy/variogram scores.

Changes to vignettes/scoring-multivariate-forecasts.Rmd:

  • Added a note that the example forecasts are for demonstration only and do not represent joint draws from a multivariate distribution.
  • Added an example of pooling across forecast horizons (joint_across = c("horizon", "target_end_date")) so each score covers a whole trajectory.
  • Added a section explaining why the energy and variogram scores cannot be applied to quantile forecasts, and why lining up quantile levels as pseudo-samples gives a misleading score.
  • Added a section comparing the energy and variogram scores, with a worked two-model example (EuroCOVIDhub-ensemble vs epiforecasts-EpiNow2) including a plot of the forecast trajectories, showing how the two scores can disagree and what each rewards.

Also adds a NEWS item.

Checklist

  • My PR is based on a package issue and I have explicitly linked it.
  • I have included the target issue or issues in the PR title as follows: issue-number: PR title
  • I have tested my changes locally (vignette renders cleanly with the dev package loaded).
  • I have added or updated unit tests where necessary (N/A — vignette and NEWS only, no code changes).
  • I have updated the documentation if required (the vignette is the documentation).
  • I have built the package locally and run rebuilt docs using roxygen2 (N/A — no changes to R code or roxygen comments).
  • My code follows the established coding standards and I have run lintr::lint_package() to check for style issues introduced by my changes (no lints in the changed vignette).
  • I have added a news item linked to this PR.
  • I have reviewed CI checks for this PR and addressed them as far as I am able.

Dev: I designed this, implemented, then used iterative LLM support

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.15%. Comparing base (06ed56d) to head (8936cfb).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1193   +/-   ##
=======================================
  Coverage   98.15%   98.15%           
=======================================
  Files          41       41           
  Lines        2225     2225           
=======================================
  Hits         2184     2184           
  Misses         41       41           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

purrr is in Imports but not attached in the vignette, so the
qualified call is required for the example to run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kathsherratt

Copy link
Copy Markdown
Author

I wanted to keep this PR very light, but noting a couple of ideas for further development -

  • The vignette pretty much gets straight into functionality without very much on the use case; we could structure the vignette with more intro and rationale for multivariate scoring in general
  • Relatedly this could draw in from updated material on multivariate scores developed for teaching by @seabbs @nickreich

@kathsherratt
kathsherratt requested a review from seabbs July 18, 2026 12:08

@nikosbosse nikosbosse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you!

Should we warn more explicitly about multivariate quantile scores in the code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants