941: Consolidate score() input preparation into a single helper#1194
Open
nikosbosse wants to merge 4 commits into
Open
941: Consolidate score() input preparation into a single helper#1194nikosbosse wants to merge 4 commits into
nikosbosse wants to merge 4 commits into
Conversation
Add internal prepare_forecast_for_scoring() that performs the input preparation steps previously duplicated across all eight score() methods: cleaning the forecast, determining the forecast unit, validating the metrics and converting to a plain data.table. Validating the metrics inside the helper also locks down a subtle ordering constraint: the default `metrics = get_metrics(forecast)` promise must be forced while `forecast` still carries its forecast class, i.e. before the rebind to a plain data.table. Closes #941 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1194 +/- ##
==========================================
+ Coverage 98.15% 98.92% +0.76%
==========================================
Files 41 41
Lines 2225 2231 +6
==========================================
+ Hits 2184 2207 +23
+ Misses 41 24 -17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Document precisely why metrics are validated inside prepare_forecast_for_scoring() and that the lazy default is evaluated on the original (pre-cleaning) forecast object. - Describe the metrics argument locally instead of inheriting it from score(), removing a dangling reference to the "Customising metrics" section that only exists on the score() help page. - Reword the NEWS item: determining the forecast unit was previously only done by the methods that need it, not duplicated across all. - Slim the helper tests down to one focused test and make the not-modified-by-reference check a real before/after comparison. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scoring an ordinal forecast was not exercised anywhere in the test suite, which codecov surfaced as missing patch coverage on the refactored score.forecast_ordinal() preamble. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
@seabbs this is a little refactor we discussed a while ago. Back then, you liked the idea |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR closes #941.
As discussed in the issue, this consolidates the boilerplate input preparation steps shared by all eight
score()methods into a single internal helper,prepare_forecast_for_scoring():The helper cleans the forecast (
clean_forecast(copy = TRUE, na.omit = TRUE)), determines the forecast unit, validates the metrics and converts to a plaindata.table— in exactly the order the methods used before, so behaviour is unchanged (full test suite passes locally).Beyond removing duplication, the helper locks down a subtle ordering constraint in one audited place: the lazy default
metrics = get_metrics(forecast)must be forced whileforecaststill carries itsforecast_*class — there is noget_metrics.default(), so forcing it after the methods rebindforecastto a plaindata.tablewould fail with "no applicable method". Keepingvalidate_metrics()inside the helper makes this impossible to get wrong when adding future forecast classes. This is documented in the helper's roxygen, including the one subtle semantic note: the default is now evaluated on the original (pre-cleaning) forecast object, which makes no difference for the built-inget_metrics()methods as none of them inspect the data.Methods that do not need
forecast_unit(binary, point, multivariate point) simply don't unpack it.Tests: a new unit test for
prepare_forecast_for_scoring()intest-score.Rfails onmain(the function does not exist there) and passes on this branch; it also checks that the input forecast is not modified by reference. Additionally, scoring an ordinal forecast turned out not to be exercised anywhere in the suite (surfaced by codecov patch coverage), so ascore()test for the ordinal format is added too.The diff was adversarially reviewed by a multi-agent workflow (finder agents per correctness angle + independent verification of every candidate finding); the confirmed findings (documentation precision, a dangling help-page cross-reference, NEWS accuracy, missing ordinal coverage) are addressed in the later commits.
Checklist
score()test).lintr::lint_package()to check for style issues introduced by my changes (no new lints in changed files).🤖 Generated with Claude Code