Add backtest checkpoints and backtest_combine(); require newer depend… - #97
Closed
RodrigoZepeda wants to merge 1 commit into
Closed
RodrigoZepeda wants to merge 1 commit into
RodrigoZepeda wants to merge 1 commit into
Conversation
…encies nowcast_backtest(checkpoint_file = ) saves each finished (engine, date) fit so an interrupted backtest resumes (#90). Only the main R session writes the file, through a temporary file and a rename, so every future plan is safe with parallel = TRUE; fits then run and are saved in waves of nbrOfWorkers(). A fingerprint of the data, seed, keep_draws, truth settings and each engine makes a mismatched call abort instead of mixing backtests; failed fits are retried. backtest_combine() joins backtests run separately -- other engines on the same dates, or the same engines on new dates -- into the object one call would have produced (#91). It aborts on a repeated (method, date) fit or on backtests that differ in strata, truth settings or quantile levels, and only_common_dates = TRUE keeps just the dates every method has. Dependencies: declare minimum versions in DESCRIPTION so that old installs fail at install time instead of in the tests: dplyr (>= 1.1.0) for `.by`, rlang (>= 1.0.0) for hash(), and tidyselect (>= 1.2.1), whose 1.2.0 gives a different error message that test-changers.R asserts on. Fix .batch_restore_date_class(): on older R (seen on R 4.3.3) setdiff() on Dates returns plain numbers and match() cannot compare them with dates, so simulate_batch() always aborted with "Every report date is closed". It now compares the numeric values. Bump the version to 1.1.1. Update NEWS.md, SKILL.md and DEVELOPMENT_SKILL.md (backtest checkpoints and combining, and testing against current R and packages), and add a backtest_combine tip to the ensemble vignette. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DW8xQ7pmptZYEJCb8KkEox
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…encies
nowcast_backtest(checkpoint_file = ) saves each finished (engine, date) fit so an interrupted backtest resumes (#90). Only the main R session writes the file, through a temporary file and a rename, so every future plan is safe with parallel = TRUE; fits then run and are saved in waves of nbrOfWorkers(). A fingerprint of the data, seed, keep_draws, truth settings and each engine makes a mismatched call abort instead of mixing backtests; failed fits are retried.
backtest_combine() joins backtests run separately -- other engines on the same dates, or the same engines on new dates -- into the object one call would have produced (#91). It aborts on a repeated (method, date) fit or on backtests that differ in strata, truth settings or quantile levels, and only_common_dates = TRUE keeps just the dates every method has.
Dependencies: declare minimum versions in DESCRIPTION so that old installs fail at install time instead of in the tests: dplyr (>= 1.1.0) for
.by, rlang (>= 1.0.0) for hash(), and tidyselect (>= 1.2.1), whose 1.2.0 gives a different error message that test-changers.R asserts on.Fix .batch_restore_date_class(): on older R (seen on R 4.3.3) setdiff() on Dates returns plain numbers and match() cannot compare them with dates, so simulate_batch() always aborted with "Every report date is closed". It now compares the numeric values.
Bump the version to 1.1.1. Update NEWS.md, SKILL.md and DEVELOPMENT_SKILL.md (backtest checkpoints and combining, and testing against current R and packages), and add a backtest_combine tip to the ensemble vignette.