Mitokic/09042026/best model updates

[Mike Tokic (mitokic)]

This pull request introduces significant improvements to the model selection and forecast validation process, focusing on balancing backtest accuracy with forecast plausibility and robust handling of rejected or incomplete forecasts. The changes enhance the reliability of model selection, improve run recovery, and ensure that only valid, high-quality forecasts are accepted and logged. Additionally, the pull request refactors how forecast files are loaded and validated, and updates documentation and versioning to reflect these improvements.

Model Selection and Quality Evaluation Enhancements:

• Improved model selection logic to balance backtest accuracy with forecast plausibility, preserve growth/seasonality patterns, and reliably reject invalid predictions, with better handling of averaging, forecast updates, and run recovery. (NEWS.md, R/agent_iterate_forecast.R) [1] [2]
• Added support for "soft" quality checks: forecasts can now be rejected for quality reasons even if accuracy goals are met, and such rejections are tracked and handled in the workflow. (R/agent_iterate_forecast.R) [1] [2] [3] [4] [5]

Forecast File Handling and Validation:

• Refactored loading of global and local forecast files to use new helper functions, improving robustness and clarity; validation now accounts for unpublished or rejected forecasts. (R/agent_iterate_forecast.R) [1] [2] [3] [4] [5] [6]
• Enhanced output generation and metric calculation to properly handle cases with unpublished or rejected selections, ensuring accurate reporting and logging. (R/agent_iterate_forecast.R)

Run Logging and Recovery Improvements:

• Added new logging function to record and retain best runs only when quality and accuracy criteria are met, and to prevent unnecessary reruns of previously rejected or accepted forecasts. (R/agent_iterate_forecast.R) [1] [2]
• Improved handling of default reforecast status, including checks to avoid refitting already rejected replacements and to restore accepted ones. (R/agent_iterate_forecast.R) [1] [2]

Documentation and Versioning:

• Updated version to 0.7.0.9006 and documented the improvements in DESCRIPTION and NEWS.md. (DESCRIPTION, NEWS.md) [1] [2]
• Updated .Rbuildignore to exclude new agent-related files. (.Rbuildignore)

These changes collectively improve the framework’s ability to select, validate, and recover high-quality forecasts while maintaining clear logging and robust error handling.

[Copilot]

🟡 Changes recommended

Default recovery can abort before forecasting when an expected best-run artifact is absent for RDS or Parquet runs.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Enhances forecast selection with plausibility checks, robust recovery, and quality-aware Agent workflows.

Changes:

• Adds trend, seasonality, validity, and accuracy-aware model selection.
• Improves hierarchical reconciliation, updates, retries, and artifact recovery.
• Expands tests, documentation, benchmarks, and package metadata.

File summaries

File	Description
R/agent_iterate_forecast.R	Integrates selection quality into Agent iterations and logging.
R/agent_run.R	Prevents retries for forecast-selection rejections.
R/agent_update_forecast.R	Adds update quality assessment and recovery.
R/ensemble_models.R	Screens ensemble inputs.
R/final_models.R	Implements quality-aware final selection and restart handling.
R/forecast_time_series.R	Documents selection behavior.
R/hierarchy.R	Validates reconciliation inputs.
R/prep_data.R	Persists original-target differencing values.
R/read_write_data.R	Uses exact reconciled artifact reads locally.
R/train_models.R	Preserves invalid predictions and reconstructs original targets.
tests/testthat/test-agent-selection-policy.R	Tests Agent selection and logging policy.
tests/testthat/test-agent-update-selection.R	Tests update-time selection behavior.
tests/testthat/test-artifact-restarts.R	Tests completed hierarchical restarts.
tests/testthat/test-best_models.R	Removes relocated shared fixtures.
tests/testthat/test-final-models-restart.R	Tests selection restart recovery.
tests/testthat/test-forecast-selection-corners.R	Covers selection edge cases.
tests/testthat/test-forecast-selection-trend.R	Covers trend-aware selection.
tests/testthat/test-multistep-daily-regression.R	Bounds expensive model parameters in tests.
tests/testthat/test-reconciled-forecast-selection.R	Tests pre-reconciliation selection.
tests/testthat/test-undifference.R	Tests distinct original-target reconstruction.
tools/benchmark-forecast-selection.R	Adds selection and hierarchy benchmarks.
tools/validate-agent-guidance.R	Validates package and development exclusions.
vignettes/ai-agent.Rmd	Documents Agent quality behavior.
vignettes/best-model-selection.Rmd	Documents selection rules and limitations.
man/final_models.Rd	Updates generated final-model documentation.
man/forecast_time_series.Rd	Updates generated forecasting documentation.
man/iterate_forecast.Rd	Updates generated iteration documentation.
man/update_forecast.Rd	Updates generated update documentation.
NEWS.md	Records the selection improvements.
DESCRIPTION	Bumps the development version.
.Rbuildignore	Excludes additional Agent tooling directories.

Review details

Files not reviewed (4)

• man/final_models.Rd: Generated file
• man/forecast_time_series.Rd: Generated file
• man/iterate_forecast.Rd: Generated file
• man/update_forecast.Rd: Generated file

• Files reviewed: 31/35 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Restart validation, optional remote artifacts, backend consistency, and logged accuracy still have correctness issues.

Get a fresh assessment by requesting another Copilot review.

Review details

Files not reviewed (4)

• man/final_models.Rd: Generated file
• man/forecast_time_series.Rd: Generated file
• man/iterate_forecast.Rd: Generated file
• man/update_forecast.Rd: Generated file

Suppressed comments (1)

R/agent_update_forecast.R:1432

• This checkpoint is optional for a genuinely new series, but the unconditional exact read only happens to degrade a missing CSV into an empty table. With local RDS/Parquet (and typical remote exact reads), a nonexistent -agent_best_run artifact raises before submit_fcst_run() runs, so new-series recovery fails for supported output backends. Use an optional exact read that treats only not-found as empty while still propagating authentication and other storage errors.

          agent_best_run_tbl <- read_file(agent_info_lean$project_info,

• Files reviewed: 36/40 changed files
• Comments generated: 4
• Review effort level: Balanced

[Mike Tokic (mitokic)]

FinnTS Model Selection: Branch Walkthrough

The main change is:

• Previously: pick the model with the lowest backtest error.
• Now: reject invalid forecasts, shortlist models close in accuracy, then prefer the more plausible forecast.

1. Use Comparable Evidence

Every candidate is checked against the same expected backtest and future dates, using historical actuals restored to their original scale.

• Prefer Target_Original, so outlier cleaning does not redefine the evaluation target. Existing missing-value imputation remains.
• Future target values never become historical evidence.
• Weekly forecasts are assessed at weekly cadence, before daily allocation.
• WMAPE remains the accuracy metric. The existing zero-actual convention and rounding are preserved.

2. Reject Invalid Candidates

These are hard failures: the candidate cannot win or contribute to a simple average.

Failure	Meaning
Incorrect coverage	Missing, duplicate, or unexpected scenario/date combinations
Non-finite predictions	Any required prediction is NA, NaN, Inf, or -Inf
Catastrophic magnitude	Predictions exceed the historical-scale backstop, normally 100 times a robust historical scale
Unavailable accuracy	No finite WMAPE can be calculated
The magnitude scale considers historical levels and variation, not just the last observation. Supported growth adjusts the future bound. All-zero history has a separate soft rule instead of dividing by zero. Invalid predictions are no longer hidden by replacing them with zero.

3. Build And Assess Averages

Finn screens existing individual and learned-ensemble outputs, then forms the requested simple averages from hard-eligible candidates, up to max_model_average. Each average is assessed as its own forecast. Invalid components cannot become acceptable merely because their errors cancel. Learned-ensemble inputs are also screened before fitting.
The selected forecast gets Best_Model = "Yes". If an average wins, Finn saves that exact combination. Otherwise, it retains the best eligible average under the same selection policy as a nonwinner, not necessarily the lowest-WMAPE average.

4. Check Future Plausibility

These are generally soft concerns, not automatic disqualifications.

Check	What Finn looks for
Level	Forecast moves unusually far from a history-based reference
Trend	Forecast rises or falls much differently from historical changes
Seasonal timing	Peaks and troughs reverse the supported historical pattern
Seasonal amplitude	Full-cycle seasonality shrinks below one-third or grows above three times historical amplitude
All-zero history	Forecast introduces a nonzero level without historical support
Level and trend checks use six robust reference scales, with level tolerance widening over the horizon. Seasonal checks require enough evidence: generally two complete historical cycles and seasonal strength of at least 0.6. Informative shorter horizons can receive timing checks with at least three future points. Unsupported checks stay unassessed, not failed.
Growth is explicitly supported. Finn can use an additive or proportional-growth reference when regular history supports it and it improves historical prediction error by at least 20% in each of two validation windows. Otherwise, the reference falls back to seasonal naive or a recent median. This reference judges the candidate; it does not replace its forecast.

5. Choose Among Close Contenders

After hard failures are removed, the shortlist includes candidates within best WMAPE + the larger of 0.5 percentage points or 5% of best WMAPE.
For example, if the best WMAPE is 8.0%, the ceiling is 8.5%. A safer 8.3% forecast can beat a questionable 8.0% forecast; an 8.8% forecast cannot.
Within that shortlist, the order is:

1. Lowest risk, measured by the worst assessed concern rather than an average.
2. Fewest concerns.
3. Best seasonal-amplitude fidelity, but only when every candidate in that tied group has an assessed score.
4. Lowest WMAPE.
5. Stable model identifier for a deterministic final tie.
   Seasonal fidelity allows normal historical amplitude variation and can favor an intact seasonal forecast over an unnecessarily flattened average. It is only a tie-breaker, not another rejection rule.
   Ordinary runs can still select a forecast with soft concerns. If everything hard-fails, standard selection errors; Agent attempts are recorded as rejected and consume iteration budget. Standard selection does not automatically train extra models.

6. Reconcile After Selection

For hierarchical forecasts, each hierarchy node can choose a different model or average. Finn reconciles that selected mixture using the existing hts solver.
There is no second plausibility-ranking pass after reconciliation, no automatic switch to one model family, and no individual bottom-row replacement after the solve. Reconciled backtests supply normal accuracy reporting. Standard getters retain available per-model reconciled outputs for inspection; hierarchical Agent output publishes the selected reconciled mixture.

7. Keep Agent Iteration Decisions Separate

The Agent’s existing 10% rule is preserved, not replaced by the new shortlist allowance:

• Start from the earliest minimum-WMAPE iteration in the current Agent version. A later eligible iteration within 10% relative of that WMAPE can become the search context if its model_avg_wmape is strictly lower; the lowest qualifying average wins.
• For local runs, that average describes individual candidate-model WMAPEs, not the WMAPE of an averaged forecast. Genuine mean, median, and spread remain; global summaries retain their existing run-level meanings.
• A promising search context does not overwrite a better saved local forecast. All saved global winners must come from one winning global iteration, promoted together, while superior local winners remain protected. Partial global results cannot promote only their successful series.
• Past iterations are compared using recorded accuracy, without repeating future-plausibility scoring. Completeness and four-decimal WMAPE govern normal stopping and local routing; soft concerns alone do not veto reaching the accuracy goal.
  For example, adding a regressor may leave ARIMA unchanged while improving the multivariate models. The average-model signal preserves that direction for further exploration.

8. Apply Stricter Rules To Updates

update_forecast() first refits the previously selected model or exact average components, not every model originally requested. Global updates use the union of required components from the single saved global iteration.
New predictions are checked before reconciliation, including after accuracy-triggered retuning. Every required component must pass hard checks, and the selected combination must have no applicable soft concerns. Passing reuse retains the original combination.
Rejected series receive a bounded default-local recovery attempt; healthy siblings remain. The replacement must also pass the applicable checks, otherwise Finn errors rather than repeatedly fitting or publishing a questionable replacement. Incomplete or rejected source hierarchies are not partially reconciled.

9. Recover Saved Results Safely

A filename, completion log, or Best_Model column alone no longer proves completion. Finn validates the saved winner’s coverage, finite predictions, identity, and average arithmetic. Incomplete selection is rebuilt from existing predictions without retraining; missing original average components require restoration. Reconciled output is checked structurally and rebuilt when necessary.
Complete saved winners are reused without retrospective plausibility reassessment. Mixed global-iteration metadata fails explicitly, including after interrupted writes.

Bottom Line

The intent is better-informed selection, not guaranteed future accuracy. Real regime changes, weak history, and reconciliation can still produce questionable outcomes. Plausibility helps choose among sufficiently accurate candidates; it does not guarantee that a good candidate exists.

[smdp2000ofc]

I think it would be helpful to add comments to some of the functions in this file of what it does and implemented in certain way, to improve readability and maintainability