diff --git a/.claude/rules/r-package.md b/.claude/rules/r-package.md index 67c45f99..833124b2 100644 --- a/.claude/rules/r-package.md +++ b/.claude/rules/r-package.md @@ -1,6 +1,7 @@ --- paths: - "R/**/*.R" + - "tests/**/*.R" - "DESCRIPTION" - "NAMESPACE" - "man/**/*.Rd" @@ -21,3 +22,12 @@ paths: - Keep errors actionable and consistent. Use `stop()` for user-facing errors, `warning()` for recoverable conditions, and `message()` for informational output. - Avoid unnecessary copies in tight loops; vectorize when it improves clarity or performance. - Never add code that deletes files. + +## Function Documentation + +- Document every function, including internal package functions, nested helpers, and test helpers. Internal visibility is not a reason to omit documentation. Add missing documentation when creating or modifying a function; keep unrelated repository-wide backfills as separately scoped work. +- Exported functions need roxygen documentation covering purpose, parameters, return value, relevant errors and side effects, and examples when useful. Regenerate the public help with `devtools::document()` when roxygen changes. +- Internal functions need a concise contract immediately above the definition, using ordinary `#` comments or existing internal roxygen with `@noRd`. Explain why the helper exists, its inputs and returned value, and any non-obvious shape, units, missing-value, ordering, or identity assumptions. Describe storage writes, cache mutation, errors, and other side effects when applicable. Do not export a helper merely to document it. +- Keep the amount of documentation proportional to complexity. A simple helper can use a short purpose/input/output block; a policy or orchestration function needs its decision boundary, invariants, and failure behavior. Document anonymous callbacks in the enclosing contract or immediately before the callback so their role and assumptions are clear. +- Explain the rationale before non-obvious algorithm or policy blocks, especially thresholds, metric units, tie-breaking, transformation order, restart reuse, and artifact ownership. Do not narrate obvious assignments or add decorative separators. +- Update the function contract and affected workflow documentation in the same change as behavior. Link to the owning guide and focused tests for complex shared policies instead of copying a long algorithm description into every caller. During review, check that the documentation matches the implementation, not just that a comment exists. diff --git a/.cursor/rules/r-package.mdc b/.cursor/rules/r-package.mdc index a176f4aa..046c5bad 100644 --- a/.cursor/rules/r-package.mdc +++ b/.cursor/rules/r-package.mdc @@ -1,6 +1,6 @@ --- description: "Use when working on FinnTS r package rules." -globs: "R/**/*.R,DESCRIPTION,NAMESPACE,man/**/*.Rd,vignettes/**/*.{Rmd,qmd},NEWS.md" +globs: "R/**/*.R,tests/**/*.R,DESCRIPTION,NAMESPACE,man/**/*.Rd,vignettes/**/*.{Rmd,qmd},NEWS.md" alwaysApply: false --- diff --git a/.github/agent-guides/architecture-map.md b/.github/agent-guides/architecture-map.md index b5c3a5e4..4332097f 100644 --- a/.github/agent-guides/architecture-map.md +++ b/.github/agent-guides/architecture-map.md @@ -25,6 +25,96 @@ Start with `tests/testthat/test-prep_data.R`, `tests/testthat/test-prep_models.R High-value tests are `test-agent-chat-serialization.R`, `test-agent-duplicate-runs.R`, `test-agent-graceful-abort.R`, `test-agent-eda-summaries.R`, `test-finalize_run.R`, and `test-combo-normalization.R` under `tests/testthat/`. +## Model Selection + +Use this map first to locate the owning workflow, then consult the [Best Model Selection vignette](../../vignettes/best-model-selection.Rmd) for thresholds, formulas, examples, and limitations. The [Agent iteration policy](../../.claude/rules/agent-runtime.md#iteration-selection-policy) is the maintenance contract; a documentation or performance change must not redefine it. + +### Five Different Decisions + +| Decision | Owner | What it decides | +| --- | --- | --- | +| Select a candidate within one run and series | `select_series_forecasts()` and `rank_forecast_candidates()` in [R/forecast_selection.R](../../R/forecast_selection.R) | Hard eligibility, then an accuracy shortlist, then future-risk ordering. Applies to individuals, learned ensembles, and requested simple averages. | +| Choose settings for the next Agent iteration | `load_run_results()` in [R/agent_iterate_forecast.R](../../R/agent_iterate_forecast.R), using `best_agent_iteration()` in [R/forecast_selection.R](../../R/forecast_selection.R) | Current-version chronological history, earliest minimum WMAPE, then later near-best iterations with a lower model-pool mean. Does not rescore old future forecasts. | +| Promote a saved Agent forecast | `log_selected_agent_run()` in [R/agent_iterate_forecast.R](../../R/agent_iterate_forecast.R) | Run-level promotion plus per-series protection of local winners. All remaining global winners must identify one global iteration. | +| Accept an updated saved choice | `update_forecast_combo()` in [R/agent_update_forecast.R](../../R/agent_update_forecast.R), using `assess_update_forecasts()` in [R/forecast_selection.R](../../R/forecast_selection.R) | Refit required saved components, assess the newly generated source paths, and retain that exact choice or route rejection to default-local recovery. | +| Produce a coherent hierarchy | `reconcile_hierarchical_data()` in [R/hierarchy.R](../../R/hierarchy.R), or `reconcile()` / `reconcile_agent_forecast()` in [R/agent_update_forecast.R](../../R/agent_update_forecast.R) | Reconcile selected source forecasts with the existing solver. No post-reconciliation future-quality ranking or model switch. | + +The within-run allowance is `best_wmape + max(0.005, 0.05 * best_wmape)`, with WMAPE expressed as a fraction. The Agent's separate search-context window is 10% relative to its minimum-WMAPE iteration. Neither is a confidence interval. Update hyperparameter retuning also has an existing 10% accuracy-degradation trigger; it is not the iteration-context rule. + +### Within-Run Call Path + +```mermaid +flowchart TD + inputs[Prepared history and existing predictions] --> restart{Complete saved winner?} + restart -->|Yes| reuse[Reuse validated selection] + restart -->|No| screen[Screen individual candidate eligibility] + screen -->|None eligible| rejected[Report selection rejection] + screen -->|Eligible candidates| averages[Form requested eligible-component averages] + averages --> evaluate[Evaluate combined candidate pool] + evaluate --> shortlist[Apply accuracy allowance] + shortlist --> rank[Risk, concerns, supported fidelity, WMAPE, ID] + rank --> chosen[Save selected identity and best-model flags] + reuse --> output[Existing output and reconciliation workflow] + chosen --> output +``` + +The coordinating function is `final_models()` in [R/final_models.R](../../R/final_models.R). It owns average construction, saved flags, intervals, and output finalization. Learned ensembles are fitted earlier by [R/ensemble_models.R](../../R/ensemble_models.R); their inputs pass `screen_ensemble_inputs()` before fitting. The shared selector itself does not fit models or write artifacts. + +Read [R/forecast_selection.R](../../R/forecast_selection.R) in these groups: + +1. `read_series_history()` / `normalize_series_history()` recover original-scale actuals, prefer R1 or use R2's `Horizon == 1`, and retain only evidence at or before the cutoff. `Target_Original` takes precedence; transformation metadata restores its scale. +2. `prepare_forecast_evaluation()` fixes the expected scenario/date keys and shares one history-only reference across candidates. `forecast_reference()` chooses supported drift or the seasonal-naive/recent-median fallback; `forecast_trend_reference()` requires two chronological historical validation blocks. These references are checks, not replacement forecasts. +3. `evaluate_forecast_candidates()` separates hard failures from soft concerns. Missing/extra/duplicate keys, non-finite predictions, catastrophic magnitude, or unavailable accuracy make a candidate ineligible. Hard-eligible candidates receive level, trend, and supported seasonal checks. +4. `rank_forecast_candidates()` shortlists on WMAPE before `order_forecast_candidates()` compares risk, concern count, supported seasonal fidelity, WMAPE, and stable ID. Fidelity participates only when every candidate in the tied risk/concern tier has an assessed value. A soft concern alone does not disqualify an ordinary run's best available candidate. +5. `validate_forecast_selection()` checks the internal selector contract: exactly one ranking row per requested candidate and either an eligible `selected_id` or `NA`. No eligible winner leads to the existing typed selection-rejection path; storage/training errors are not converted into quality rejections. + +**Use `selected_id`, not the first row of `rankings`.** The returned table includes all candidates, ordered for diagnostics, even those outside the accuracy shortlist. Its first row need not be the delivered winner. `Eligible` expresses hard validity; `Risk` is the maximum assessed soft score; `Violations` counts soft reasons; `Seasonal_Fidelity = NA` means unassessed, not perfect. A hard-rejected path can have zero risk because its soft checks were skipped. + +### Identities And Accuracy + +| Term | Meaning | +| --- | --- | +| `Combo` | One series identity, including a prepared hierarchy node when selection occurs before reconciliation. | +| `Model_ID` / `selected_id` | A candidate identity within that series/run. A simple-average ID preserves its exact component IDs; it does not mean every requested model. | +| `Best_Model = "Yes"` | The delivered candidate's flag. All individual flags can be `"No"` when a saved average wins. | +| `Best-Model` reconciled output | The coherent result of the selected mixture. Different source nodes can choose different models or averages. | +| `model_avg_wmape` | A search-direction statistic, not the accuracy of an averaged forecast. Local mean/median/standard deviation describe individual candidate WMAPEs, excluding simple-average forecasts. Existing global fields use overall run WMAPE for mean/median and zero spread. | + +`forecast_backtest_accuracy()` matches original actuals to native prediction dates, retaining overlapping backtest observations. `agent_forecast_accuracy()` instead scores selected completed-output backtests for ordinary Agent goal and saved-winner decisions. With weekly-to-daily output these can differ because of zero-actual handling and rounding. `calculate_fcst_metrics()` carries `forecast_accuracy`, `model_accuracy`, and `selection_ok` attributes into logging without another artifact read. Update logging preserves its native aggregate through `aggregate_wmape`, while completed-output rows supply per-series and local model-pool statistics. Do not collapse these distinct metric contracts into one score. + +### Agent Search And Persistence + +`best_agent_iteration()` expects chronological rows. The earliest eligible minimum-WMAPE row anchors the 10% window; it does not repeatedly widen the window after choosing another row. Among later eligible rows inside that window, a strictly lower finite `model_avg_wmape` can advance the settings context. Earlier ties remain preferred. Partial/rejected or other-version results cannot win. + +For example, adding an external regressor may improve multivariate models while ARIMA remains the winner. Lower model-pool error preserves that search direction. It does not promise future improvement or authorize overwriting a superior saved local forecast. `agent_model_accuracy()` and `record_agent_selection_attempt()` preserve those statistics and their distinct local/global meanings. + +`log_selected_agent_run()` applies the run promotion decision before writing per-series records. Global rows move as one iteration-level choice, even when an individual series gets worse; superior local winners remain protected. `validate_global_iteration()` rejects mixed global `best_run_name` values on reload/finalization/update. Exact post-write verification detects mismatched records, but the per-series writes are not a transaction. + +Ordinary accuracy-goal stopping uses complete, finite WMAPE without a second soft-quality veto. Search context, saved forecast, and stopping are separate decisions; changes to one must not silently alter the others. + +### Restarts, Updates, And Artifacts + +- `completed_forecast_selection()` checks combined saved winner content, expected keys, finite predictions, and exact saved-average arithmetic. It returns `NULL` for a reconstructable unfinished selection. Missing original average components require restoring artifacts rather than substituting a different combination. +- A complete saved winner is reused without new fitting or retrospective plausibility scoring. An unfinished selection is rebuilt from existing predictions. Weekly output is restored to native cadence, preserving invalid values from any expanded day. Reconciled-output reuse also validates content, not just filename existence or a completion log. +- An update is different: it generates new forecasts from the saved choice. `read_global_update_selection()` recovers per-series component mappings; `update_global_models()` refits only their required union from one global run. `assess_update_forecasts()` requires every component to be hard-eligible and the delivered choice to have no soft concerns, then preserves the saved identity rather than choosing a new runner-up. +- Retuning produces new predictions and therefore needs another acceptance check. Quality-rejected series enter the existing default-local replacement path separately from execution failures. A replacement must pass the stricter acceptance gate before reconciliation; no post-solver quality loop refits it again. +- `read_selection_file()` / `read_series_history()` use exact artifact paths and workflow-scoped caches. Logs are CSV even when forecast data uses another format. A genuinely missing optional average is allowed; authentication, storage, empty/corrupt required content, and deserialization failures must not masquerade as absence. +- Rankings/references remain in memory. Existing source predictions, average artifacts, selected flags, run logs, and best-run records carry the persisted evidence. Standard retrieval retains successfully reconciled per-model outputs plus the selected mixture; hierarchical Agent publication retrieves the selected mixture only. + +### Tests To Read First + +| Contract | Existing coverage | +| --- | --- | +| Original truth, eligibility, shortlist, averages, fidelity, exact history reads | [test-forecast-selection.R](../../tests/testthat/test-forecast-selection.R) | +| Historical drift support, units, seasonal alignment, reference caching | [test-forecast-selection-trend.R](../../tests/testthat/test-forecast-selection-trend.R) | +| Boundary cases, missing evidence, key completeness, deterministic ordering | [test-forecast-selection-corners.R](../../tests/testthat/test-forecast-selection-corners.R) | +| Saved-average arithmetic, combined flags, partial restarts, reconciliation repair | [test-final-models-restart.R](../../tests/testthat/test-final-models-restart.R) | +| Iteration context, local protection, one global iteration, rounded completed metrics, no repeated quality reads | [test-agent-selection-policy.R](../../tests/testthat/test-agent-selection-policy.R) and [test-load_run_results.R](../../tests/testthat/test-load_run_results.R) | +| Required update components, retained identities, native/completed metrics | [test-agent-update-selection.R](../../tests/testthat/test-agent-update-selection.R) | +| Pre-reconciliation selection, no post-solver switch, real solver behavior | [test-reconciled-forecast-selection.R](../../tests/testthat/test-reconciled-forecast-selection.R) | + +Follow the [validation matrix](validation-matrix.md) for the touched boundary. Document new or modified helpers using the [function documentation contract](../../.claude/rules/r-package.md#function-documentation), and update this map when ownership changes. Keep detailed scoring formulas in the vignette rather than duplicating them in every caller. + ## Multistep Models `R/multistep_helper.R` contains shared horizon and training-data behavior. Model adapters live in `R/multistep_*.R`; their cross-frequency and daily regression coverage lives in `tests/testthat/test-multistep*.R`. diff --git a/.github/agent-guides/evaluation-cases.md b/.github/agent-guides/evaluation-cases.md index 6d37d673..5031ac26 100644 --- a/.github/agent-guides/evaluation-cases.md +++ b/.github/agent-guides/evaluation-cases.md @@ -5,6 +5,7 @@ Use these cases after changing agent instructions, rules, skills, or discovery s | Representative request | Expected scoped context | Expected response behavior | | --- | --- | --- | | Add a documented argument to an exported function in `R/` | `r-package.md` | Preserve compatibility, update roxygen, run a focused test, regenerate documentation, and inspect generated files | +| Add or modify an internal function in `R/forecast_selection.R` | `r-package.md` and `agent-runtime.md` | Document purpose, inputs, output, invariants, and relevant errors/side effects before the definition; explain non-obvious policy blocks; do not export it merely for documentation; preserve the selection policies and update affected workflow guidance | | Change `reason_inputs()` or `finalize_run()` | `agent-runtime.md` and `r-package.md`; `testing.md` when tests are edited | Preserve typed graceful failures, hard operational errors, history separation, and bounded storage access; run Agent-focused tests | | Change a fit or predict method in `R/multistep_*.R` | `multistep.md` and `r-package.md` | Check routing, training rows, lag eligibility, forecast identity, all date frequencies, and the daily matrix | | Change `vip`, Boruta, ranger, or `nixtlar` handling | `optional-dependencies.md` and `r-package.md`; `testing.md` for CI/tests | Preserve `Suggests`, actionable missing-package behavior, core installation without the package, and CI isolation coverage | diff --git a/.github/instructions/r-package.instructions.md b/.github/instructions/r-package.instructions.md index 41d91292..e650da87 100644 --- a/.github/instructions/r-package.instructions.md +++ b/.github/instructions/r-package.instructions.md @@ -1,5 +1,5 @@ --- -applyTo: "R/**/*.R,DESCRIPTION,NAMESPACE,man/**/*.Rd,vignettes/**/*.{Rmd,qmd},NEWS.md" +applyTo: "R/**/*.R,tests/**/*.R,DESCRIPTION,NAMESPACE,man/**/*.Rd,vignettes/**/*.{Rmd,qmd},NEWS.md" --- # R Package Rules diff --git a/AGENTS.md b/AGENTS.md index 65a40cfa..8197165f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ Make changes that are testable, documented, backward-compatible where practical, Detailed rules load conditionally from `.claude/rules/`, a format shared by Claude Code and VS Code. Agents that do not load that directory automatically must read the matching rule before editing: -- `.claude/rules/r-package.md` for R source, package metadata, roxygen, vignettes, and release notes, including the single-development-version policy. +- `.claude/rules/r-package.md` for R source, documentation of every function (including internal helpers), package metadata, roxygen, vignettes, and release notes, including the single-development-version policy. - `.claude/rules/testing.md` for tests, CRAN profiles, credentials, runtime budgets, and PSOCK behavior. - `.claude/rules/agent-runtime.md` for Agent graphs, iteration selection and its accuracy signals, LLM sessions, reasoning retries, history, artifacts, combo identity, and EDA prompts. - `.claude/rules/multistep.md` for multistep adapters, lag generation, routing, and prediction. @@ -48,13 +48,15 @@ On Windows, when `R` or `Rscript` is not on `PATH`, use `./tools/run-r.ps1 -Expr - `NEWS.md`: user-visible changes and releases. - `docs/`: generated pkgdown site; do not use it for internal agent guidance. +For model selection, start with the [architecture map](.github/agent-guides/architecture-map.md#model-selection), then the [selection walkthrough](vignettes/best-model-selection.Rmd). The map distinguishes within-run model choice, Agent search context, saved-winner promotion, update acceptance, and reconciliation, with source and test pointers. + ## Working Method 1. Start from the named file, symbol, failure, or closest owning implementation. 2. State one local hypothesis and the cheapest check that could disprove it. 3. Make the smallest grounded change and run a focused executable check immediately. 4. Add or update tests for the happy path, relevant edge cases, and the regression being fixed. -5. Update roxygen or vignettes when the public API or user workflow changes, then regenerate documentation. +5. Document every new or modified function, including internal helpers, following the [function documentation contract](.claude/rules/r-package.md#function-documentation). Update roxygen or vignettes when the public API or user workflow changes, then regenerate documentation. 6. Run the broadest practical validation before finishing and report anything not run. Keep edits focused. Do not refactor unrelated code, change public APIs without approval, or overwrite user changes in a dirty worktree. @@ -88,4 +90,5 @@ State one or two plausible interpretations, their tradeoffs, and the likely file ## Code Review Rules - Prioritize behavioral regressions, unsafe error handling, compatibility breaks, optional-dependency violations, non-deterministic CRAN behavior, and missing focused tests. +- Check that new and modified functions have accurate documentation, including internal helpers; follow the function documentation contract above. - Report findings with file references and a safe correction path. Leave formatting and generated-file checks to automated validation unless they reveal a behavioral risk. diff --git a/CLAUDE.md b/CLAUDE.md index 6a12d8f4..2b5c7171 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,6 +6,6 @@ - Start nontrivial work with a concise plan naming the files likely to change. - Load and follow matching path-scoped files under `.claude/rules/`; use `/context` when diagnosing whether instructions were loaded. -- Follow `.claude/rules/r-package.md` for development-version and release-note changes. +- Follow [.claude/rules/r-package.md](.claude/rules/r-package.md) for documentation of every function, including internal helpers, and for development-version and release-note changes. Use the model-selection navigation in AGENTS.md when working on selection policy. - Keep project-specific instructions version-controlled. Put personal or machine-specific preferences in an ignored `CLAUDE.local.md`, never in this file. - Keep this file limited to Claude-specific behavior. Shared package guidance belongs in `AGENTS.md` or a scoped rule. diff --git a/R/agent_iterate_forecast.R b/R/agent_iterate_forecast.R index c74f82da..cedc9fad 100644 --- a/R/agent_iterate_forecast.R +++ b/R/agent_iterate_forecast.R @@ -2467,6 +2467,8 @@ calculate_fcst_metrics <- function(run_info, fcst_tbl, aggregate_wmape = NULL) { if (!is.null(run_info$forecast_selection) && length(run_info$forecast_selection$selections)) { + # Reuse selection completeness, but score the delivered backtests. Native + # weekly selection WMAPE can differ after daily allocation and rounding. summary <- agent_selection_summary(run_info$forecast_selection) accuracy <- agent_forecast_accuracy(fcst_tbl, names(run_info$forecast_selection$selections)) if (!is.null(aggregate_wmape)) { @@ -2477,6 +2479,8 @@ calculate_fcst_metrics <- function(run_info, if (is.finite(accuracy$weighted_mape)) accuracy$weighted_mape <- round(aggregate_wmape, 4) } value <- if (isTRUE(summary$acceptable)) accuracy$weighted_mape else Inf + # These attributes carry already-computed evidence to the logger; the scalar + # remains compatible with callers that compare it to the accuracy goal. attr(value, "selection_ok") <- isTRUE(summary$acceptable) && is.finite(value) attr(value, "forecast_accuracy") <- accuracy attr(value, "model_accuracy") <- agent_model_accuracy(fcst_tbl) @@ -2775,6 +2779,13 @@ log_best_run <- function(agent_info, return("Run logged successfully.") } +# Persist a completed iteration using its in-memory selections and metric +# attributes; run_history supplies recorded search-context evidence, not forecasts. +# Global promotion is one run-level decision; per-series guards protect local +# winners. check_best_run = FALSE supports writing an update's accepted subset. +# Writes existing run/best-run logs and returns status plus retained/written combos. +# Exact post-write reads verify identity; storage errors propagate. These writes +# are not a transaction, so reload/update must also reject mixed global metadata. log_selected_agent_run <- function(agent_info, run_info, combo = NULL, check_best_run = TRUE, weighted_mape = NULL, run_history = NULL) { current_log <- read_selection_file(run_info, "logs") @@ -2840,6 +2851,8 @@ log_selected_agent_run <- function(agent_info, run_info, combo = NULL, check_bes } written <- character() retained <- character() + # Apply the global gate uniformly. A single global run may improve the overall + # result while worsening one series; never mix its global rows with an old run. for (series in names(current_result$selections)) { selected <- current_result$selections[[series]] if (is.null(selected) || is.na(selected$selected_id)) next @@ -2865,6 +2878,8 @@ log_selected_agent_run <- function(agent_info, run_info, combo = NULL, check_bes } same_version <- nrow(previous) == 1 && isTRUE(as.numeric(previous$agent_version) == as.numeric(agent_info$agent_version)) + # Advancing search settings is not permission to overwrite a superior local + # forecast. Existing global rows instead follow the shared promotion gate. protect_individual <- !global || !identical(as.character(previous$model_type), "global") if (same_version && protect_individual && isTRUE(is.finite(as.numeric(previous$weighted_mape))) && isTRUE(as.numeric(previous$weighted_mape) <= score$WMAPE)) { diff --git a/R/agent_update_forecast.R b/R/agent_update_forecast.R index 7319201b..471f9cee 100644 --- a/R/agent_update_forecast.R +++ b/R/agent_update_forecast.R @@ -1856,6 +1856,9 @@ update_forecast_combo <- function(agent_info, selection_log <- read_selection_file(new_run_info, "logs") selection_log$negative_forecast <- prev_run_log_tbl$negative_forecast selection_cache <- new.env(parent = emptyenv()) + # This is acceptance of new predictions for the saved choice, not a fresh + # candidate search. Assess source components before any hierarchy solve; + # return quality rejections to the existing default-local recovery workflow. assessment <- assess_update_forecasts( final_fcst_tbl, new_run_info, selection_log, model_train_test_tbl, expected_components = if (is.null(selected_models)) model_id_list else selected_models$components, @@ -1900,6 +1903,7 @@ update_forecast_combo <- function(agent_info, selected_models = selected_models ) + # Retuning created a new path, so the first acceptance result cannot be reused. assessment <- assess_update_forecasts( final_fcst_tbl, new_run_info, selection_log, model_train_test_tbl, expected_components = if (is.null(selected_models)) model_id_list else selected_models$components, diff --git a/R/final_models.R b/R/final_models.R index 6b49989b..7b4e9e3a 100644 --- a/R/final_models.R +++ b/R/final_models.R @@ -314,6 +314,9 @@ final_models <- function(run_info, saved_average <- read_selection_file(run_info, "forecasts", "-average_models", combo_name, optional = TRUE) saved_rows <- dplyr::bind_rows(native_forecast_rows(all_model_tbl, date_type), if (average_models) native_forecast_rows(saved_average, date_type)) + # Completion belongs to the combined artifacts: all individual flags can + # be No when an average is the winner. Reuse only validated content; + # otherwise rebuild selection below from saved predictions, without fits. existing_selection <- completed_forecast_selection(saved_rows, series_data, model_train_test_tbl) if (!is.null(existing_selection)) { return(selection_worker_result(combo_name, existing_selection, reused = TRUE)) @@ -430,6 +433,8 @@ final_models <- function(run_info, final_model_list <- c(local_model_list, global_model_list) + # Screen components before forming averages. Soft concerns do not remove + # a component here, but averaging must never conceal a hard-invalid path. individual_selection <- select_series_forecasts( predictions_tbl, series_data, model_train_test_tbl, unique(predictions_tbl$Model_ID) @@ -536,6 +541,8 @@ final_models <- function(run_info, if (!is.null(averages_tbl)) unique(averages_tbl$Model_ID) else character(0) )) + # Rank the delivered candidate pool once it includes requested averages. + # A standard run may keep the best available candidate with soft concerns. selection <- select_series_forecasts( final_predictions_tbl, series_data, model_train_test_tbl, eligible_model_ids ) @@ -543,6 +550,8 @@ final_models <- function(run_info, abort_forecast_selection(unique(predictions_tbl$Combo), selection) } if (!is.null(averages_tbl)) { + # Retain one average artifact even when an individual wins. Rank this + # subset with its own accuracy allowance, not by minimum WMAPE alone. average_selection <- rank_forecast_candidates(selection$rankings[ selection$rankings$Model_ID %in% averages_tbl$Model_ID, , drop = FALSE ]) diff --git a/R/forecast_selection.R b/R/forecast_selection.R index 06b01a64..0211182a 100644 --- a/R/forecast_selection.R +++ b/R/forecast_selection.R @@ -1,3 +1,10 @@ +# Shared selection policy: score existing predictions, then choose one candidate +# per series. Callers own fitting, averaging, artifact writes, and reconciliation. +# Read vignettes/best-model-selection.Rmd for the rules and examples, and +# .github/agent-guides/architecture-map.md for workflow owners and regression tests. + +# Build original-scale evidence at the historical cutoff. R2 repeats each origin +# across horizons, so only Horizon 1 contributes a unique historical observation. normalize_series_history <- function(data, hist_end_date, recipe = "R1", combo_info = NULL, stationary = FALSE, box_cox = FALSE) { @@ -54,6 +61,8 @@ normalize_series_history <- function(data, hist_end_date, recipe = "R1", list(history = history, calendar = calendar, hist_end_date = as.Date(hist_end_date)) } +# One run/series cache entry serves every candidate; known artifact names are +# read directly, not discovered again inside the candidate loop. read_series_history <- function(run_info, combo, run_log = NULL, cache = NULL) { if (length(combo) != 1 || is.na(combo)) { stop("Read prepared history for exactly one series at a time.", call. = FALSE) @@ -158,6 +167,8 @@ finite_mad <- function(values) { if (length(values)) stats::mad(values) else 0 } +# Estimate a robust same-phase drift and seasonal intercepts, not another model +# candidate. NULL means the history does not support a stable nonzero drift. forecast_trend_fit <- function(values, period, horizon) { if (length(values) < max(8, 2 * period) || any(!is.finite(values))) return(NULL) slopes <- (values[-seq_len(period)] - utils::head(values, -period)) / period @@ -177,6 +188,9 @@ forecast_trend_fit <- function(values, period, horizon) { projection = projection, values = values) } +# Admit a history-only additive/log reference only after two prefix holdouts +# beat the no-drift baseline. Unsupported or unrepresentable growth returns NULL +# so forecast_reference() retains its seasonal-naive/recent-median fallback. forecast_trend_reference <- function(history, horizon, context, normalization, scale) { period <- forecast_seasonal_period(context) history_size <- nrow(history) @@ -428,6 +442,8 @@ forecast_path_risk <- function(forecasts, reference) { seasonal_fidelity = seasonal_fidelity) } +# All candidates share the same expected split/date keys and history-only +# reference. Future Target placeholders must never become evaluation evidence. prepare_forecast_evaluation <- function(history, context) { history$Date <- as.Date(history$Date) cutoff <- as.Date(context$hist_end_date) @@ -443,6 +459,8 @@ prepare_forecast_evaluation <- function(history, context) { reference = forecast_reference(history, nrow(expected_forecasts), context)) } +# Match original actuals by date without collapsing overlapping backtest splits. +# Preserve the zero-to-0.1 and pointwise four-decimal conventions used by Finn. forecast_backtest_accuracy <- function(history, backtest) { actuals <- history$Target[match(as.Date(backtest$Date), as.Date(history$Date))] target <- ifelse(actuals == 0, 0.1, actuals) @@ -459,6 +477,8 @@ forecast_backtest_accuracy <- function(history, backtest) { list(WMAPE = weighted_mape, Log_Weight = log_weight) } +# Hard failures control Eligible; soft path concerns only affect ranking. +# Log_Weight carries backtest target mass for stable aggregation across series. evaluate_forecast_candidates <- function(history, backtests, forecasts, context) { evaluation <- context$forecast_evaluation %||% prepare_forecast_evaluation(history, context) history <- evaluation$history @@ -499,6 +519,7 @@ evaluate_forecast_candidates <- function(history, backtests, forecasts, context) } accuracy <- forecast_backtest_accuracy(history, backtest) if (!is.finite(accuracy$WMAPE)) reasons <- c(reasons, "unavailable_accuracy") + # A hard-invalid path is not soft-scored. Zero risk here is not eligibility. risk <- forecast_path_risk(if (length(reasons) == 0) future$Forecast else numeric(), reference) tibble::new_tibble(list( Model_ID = candidate_id, Eligible = length(reasons) == 0, @@ -522,6 +543,8 @@ order_forecast_candidates <- function(rankings) { groups <- split(seq_along(indices), cumsum(c(TRUE, !same_tier))) for (positions in groups) { fidelity <- ordered$Seasonal_Fidelity[positions] + # Compare fidelity only with equal evidence in the whole risk/concern tier. + # An unassessed seasonal check is NA, not a perfect zero-distortion score. if (all(is.finite(fidelity) & fidelity >= 0)) { indices[positions] <- indices[positions][order(fidelity, ordered$WMAPE[positions], ordered$Model_ID[positions], method = "radix", na.last = TRUE)] @@ -541,12 +564,16 @@ rank_forecast_candidates <- function(rankings) { selected_id <- NA_character_ if (nrow(eligible) > 0) { best_accuracy <- min(eligible$WMAPE) + # WMAPE is a fraction: allow 0.5 percentage points or 5% relative, whichever + # is larger. This is not the Agent's separate 10% iteration-context rule. ceiling <- best_accuracy + max(0.005, 0.05 * best_accuracy) tolerance <- 8 * .Machine$double.eps * max(1, abs(ceiling)) shortlist <- eligible[eligible$WMAPE <= ceiling + tolerance, , drop = FALSE] shortlist <- shortlist[order_forecast_candidates(shortlist), , drop = FALSE] selected_id <- shortlist$Model_ID[1] } + # Keep all candidates for diagnostics. Consumers must use selected_id, not + # rankings[1, ], because this full ordering is not restricted to the shortlist. rankings <- rankings[order_forecast_candidates(rankings), , drop = FALSE] list(selected_id = selected_id, rankings = rankings) } @@ -555,6 +582,8 @@ select_forecast_candidate <- function(history, backtests, forecasts, context) { rank_forecast_candidates(evaluate_forecast_candidates(history, backtests, forecasts, context)) } +# Storage-free adapter: split one series' existing predictions by run type and +# require exactly one ranking row per requested ID from the internal selector. select_series_forecasts <- function(predictions, series_data, splits, candidate_ids = unique(predictions$Model_ID), selector = select_forecast_candidate) { @@ -690,6 +719,10 @@ read_final_predictions <- function(run_info, combo_hash, suffix) { rows } +# Restart contract: reuse one complete saved winner, including the exact mean +# of its original components when it is an average. NULL requests reconstruction +# from existing predictions; missing average components require artifact repair. +# This validates content and accuracy, not a fresh future-plausibility decision. completed_forecast_selection <- function(predictions, series, splits) { required <- c("Best_Model", "Model_ID", "Train_Test_ID", "Date", "Forecast", "lo_80", "lo_95", "hi_80", "hi_95") @@ -729,6 +762,9 @@ unfinalized_forecast_rows <- function(rows, date_type) { rows[, setdiff(names(rows), c("Best_Model", "lo_80", "lo_95", "hi_80", "hi_95", "Run_Type")), drop = FALSE] } +# Bookkeeping for an already selected path: validate identity/coverage/finite +# values and compute backtest accuracy. Quality fields stay NA because they were +# not reassessed; reconciled backtest reporting may omit the future-key check. selected_forecast_accuracy <- function(predictions, series, splits, require_forecast = TRUE) { series$train_test_split <- splits selected <- if ("Best_Model" %in% names(predictions)) { @@ -796,6 +832,8 @@ assess_agent_run <- function(run_info, run_log, combos, cache = new.env(parent = result } +# Keep pre-reconciliation source quality separate from delivered bottom-level +# backtest accuracy. Never score reconciled future shapes or choose new models. hierarchical_selection_result <- function(run_info, run_log, source_selections, forecasts, splits, combos = NULL, cache = new.env(parent = emptyenv())) { hierarchy <- read_selection_hierarchy(run_info, cache) @@ -813,6 +851,10 @@ hierarchical_selection_result <- function(run_info, run_log, source_selections, rejected_combos = names(selections)[vapply(selections, function(selection) is.na(selection$selected_id), logical(1))]) } +# Missing entries in the supplied selection set make it partial/rejected. +# Callers must include the full expected series set, including missing selections. +# Ordinary iteration decisions require completeness and finite accuracy only; +# update/default acceptance explicitly opts into the stricter quality boundary. agent_selection_summary <- function(result, check_quality = FALSE) { chosen <- lapply(result$selections, function(selection) { if (is.null(selection) || is.na(selection$selected_id)) return(NULL) @@ -945,6 +987,10 @@ read_selection_hierarchy <- function(run_info, cache = NULL) { hierarchy } +# Assess newly refitted predictions while preserving the saved model identity. +# Every required component must be hard-eligible and the delivered choice must +# have no soft concerns. Return rejected combo hashes for default recovery; +# a hierarchy is reconciled only after all its source nodes pass. assess_update_forecasts <- function(forecasts, run_info, run_log, splits, expected_components = NULL, cache = new.env(parent = emptyenv()), combos = NULL) { @@ -973,6 +1019,7 @@ assess_update_forecasts <- function(forecasts, run_info, run_log, splits, delivered <- selection$rankings[selection$rankings$Model_ID %in% selected_ids, , drop = FALSE] if (hard_pass && nrow(delivered) == 1 && delivered$Violations == 0) { accepted <- c(accepted, combo) + # Assessment must not replace the saved combination with today's runner-up. selection$selected_id <- selected_ids } else { rejected <- c(rejected, combo) @@ -1000,6 +1047,11 @@ assess_update_forecasts <- function(forecasts, run_info, run_log, splits, result } +# Rank iteration context from recorded metrics. Callers also use this ranking to +# gate promotion, with separate protection for saved local winners. +# Callers supply chronological rows: earliest minimum WMAPE is the anchor; +# a later near-best result can advance search when its model-pool mean improves. +# Return the original row index, or NA when no eligible iteration exists. best_agent_iteration <- function(run_logs, agent_version = NULL) { if (!is.data.frame(run_logs) || !nrow(run_logs) || !"weighted_mape" %in% names(run_logs)) { return(NA_integer_) @@ -1018,6 +1070,7 @@ best_agent_iteration <- function(run_logs, agent_version = NULL) { winner <- eligible[which.min(accuracy[eligible])] average <- as.numeric(run_logs[["model_avg_wmape"]]) if (length(average) == nrow(run_logs) && is.finite(average[winner])) { + # Keep the window anchored to the minimum, not a chain of 10% relaxations. later <- eligible[eligible > winner & abs(accuracy[eligible] - accuracy[winner]) <= accuracy[winner] * 0.10 & is.finite(average[eligible]) & average[eligible] < average[winner]] @@ -1026,6 +1079,8 @@ best_agent_iteration <- function(run_logs, agent_version = NULL) { as.integer(winner) } +# Global updates replay one run's settings. Different models within that run +# and separate local winners are valid; mixed global run identities are not. validate_global_iteration <- function(best_runs) { if (!is.data.frame(best_runs) || !nrow(best_runs) || !"model_type" %in% names(best_runs)) { return(invisible(best_runs)) @@ -1040,6 +1095,10 @@ validate_global_iteration <- function(best_runs) { invisible(best_runs) } +# Score selected completed-output backtests at their delivered cadence, including +# daily-expanded weekly output. Do not substitute native selector WMAPE: the +# zero-actual convention and rounding can change goal and promotion decisions. +# Missing/unusable series retain Inf and prevent an aggregate success. agent_forecast_accuracy <- function(forecasts, combos) { accuracy <- list(weighted_mape = Inf, by_series = stats::setNames(rep(Inf, length(combos)), combos)) @@ -1065,6 +1124,9 @@ agent_forecast_accuracy <- function(forecasts, combos) { accuracy } +# Local search-direction statistics summarize individual candidate WMAPEs, not +# the WMAPE of a simple-average forecast. Keep unavailable evidence as NA; +# record_agent_selection_attempt() handles the distinct global summary meaning. agent_model_accuracy <- function(forecasts) { unavailable <- list(model_avg_wmape = NA_real_, model_median_wmape = NA_real_, model_std_wmape = NA_real_) diff --git a/vignettes/best-model-selection.Rmd b/vignettes/best-model-selection.Rmd index 887d082c..2128d5b1 100644 --- a/vignettes/best-model-selection.Rmd +++ b/vignettes/best-model-selection.Rmd @@ -16,6 +16,33 @@ knitr::opts_chunk$set( The `Best_Model` flag identifies the selected forecast for each series. Finn first checks candidate validity, then balances backtest weighted MAPE with future-forecast plausibility. Individual models, learned ensembles, and simple averages are evaluated on the same series and expected backtest dates. +## Process at a glance + +Model selection is several connected decisions, not one minimum-error calculation: + +| Stage | Question answered | Result | +| --- | --- | --- | +| Candidate validity | Does each model have complete, finite predictions and usable backtest accuracy? | Hard-invalid candidates cannot win or contribute to a simple average. | +| Within-run selection | Among sufficiently accurate candidates, which future path ranks best by risk, concern count, supported seasonal fidelity, WMAPE, and stable ID? | One selected model or exact average for each series. Soft concerns can remain when no better eligible alternative exists. | +| Agent iteration context | Which completed iteration's settings should guide further optimization? | A near-best iteration can be preferred when its individual model-pool accuracy improves. This does not automatically replace a better saved local forecast. | +| Saved forecast promotion | Should the current iteration replace the saved result? | Local winners stay protected; globally selected series reference one winning global iteration. | +| Update acceptance | Is the newly refitted version of the saved choice still acceptable? | Keep that choice or attempt the existing default-local replacement workflow before reconciliation. | + +Seasonal fidelity breaks a risk-and-concern tie only when every candidate in that +tied group has a finite assessed score; otherwise WMAPE and stable ID decide the tie. + +For a standard run, follow [candidate checks](#future-forecast-checks), the +[accuracy allowance](#accuracy-allowance), and +[selection and saved averages](#standard-forecasts-and-saved-averages). +The [Agent](#iterative-agent-forecasts) and [update](#updated-agent-forecasts) +sections describe the additional decisions. [Reconciliation](#reconciliation) +makes the selected hierarchy coherent; it does not select the models again. + +Two uses of "average" are important: a **simple-average forecast** combines +predictions from selected components, while **average model accuracy** summarizes +individual model WMAPEs to guide Agent search. Improving the latter can be useful +even when the winning model has not changed. + ## Future-forecast checks These checks apply to candidate base forecasts before reconciliation, including averages. Candidates with incomplete or non-finite predictions are ineligible. Backtests and future predictions without a supported trend are rejected when their absolute magnitude exceeds 100 times a positive robust historical scale. That scale uses the 95th percentile of absolute levels and robust level/change variation, not just the last observation. A supported trend supplies a pointwise future bound of `100 * max(robust_scale, abs(projected_reference))`, so sustained growth is not rejected merely for accumulating over a long horizon. The projection comes only from history, never from the candidate being screened. Existing finite-negative handling still follows `negative_forecast`, but missing or infinite predictions are no longer replaced with zero. @@ -113,6 +140,13 @@ Choosing an iteration's settings for further optimization is distinct from repla Normal accuracy-goal stopping and local optimization routing use completeness and WMAPE, without another soft-quality veto. Consequently, an iteration winner with soft concerns can beat an earlier winner or meet the accuracy goal; hard-invalid and incomplete results still cannot claim successful completion. Rejected runs consume iteration budget. At the limit, an eligible best-available result may be retained with concerns; if none exists, Finn fails or uses an already enabled local phase for unresolved global series. Final outer reconciliation does not initiate another quality-selection or refitting loop. Avoided evaluator calls and artifact reads are covered by tests; no fixed runtime reduction is promised. +For ordinary Agent iterations, the goal and saved-forecast comparisons use selected +completed-output backtests, including daily-expanded rows when weekly forecasts +are delivered daily. Within-run candidate selection stays at native cadence. +Those scores are not interchangeable: the zero-actual convention and rounding can +produce different WMAPEs after daily allocation. Future target placeholders never +contribute to either accuracy calculation. + ## Updated Agent forecasts `update_forecast()` follows a different fitting path but uses the same evaluator: @@ -126,6 +160,11 @@ Normal accuracy-goal stopping and local optimization routing use completeness an The optional accuracy-degradation `allow_iterate_forecast` workflow remains separate. Checks after refit or retune assess newly generated predictions, not old iteration winners. A default's saved acceptance or rejection prevents repeating that acceptance decision after it has completed; an interrupted acceptance step can still assess the new output once. Quality recovery itself does not ask the LLM to waive a check. Default refitting can increase compute or configured-provider cost, and it is a bounded recovery attempt rather than a promise of a satisfactory forecast. +Update accuracy keeps the existing native-cadence aggregate used for refit/retune +and aggregate logging. Completed-output backtests supply per-series saved-winner +accuracy and local model-pool statistics. This distinction preserves the update +workflow's metric meanings when weekly output is expanded to daily rows. + ## Evidence and limitations Developer tests compare two paths on the same fake candidate outputs: accuracy-only selection followed by real `hts`, and quality-aware selection/averaging followed by real `hts`. Held-out future truth is used only by the tests, never supplied to the selector. They measure bottom-level error and shape, including effects on uninjected siblings. Small regressions run with package tests; broader stress and full-catalogue averaging timings are separate developer experiments.