ARF is kept running as a "backup signal" (ADR-006, 2026-05-29). On the failure mode that has cost this pipeline the most vintages — stale upstream data — it does not act as one, because both models share the cause and ARF degrades worse than the thing it backs up.
Evidence (2026-09-10 EDH stall)
LightGBM is data-anchored. predict_72h builds features for one row at t0 and gets all 72 horizons from horizon-as-feature stacking, so it never needs forward exogenous. With a stale parquet it re-emits from the old anchor — verified byte-identical against the previous run (forecast, forecast_upper, forecast_lower all identical; only last_updated moved).
ARF is wall-clock anchored. generate_forecast loops from datetime.now(), and OnlineFeatureBuilder.build returns None unless it has both a 1h and 24h price lag (ml/features/online_features.py:112). With prices ending 09-09T20:00 and the run at 09-10T03:01, the 1h lag is missing at the very first hour; nothing is pushed back into the buffer, so every subsequent hour fails identically. Result: forecast: {}, 0 of 72 hours — the [ALARM: ARF forecast 0h] seen on 09-01..09-04 and again on 09-10.
So production degrades to stale and the backup degrades to nothing.
Three readings, and they point different ways
- It was never an availability backup — ARF is a modelling fallback and still feeds the Model-tab widgets, which is the job it was kept for. Then the label is imprecise and the wall-clock anchor is an ordinary bug. Checkable from ADR-006 /
docs/river-arf-retrospective.md alone.
- The anchor is trivially fixable — anchor ARF on
parquet.index.max() like the shadow path and it degrades to stale rather than empty. Under an hour.
- Stop running it. ARF was retired as a model 2026-04-28 and kept for one rolling-window cycle; that cycle is long over. If the Model-tab widgets can be sourced from
eval_log.jsonl, the honest move is retirement, not repair — and daily_update.sh loses a step.
Decide between 2 and 3 before doing any work — they point opposite ways and the cost difference is an hour versus a deletion. Read the ADR first; option 1 may close this outright.
Not urgent: ARF is non-fatal in daily_update.sh (set +e since 2026-07-03) and cannot abort the production path.
Position: docs/hypothesis-log.md [2026-09-10]. Review by 2026-10-01.
ARF is kept running as a "backup signal" (ADR-006, 2026-05-29). On the failure mode that has cost this pipeline the most vintages — stale upstream data — it does not act as one, because both models share the cause and ARF degrades worse than the thing it backs up.
Evidence (2026-09-10 EDH stall)
LightGBM is data-anchored.
predict_72hbuilds features for one row att0and gets all 72 horizons from horizon-as-feature stacking, so it never needs forward exogenous. With a stale parquet it re-emits from the old anchor — verified byte-identical against the previous run (forecast,forecast_upper,forecast_lowerall identical; onlylast_updatedmoved).ARF is wall-clock anchored.
generate_forecastloops fromdatetime.now(), andOnlineFeatureBuilder.buildreturnsNoneunless it has both a 1h and 24h price lag (ml/features/online_features.py:112). With prices ending 09-09T20:00 and the run at 09-10T03:01, the 1h lag is missing at the very first hour; nothing is pushed back into the buffer, so every subsequent hour fails identically. Result:forecast: {}, 0 of 72 hours — the[ALARM: ARF forecast 0h]seen on 09-01..09-04 and again on 09-10.So production degrades to stale and the backup degrades to nothing.
Three readings, and they point different ways
docs/river-arf-retrospective.mdalone.parquet.index.max()like the shadow path and it degrades to stale rather than empty. Under an hour.eval_log.jsonl, the honest move is retirement, not repair — anddaily_update.shloses a step.Decide between 2 and 3 before doing any work — they point opposite ways and the cost difference is an hour versus a deletion. Read the ADR first; option 1 may close this outright.
Not urgent: ARF is non-fatal in
daily_update.sh(set +esince 2026-07-03) and cannot abort the production path.Position:
docs/hypothesis-log.md[2026-09-10]. Review by 2026-10-01.