Point-forecast skill floor: production does not reliably beat seasonal-naive (verdict pre-committed, ~2026-09-27)

[ducroq]

What

Every comparison in this project is relative — eval_log.jsonl scores LightGBM vs ARF, experiments/registry.jsonl scores variants vs production, metrics.diebold_mariano compares two candidates. None can see the case where every arm is worse than a trivial baseline; that reads as a healthy log with a clear winner.

Measured 2026-09-06 against seasonal-naive ("same clock hour, last day available at t0"):

Live record — horizon-matched, production p50, 24 eval days, t0 ∈ [2026-07-30, 2026-08-24]:

horizon	n	LGBM p50	naive	winner
1–24h	576	26.92	23.34	naive
25–48h	576	33.87	28.94	naive
49–72h	552	36.92	32.71	naive

Offline, 260 vintages (EXP-035, re-scoring stored EXP-018/EXP-021 predictions):

arm	MAE	naive	skill	DM p
chronos_bolt_base	22.72	30.31	+0.250	<1e-6
lgbm_drop_rolling	27.07	30.31	+0.107	3.2e-5
lgbm_full (production)	28.76	30.31	+0.051	0.033
lgbm_drop_calendar	30.82	30.31	−0.017	0.720

The precise claim

Not "the model has no skill" — across the full year it edges naive by 5%. The claim is narrower and worse: the edge is regime-dependent and collapses where a forecast is worth having. lgbm_full is below naive in 2 of 9 months, and in August 2026 (the current live regime, mean 131.9 EUR/MWh) all eight LightGBM variants are below naive. Chronos-bolt-base is above naive in 9 of 9 months.

Shipped

evaluate_shadow.py now logs n_naive_hours, naive_mae, lightgbm_mae_on_naive_hours, lightgbm_skill_vs_naive, naive_min/max_horizon_h per eval row (89d4ac1). update_shadow.py records t0_utc per prediction so the anchor is never re-derived.

Open

Verdict is pre-committed in docs/hypothesis-log.md [2026-09-06] on ≥21 rows carrying non-null lightgbm_skill_vs_naive (≈2026-09-27), with buckets at skill <0, [0, 0.05], >0.05. EXP-035 does not discharge it — same offline vintages EXP-021 used, and its LightGBM arms are harness rebuilds rather than the deployed model.

Untested alternative, and the cheapest next check: Augur exists for load shifting, where ranking the cheap hours matters more than the level. A rank-correlation or cheapest-k-hours hit-rate metric might show the model ahead of naive where MAE says otherwise — that would make this issue's verdict narrow rather than wrong, and would mean the dashboard should be scored on that instead.

Reproduce: python scripts/exp035_naive_floor.py

Related: #15 (the model class that clears the floor), #19 (calibration is the second problem), #28 (drop_rolling is the best arm against the floor)

[ducroq]

EXP-036 pre-committed 2026-09-10 (docs/experiment-backlog.md entry 9, pinned at 2113e1f): the floor this issue is measured against is a single-day carry — 24*ceil(h/24) hours back, one sample per forecast hour. The obvious strengthening is a profile: the mean of the same clock hour over the last N days, which should cut the baseline's variance ~1/K while keeping the diurnal shape that actually repeats.

Position: against the best profile arm, lgbm_full's +5.1% falls to ≤2% overall and goes negative in ≥4 of 9 months (from 2 of 9). Arms: N1 (current floor), N2/N3/N5/N7 unweighted, a half-life-2 decayed arm, and a day-type-matched arm. Runs on the 260 stored vintages — minutes of CPU, no GPU, no fresh vintages, so it does not compete with EXP-018a/021a/028a.

This does not move this issue's verdict retroactively. The pre-commitment here fires on ≥21 fresh vintages against the floor as specified on 2026-09-06, whatever EXP-036 finds — re-reading an already-pre-committed criterion against a floor chosen later is the post-hoc criterion swap ADR-007 forbids. If a profile arm is adopted it becomes an additional field (naive_profile_mae), never a replacement, so pre-2026-09-10 rows stay comparable, and both numbers get reported side by side for at least one review cycle.

⚠️ Schedule slip: two further vintages are gone — EDH published nothing on 09-07 (ENTSO-E outage) or 09-09 (a NED.nl timeout that aborted the publish of all 19 healthy feeds upstream). No backfill exists for either. Count vintages, do not read dates.

[ducroq]

Two pre-committed checks ran on 2026-09-11. Both made this finding worse rather than narrower.

EXP-037 — the product lens is the harsher one

Alternative 3 of the hypothesis-log entry was the one that could have narrowed this: Augur exists so a load runs during the cheap hours, which is a ranking problem, and ranking within a day is invariant to exactly the level error this model makes. If it ranked well while missing levels, the fix would have been presentation, not prediction.

Scored as cost regret in EUR/MWh — what you actually pay running a load for the k hours a forecaster calls cheapest, against perfect foresight — on 755 blocks from 253 stored vintages:

arm	h1-24	h25-48	h49-72	pooled	vs carry	DM p
carry (baseline)	4.97	5.30	4.75	5.01	—	—
chronos-bolt-base	2.02	2.63	2.74	2.46	+2.55	<1e-6
lgbm_full (production)	4.13	9.15	10.69	7.99	−2.98	0.999987

Worse in 2 of 3 horizon groups, worse in 6 of 9 months, Spearman 0.718 vs the carry's 0.787. My pre-committed position predicted the opposite in both direction and magnitude; it is recorded unedited.

Mechanism — amplitude collapse. Within-block forecast spread falls from 26.6 EUR/MWh at 1-24h to 14.9 at 49-72h, against a realised spread of 42.7 and a carry holding 41.1. Beyond a day the model forecasts about a third of the true diurnal amplitude. A flattened forecast has a better MAE — hedging toward the conditional median is what minimising absolute error buys — and a nearly arbitrary ranking. The training objective is buying MAE by destroying the product value.

Not an artefact of the shipped column: the raw tau=0.50 output scores 7.95.

EXP-036 — the floor itself was too weak

Every multi-day profile beats the single-day carry this issue is measured against, monotonically in K. The adopted arm (same clock hour, last 5 matching day types) beats it by 15.8% — MAE 24.35 vs 28.92, DM p<1e-5.

So production's +5.1% over the carry becomes −12.4% below an honest floor, and months below go from 2 of 9 to 7 of 9. drop_rolling is also below (−5.5%). Chronos clears every arm in every month.

What changed in the criterion, and what did not

Nothing. EXP-036's Alternative 3 fired (the profile's gain is horizon-dependent, +7.1% at 1-24h vs +19.8% at 49-72h) and its pre-committed consequence says restate this issue's threshold per horizon group. That collides with this issue's own boundary, and the collision resolves against changing anything now — re-specifying a criterion while its data is arriving is the post-hoc swap ADR-007 forbids.

• The ≥21 verdict fires as written: mean skill, against the single-day carry, in the three buckets.
• It is reported alongside the per-horizon split and the profile-floor skill. evaluate_shadow.py now logs naive_profile_mae and lightgbm_skill_vs_naive_profile additively (EXP-036 gate G2), so pre-2026-09-11 rows stay comparable.
• Any successor criterion is per-group.

Counting

Now 2 of 21 qualifying rows, not 3. The 2026-09-09 row was scored on a single paired hour at horizon 25, scoring +0.762 — the most flattering number in the log, from one draw, and at the time a third of the entire evidence base. A ≥12-paired-hour floor now applies to both the trigger and the nightly 7-row window, and the daily job prints Seasonal-naive verdict trigger: N/21 qualifying rows.

The ~2026-09-27 date assumed one row/day from 09-06. It is optimistic.