EXP-018: feature-set ablation → reduction bet (EXP-018a); supersedes #2/#3/#4/#22

[ducroq]

Context

Production LightGBM-Quantile uses 24 features from 4 columns (price_eur_mwh, wind_speed_80m, solar_ghi, load_forecast). energyDataHub already collects ~7 more series that never reach consolidate.py: ned_production, grid_imbalance, cross_border_flows, gas_storage, gas_flows, market_proxies, generation_forecast.

Scope

Consolidate the individual feature issues (#2, #3, #4, #22) into one evaluated experiment, per the provisional position in docs/hypothesis-log.md (2026-08-20).

Two stages:

1. Stage 0 — per-feature-group ablation (cheap, diagnostic): drop lags / rolling / calendar / wind / solar / load, measure MAE + quantile score + coverage.
2. Stage 1 — walk-forward backtest: incumbent (24 features, 3 quantiles) vs candidate (24 + residual load, generation mix, volatility, calendar-gaps), temporal split, 56-day window, vintage-corrected.

Pre-committed gates (ADR-007): calibration — lower AND upper coverage ∈ [0.85, 0.95] AND Winkler ≤ 1.05× incumbent; skill guardrail — paired DM on |y−p50| (α=0.10, HAC=71) not worse.

Prerequisites

• consolidate.py parsers for the 7 unused series (currently hardcoded to 4).
• features_pandas.py + update_shadow.py + online_features.py in lockstep.
• Watch the known freshness skew (consolidate overwrites rows with later vintages → backtest is optimistic).

Compute

LightGBM feature work is CPU-cheap; not gated on GPUs. Three GPU hosts are available over Tailscale (jwasys-b650-eagle-ax.taileb31bb.ts.net, sadaltager, gpu-server.tailce506.ts.net) — held for a possible neural-quantile track, not this.

Status: deferred — pick up next session. See docs/hypothesis-log.md (2026-08-20) and augur#19.

[ducroq]

Stage 0 ran 2026-08-25. The premise of this issue is refuted — the lever is subtraction, not addition.

Harness: scripts/exp018_stage0_ablation.py — production-shaped walk-forward (t0 = last clean feature row per vintage day, 56-day window, h+1..h+72, no CQR), 263 vintages 2025-12-01..2026-08-22, 3682 fits across two sweeps. Records in ml/shadow/exp018_stage0*/summary.json; registry entries EXP-018 (kept) and EXP-019 (rejected).

variant	nfeat	MAE	ΔMAE%	QS	ΔQS%	cov_lo
full	24	28.97	—	10.54	—	0.778
drop_lags	16	28.75	−0.8	10.53	−0.1	0.776
drop_rolling	18	27.24	−6.0	9.72	−7.8	0.805
drop_calendar	17	31.08	+7.3	11.51	+9.2	0.773
drop_wind / solar / load	23	~28.9	−0.3/−0.3/−0.4	~10.5	~0	~0.777
drop_exog	21	29.01	+0.1	10.62	+0.8	0.773
lean (rolling+exog out)	15	27.08	−6.5	9.69	−8.1	0.810

• The six rolling-stat features actively hurt: removing them buys −6.0% MAE / −7.8% quantile score and better lower-side coverage. DM p<0.0001 on both QS and |error| (HAC 71), wins in 7 of 9 months, all three horizon groups, biggest in the volatile recent regime (May −14%, Jul −11%, Aug −9%); costs only Dec 2025 (+5.8%).
• Calendar is the only group clearly earning its place.
• The exogenous trio is inert — ±0.4% each, despite sensible correlations over the last 120 days (load 0.59, solar −0.53). This is what refutes the "add more fundamentals" premise: the model extracts nothing from the three series it already has, so Add NED actual production as model feature #2/Add gas/carbon price history as model features #3/Add cross-border flows and grid imbalance as lag features #4/EXP-018 candidate: HDD/CDD as long-horizon demand features (h+25..h+72) #22 are weak bets until that changes. Demoting them accordingly (comment posted on each).
• EXP-019 tested the obvious alternative — that the lever is stationarity, not removal. Anchor-relative spreads (price_lag_Hh − price_rolling_mean_168h) tie plain deletion (QS p=0.405, |error| p=0.160), while re-adding price_rolling_mean_168h as the single level column costs most of the gain (p=0.0001). Mechanism revised: raw lags are absolute levels too and are harmless, so redundant smoothed-level columns diluting the split search fits better than pure drift.

Nothing is deployed. The lean set is a best-of-eight selection on one window, so it ships only through the pre-committed EXP-018a entry in docs/hypothesis-log.md — four gates (DM one-sided p<0.10 on QS, ≥3% QS effect survives, per-side coverage not >0.02 worse, Winkler ≤1.05×) on fresh vintages with t0 ≥ 2026-08-25 (≈2026-09-09), then a 14-vintage live watch with a one-line revert. The live risk is Alternative 1 (regime-dependence): both lean variants lose in Dec 2025/Jan 2026 and win 10–13% in Mar/May/Jul/Aug.

[ducroq]

Data-defect note for Stage 2 (gates unchanged)

The 2026-08-23..27 pipeline incident (augur#14, ducroq/energydatahub#50) left a permanent hole at 2026-08-25 in ml/shadow/eval_log.jsonl — no vintage was ever created for that day, so it can never be evaluated. Third such hole, after 06-08/06-10.

Effect on this experiment, both halves checked:

• Stage 1 is unaffected. scripts/exp018_stage0_ablation.py replays its own t0 grid off ml/data/training_history.parquet rather than off shadow vintages, so a missing shadow vintage costs it no holdout day — provided the parquet's price rows backfill, which is worth confirming when the window opens. The revisit trigger (≥14 vintages with t0 ≥ 2026-08-25, ≈2026-09-09) stands.
• Stage 2 is affected. Check (1) reads a trailing-14 mean of lightgbm_mae from eval_log.jsonl. Read 14 present rows; do not let the hole silently shorten the window or count as a zero, and apply the same rule to the pre-deploy baseline so both sides are measured identically.

Recorded as an addendum on the docs/hypothesis-log.md [2026-08-25] EXP-018a entry. Per ADR-007 this documents a defect in the data the Method reads — the four Stage-1 gates and three Stage-2 checks are unchanged.

Also relevant to the timing of this experiment: the t0-advance guard deployed 2026-08-28 means any further vintage loss before the Stage-1 window opens will be announced in the daily commit subject rather than discovered afterwards. The reliability position itself is now a hypothesis-log entry ([2026-08-28]) with a 14-run review that lands ≈2026-09-11, deliberately just after this experiment's own trigger.

[ducroq]

Status note (2026-08-29): EXP-018a Stage 1 is unaffected by today's work, and gets some indirect support.

Two things from EXP-020 that matter here:

1. The parquet is unperturbed. ml/data/consolidate.py gained four fundamentals columns today. The additive-only invariant was verified, not assumed — rebuilding the parquet from the same data directory with and without the new parsers gives an identical index and bit-identical values for all five pre-existing columns. Stage 1 fires ≈2026-09-09 off this same file and reads it through build_features(df)[FEATURE_COLUMNS], which ignores the new columns. Nothing to adjust.

2. lean beat full again, on two more windows. EXP-020 ran full and lean as arms:

window	vintages	lean vs full QS	DM p
2026-02-05 .. 2026-08-22	195	−7.8%	<0.0001
2025-12-01 .. 2026-08-22 (control)	263	−7.1%	<0.0001

against EXP-018's original −8.1%.

This does NOT discharge Stage 1. Both windows overlap the discovery data, so the effect size still carries the same selection bias the pre-commit was written to guard against — the whole point of Stage 1 is fresh vintages (t0 ≥ 2026-08-25). But it does make Alternative 3 (selection-bias mirage) less likely: an effect that survives at −7.1%/−7.8% across two different sub-windows is behaving more like a real one than an artifact of one lucky split. The gates are unchanged.

Also relevant to Alternative 4 ("exogenous removal is the wrong half"): EXP-020's lean_load arm — the lean set with load_forecast added back — is a tie with lean (p=0.851). That is weak evidence that dropping the exogenous half costs nothing, i.e. against Alternative 4, though the decisive test is still drop_rolling vs drop_rolling_and_exog on fresh vintages as pre-committed.

Evidence: experiments/registry.jsonl EXP-020, ml/shadow/exp020_fundamentals{,_ctl}/summary.json.

[ducroq]

Heads-up for whoever runs Stage 1: load_forecast became unreliable on 2026-08-30, and that entangles with this experiment's verdict.

On 2026-08-30 ENTSO-E returned A44 day-ahead prices for the next day but not A65 day-ahead load, so EDH's load_forecast spanned 24h while price spanned 48h. That crashed the nightly run (No clean feature row at t0) and cost the 2026-08-30 vintage. Fixed Augur-side by 87ed30c/1bfd728; upstream filed as ducroq/energydatahub#51.

Why it matters here. load_forecast is one of the three exogenous columns EXP-018 measured as inert (±0.4% MAE each) and that EXP-018a is pre-committed to test removing. The obvious reaction to the incident was "drop load_forecast and the problem disappears" — deliberately not done, because deciding it via an incident fix would pre-empt this pre-commitment and contaminate the result. The column stays until Stage 1 rules on it on the evidence.

Two consequences for Stage 1:

1. The window slipped. Stage 1 needs ≥14 vintages with t0 ≥ 2026-08-25; 08-30 is permanently lost, so the earliest date moves out by a day and the "only if no day is missed" caveat is now materially binding — EDH already skips ~11% of days (Scheduled publish silently skips whole days (4 of the last 35) — downstream Augur loses forecast vintages energydatahub#50).
2. A held-back t0 is a different vintage shape. While a feed is short, latest_feasible_t0 anchors up to ~24h behind the price horizon, and the commit subject carries [ALARM: t0 held back Nh — <feeds> short]. If any vintage in Stage 1's window carries that marker, decide before scoring whether to include it — the anchor hour differs from a normal vintage, and silently mixing the two is exactly the vintage-mismatch class the EXP-013 code review caught.

A separate hypothesis-log entry [2026-08-31] pre-commits whether the divergence is transient (review ≈2026-09-07, i.e. the same date this unblocks). Worth resolving that one first — Stage 1's verdict on load_forecast reads cleaner if we already know whether the column is reliably present.

[ducroq]

Stage 1 slipped ~5 days: ≈09-09 → ≈09-14 at the earliest

EDH published nothing from 2026-08-30T19:03 to 2026-09-04 (ducroq/energydatahub#51, PR #64), so Augur's t0 sat at 2026-08-31 for five consecutive runs and five vintages in the t0 ≥ 2026-08-25 window never existed.

The pre-committed threshold is unchanged — ≥14 vintages with t0 ≥ 2026-08-25 — only the date it lands on moved. That is a floor, not an estimate: EDH's publish reliability over 2026-07-25..08-28 was 31 of 35 days, so a further miss moves it again. Count vintages on sadalsuud, don't read the date:

.venv/bin/python -c "import pandas as pd; d=pd.read_parquet('ml/data/training_history.parquet'); print((d.index>='2026-08-25').sum(),'hours;',d.index.max())"

The local parquet is gitignored and stale; use /home/jeroen/local_dev/augur.

Two things that have not changed and are worth restating, since a delay is when scope creeps:

• Stage 1 still has first claim on the fresh-vintage window, ahead of EXP-021a (also → ≈09-14) and EXP-028a (→ ≈09-28). It decides lean vs full, and the others run against whichever base it leaves standing.
• load_forecast must not be dropped as a side effect of the horizon incident. It halved from 192 to 96 points on 08-28 and an A65 probe on 09-04 suggests NL has recovered; either way, whether that column earns its place is this experiment's call and not an incident fix's. That separation was made deliberately on 08-31 and holds.

The five lost days also make this entry's own "only if no day is missed" caveat materially more binding than when it was written.

[ducroq]

Supporting evidence from EXP-035 (2026-09-06), independent of Stage 1's pre-committed test.

Re-scoring all eight stored ablation variants against a seasonal-naive floor, 260 vintages, 18,340 paired rows each:

variant	MAE	skill vs naive	Aug-2026 skill
drop_rolling	27.07	+0.107	−0.104
drop_lags	28.51	+0.059	−0.162
drop_wind	28.62	+0.056	−0.182
drop_load	28.69	+0.054	−0.167
drop_solar	28.69	+0.053	−0.168
full	28.76	+0.051	−0.157
drop_exog	28.77	+0.051	−0.188
drop_calendar	30.82	−0.017	−0.408

Two readings:

1. drop_rolling is the strongest arm against the floor — double full's skill. That points the same way EXP-018a's pre-committed direction does, on a metric EXP-018 did not use. Encouraging, but not a substitute: this is the same offline window Stage 0 discovered on, so it carries the same selection bias that motivated Stage 1 in the first place. Stage 1's 14-fresh-vintage test stands unchanged. Count: 7 of 14 as of 2026-09-06.
2. drop_calendar is the only variant below naive over the whole window. The calendar block is the one feature group unambiguously carrying its weight — worth stating explicitly, since the ablation arc has otherwise been a run of nulls and negatives.

Caveat: every variant is below naive in August 2026, drop_rolling included. Feature reduction changes the ranking within LightGBM; it does not lift the class above the floor in the current regime. See #29.