tinycast

Do you need a transformer to forecast? Five tiny models — none bigger than 80k parameters — take on the ETTh1 long-horizon benchmark, evaluated exactly like the papers, and win.

▶ Live demo — the winning models run in your browser as plain JavaScript matrix products. No backend, no framework.

• The question — and why it matters
• The contenders
• Results
• Why does subtracting one number beat a transformer?
• The demo
• Key design decisions
• Limitations
• Reproduce it
• Project layout

The question — and why it matters

Between 2019 and 2022, long-horizon time-series forecasting was swept by ever-larger transformers — Informer (AAAI '21 best paper), Autoformer, FEDformer. Then Zeng et al. asked an embarrassing question (Are Transformers Effective for Time Series Forecasting?, AAAI '23): does a one-layer linear model beat them all on their own benchmarks?

This project re-runs that argument from scratch, honestly, on ETTh1 — hourly oil-temperature and load readings from a Chinese electricity transformer, the most-cited long-horizon benchmark — under the exact published protocol (same splits, same standardized multivariate MSE), and adds the two obvious follow-ups the paper skipped:

1. Training-free baselines. If nobody reports what seasonal naive scores, "beating Informer" means less than it sounds.
2. Small deep models. A dilated-causal temporal CNN and an LSTM, given the same modern normalization trick the linear models get — is depth the problem, or was it something else?

Everything trains in minutes on a laptop (Apple M1, MPS).

The contenders

Model	Params (h=96)	What it is
Persistence	0	repeat the last value
Seasonal naive	0	repeat yesterday, hour by hour
Linear	32k	one matrix: 336 past hours → 96 future hours
NLinear	32k	same, but subtract the window's last value first, add it back after
DLinear	65k	moving-average decomposition, one linear head for trend + one for remainder
TCN	25k	8 dilated causal conv blocks (receptive field 511) + RevIN
LSTM	79k	single-layer, hidden 128 + RevIN

All learned models are channel-independent (one shared set of weights applied to each of the 7 variables separately) and forecast all horizons directly — no autoregressive rollout, so no error compounding.

For scale: the transformers they're compared against are in the 10–15M parameter range — roughly 300× larger than anything here.

Results

Test MSE on standardized multivariate ETTh1 (lower is better), look-back 336. Published rows are from Zeng et al. 2023, Table 2 (transformers use their best setting, look-back 96):

Model	h=96	h=192	h=336	h=720
Persistence	1.294	1.325	1.330	1.335
Seasonal naive	0.512	0.581	0.650	0.655
Linear (ours)	0.395	0.447	0.490	0.527
NLinear (ours)	0.400	0.423	0.450	0.455
DLinear (ours)	0.395	0.435	0.474	0.501
TCN (ours)	0.385	0.438	0.485	0.524
LSTM (ours)	0.410	0.454	0.460	0.527
Informer (published)	0.865	1.008	1.107	1.181
Autoformer (published)	0.449	0.500	0.521	0.514
FEDformer (published)	0.376	0.420	0.459	0.506
DLinear (published)	0.375	0.405	0.439	0.472
NLinear (published)	0.374	0.408	0.429	0.440

Three things worth staring at:

• NLinear beats every published transformer at every horizon — 54–61% below Informer, under Autoformer and FEDformer across the board — and lands within 3–7% of the paper's own tuned linear numbers, untuned, single seed.
• The zero-parameter seasonal-naive baseline beats Informer at every horizon (0.512 vs 0.865 at h=96). An AAAI best-paper transformer loses to repeat yesterday — which is why baselines belong in every results table.
• Depth isn't the villain. Given RevIN, the TCN is actually the single best model at h=96 (0.385) and the LSTM is fine — but at long horizons neither buys anything the one-matrix model doesn't already have.

In real units: the best h=96 model's average oil-temperature error four days out is 1.575 °C.

Why does subtracting one number beat a transformer?

ETTh1's test year simply doesn't live at the same temperature level as the training year — the 30-day mean drifts far from the training mean. Any model that memorizes absolute levels inherits that shift as bias. NLinear's whole trick is to forecast relative to the window's last value, which cancels the level entirely; RevIN does the same for the deep models by normalizing each window. The test_nlinear_shift_equivariance test pins the property down: shift the input by +5, the forecast shifts by exactly +5.

That's the honest reading of the "linear beats transformers" result: the benchmark rewards distribution-shift robustness far more than it rewards capacity, and 2019–22 era transformers spent their parameters on the wrong problem.

The demo

The live Space is fully static: shim.js intercepts the UI's /api/* calls, fetches the raw ETTh1 CSV straight from GitHub, and runs NLinear/DLinear as hand-written JS matrix products against exported Float32Array weights — the same UI file the FastAPI service serves locally. A pytest runs the actual JS under Node and checks it against PyTorch to 1e-4.

Run it locally with the full model zoo (TCN and LSTM included):

make serve   # then open http://127.0.0.1:8012

Key design decisions

• Paper-identical protocol, verified against the paper. 12/4/4-month chronological split with the Informer border convention (val/test reach back seq_len rows for history only), standardization fit on train rows only, MSE/MAE on standardized values. The published numbers in evaluate.py were transcribed from the DLinear paper and cross-checked at build time.
• Baselines before parameters. Persistence and seasonal-naive run through the identical windowing and metric code as the learned models — one if away in the results table, not an afterthought.
• Channel independence over channel mixing. One shared univariate model per channel (the PatchTST finding). It's also what makes the JS port trivial.
• RevIN for the deep models. Comparing a normalized linear model against un-normalized deep models would manufacture the "deep learning fails" headline. Giving TCN/LSTM the same defense makes the comparison mean something.
• Direct multi-horizon output. Every model emits the whole horizon in one shot; a separate model per horizon, as in the papers.
• Leakage tested, not assumed. Corrupting val/test rows must not change the scaler; window y must start exactly one step after x; val history must be exactly the train tail. All of these are unit tests, not comments.
• The deployed thing is the evaluated thing. Weights are exported as one flat Float32Array blob + JSON manifest, and the parity chain PyTorch → numpy → Node is enforced by tests at every link.

Limitations

• One dataset. This reproduces the ETTh1 column of the argument. The DLinear paper shows the same pattern on eight other benchmarks; nothing here licenses claims beyond ETTh1.
• Published numbers are taken, not re-run. Informer/Autoformer/FEDformer results come from Zeng et al.'s table (their best settings, look-back 96). Re-training them was out of scope — and out of an M1's league.
• No hyperparameter search. Sensible defaults, early stopping on val, one seed. The point is that without tuning, tiny models land at (or under) the published transformer numbers; tuned numbers would only sharpen the same conclusion.
• Modern transformers do better. PatchTST (2023) and later models close much of this gap. The claim is not "attention is useless" — it's that benchmark wins bought with 300× more parameters deserve a naive-baseline sanity check.
• The browser demo ships the linear family only. TCN/LSTM inference stays in the FastAPI service; porting conv/RNN loops to JS wasn't worth it for models the linear ones beat.

Reproduce it

git clone https://github.com/UsmarHaider/tinycast && cd tinycast
make venv    # python3 -m venv + pip install -e ".[dev]"
make all     # download → train (≈25 min on an M1) → evaluate → figures → export → test
make serve   # local demo at http://127.0.0.1:8012

Deploy the Space (needs HF_TOKEN in .env, see .env.example):

.venv/bin/python scripts/build_space.py
.venv/bin/python scripts/deploy_space.py

Data is downloaded from ETDataset at build time and never committed.

Project layout

src/tinycast/
  config.py      # paths, benchmark borders, hyperparameters
  data.py        # download, split, standardize, window
  baselines.py   # persistence, seasonal naive
  models.py      # Linear / NLinear / DLinear / TCN / LSTM
  train.py       # early-stopped training for all model×horizon pairs
  evaluate.py    # paper-protocol metrics + published comparison table
  export.py      # Float32 weight bundle + numpy reference forwards
  figures.py     # README figures
  service.py     # FastAPI demo API
  web/
    index.html   # the whole UI — one file, no build step, light+dark
    shim.js      # in-browser engine for the static Space
scripts/         # build_space.py, deploy_space.py
tests/           # 41 tests: leakage, properties, parity, service

Acknowledgements

• Data: ETDataset (Zhou et al., Informer, AAAI 2021)
• The argument being reproduced: Zeng et al. 2023; RevIN: Kim et al., ICLR 2022