This repository contains the official implementation of the paper:
Entropy-Guided Dynamic Patch Segmentation for Time Series Transformers
Under review.
EntroPE introduces entropy-driven dynamic patching for time series Transformers. Instead of fixed-length patches, we use information-theoretic principles to place patch boundaries where predictive uncertainty is highest, creating semantically meaningful temporal segments.
EntroPE framework architecture: (A) Entropy-Based Dynamic Patcher calculates entropy to identify boundaries; (B) Adaptive Patch Encoder aggregates patches into fixed-size embeddings; (C) Fusion Decoder combines global and local context for forecasting.
Static patching (top) vs. our entropy-driven dynamic approach (bottom).
- Entropy-Based Dynamic Patcher (EDP): Uses conditional entropy to identify temporal transition points and place patch boundaries at positions with high predictive uncertainty
- Adaptive Patch Encoder (APE): Employs cross-attention mechanisms to encode variable-length patches into fixed-size representations while preserving dependencies
- Temporally-Informed Architecture: Preserves temporal coherence by respecting natural time series structure, addressing train-inference mismatch
# Create environment
conda create --name entrope python=3.10
conda activate entrope
pip install -r requirements.txtWe evaluate on 6 benchmark datasets:
- ETT family: ETTh1, ETTh2, ETTm1, ETTm2 (Energy)
- Weather: Meteorological data
- Electricity: Power consumption
# Run specific dataset
sh scripts/etth1.shEntroPE achieves significant improvements over existing methods:
- ~20% improvement on ETTh1 compared to PatchTST
- ~15% improvement on Electricity dataset
- ~10% average improvement across all benchmarks
- Better computational efficiency through dynamic patching
- threshold_global: Controls patch granularity (higher = fewer patches)
- threshold_relative: Controls sensitivity to entropy changes
- Robust across range [75%, 95%] for most datasets
- Information-theoretic approach using conditional entropy
- Dual-threshold mechanism (global + relative thresholds)
- Respects temporal causality and predictive difficulty
- Handles variable-length patches efficiently
- Cross-attention refinement preserves intra-patch dependencies
- Fixed-size output suitable for transformer processing
- Reduces token count through intelligent boundary placement
- Maintains computational tractability
- Scales well with sequence length
Evaluation Metrics: MSE, MAE, MACs (Multiply-Accumulate Operations)
If you found this work useful for you, please consider citing it.
@inproceedings{sachith_entrope_26,
title={Entropy Guided Dynamic Patch Segmentation for Time Series Transformers},
author={Abeywickrama, Sachith and Eldele, Emadeldeen and Wu, Min and Li, Xiaoli and Yuen, Chau},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
year={2026}
}This project is licensed under the Apache License - see the LICENSE file for details.
This work builds upon and is inspired by several key contributions in the field:
-
PatchTST: Our approach is built on the foundation of PatchTST and other patch-based time series transformers, which demonstrated the effectiveness of patch-based architectures for time series forecasting.
- Repository: https://github.com/yuqinie98/PatchTST
-
nanoGPT: The Entropy Model GPT-2 architecture implementation partially incorporates code from Andrej Karpathy's nanoGPT implementation. We gratefully acknowledge this clean and educational codebase.
- Repository: https://github.com/karpathy/nanoGPT
-
Byte Latent Transformer: Our dynamic patching approach draws inspiration from advances in NLP, particularly the Byte Latent Transformer's innovative approach to variable-length tokenization.
- Repository: https://github.com/facebookresearch/blt
We thank the authors of these works for their contributions to the open-source community and for advancing the state of the art in time series forecasting and transformer architectures.