Decoupled Entropy Minimization

Jing Ma¹  Hanlin Li¹  Xiang Xiang^1,2

¹HAIV Lab, Huazhong University of Science and Technology (HUST) 

²Peng Cheng National Laboratory 

Introduction

This repository contains the official implementation of Decoupled Entropy Minimization published in NeurIPS 2025. Please check the paper for more details.

Abstract :

Entropy Minimization (EM) is beneficial to reducing class overlap, bridging domain gap, and restricting uncertainty for various tasks in machine learning, yet its potential is limited. To study the internal mechanism of EM, we reformulate and decouple the classical EM into two parts with opposite effects: cluster aggregation driving factor (CADF) rewards dominant classes and prompts a peaked output distribution, while gradient mitigation calibrator (GMC) penalizes high-confidence classes based on predicted probabilities. Furthermore, we reveal the limitations of classical EM caused by its coupled formulation: 1) reward collapse impedes the contribution of high-certainty samples in the learning process, and 2) easy-class bias induces misalignment between output distribution and label distribution. To address these issues, we propose Adaptive Decoupled Entropy Minimization (AdaDEM), which normalizes the reward brought from CADF and employs a marginal entropy calibrator (MEC) to replace GMC. AdaDEM outperforms DEM*, an upper-bound variant of classical EM, and achieves superior performance across various imperfectly supervised learning tasks in noisy and dynamic environments.

BibTeX :

@article{ma2025decoupled,
  title={Decoupled Entropy Minimization},
  author={Ma, Jing and Li, Hanlin and Xiang, Xiang},
  journal={arXiv preprint arXiv:2511.03256},
  year={2025}
}

Installation

Requirements

• Python >= 3.6
• PyTorch >= 1.6.0
• timm >= 0.4.5
• nni >= 2.0

Datasets

Download ImageNet and update --data in main.py to the path of ImageNet.

Download ImageNet-C and update --data_corruption in main.py to the path of ImageNet-C.

Usage

EM (Tent)

python main.py --exp_type ${EXP} --arch ${MODEL} --lr ${LR} --seed ${SEED}

• ${EXP}: experiment type, including single and continual.
• ${MODEL}: model architecture, including resnet50 and ViT_B16.
• ${LR}: learning rate, e.g., 0.001.
• ${SEED}: random seed, e.g., 1.

An example of running EM is provided in scripts/EM.sh.

DEM*

python main.py --exp_type ${EXP} --arch ${MODEL} --lr ${LR} --seed ${SEED} --dem --tau ${TAU} --alpha ${ALPHA}

• ${TAU}: temperature for CADF, e.g., 1.0.
• ${ALPHA}: weight for GMC, e.g., 1.0.

An example of running DEM* is provided in scripts/DEM*.sh.

We also provide scripts for searching optimal hyperparameters $(\tau, \alpha)$ of DEM in scripts/DEM.sh using NNI.

AdaDEM

python main.py --exp_type ${EXP} --arch ${MODEL} --lr ${LR} --seed ${SEED} --adadem --pi ${PI}

• ${PI}: momentum for MEC, e.g., 0.1.

An example of running AdaDEM is provided in scripts/AdaDEM.sh.

Citation

If you find our code useful or our work relevant, please consider citing:

@article{ma2025decoupled,
  title={Decoupled Entropy Minimization},
  author={Ma, Jing and Li, Hanlin and Xiang, Xiang},
  journal={arXiv preprint arXiv:2511.03256},
  year={2025}
}