Learning from Noisy Labels with Deep Neural Networks: A Survey

This is a repository to help all readers who are interested in handling noisy labels.

If your papers are missing or you have other requests, please contact to ghkswns91@gmail.com.
We will update this repository and paper on a regular basis to maintain up-to-date.

• Feb 16, 2022: Our survey paper was accepted to TNNLS journal (IF=10.451) [arxiv version]
• Feb 17, 2022: Last update: including papers published in 2021 and 2022

Citation (.bib)

@article{song2022survey,
  title={Learning from Noisy Labels with Deep Neural Networks: A Survey},
  author={Song, Hwanjun and Kim, Minseok and Park, Dongmin and Shin, Yooju and Jae-Gil Lee},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2022}}

Contents

• List of Papers with Categorization

List of Papers with Categorization

All Papers are sorted chronologically according to five categories below, so that you can find related papers more quickly.

We also provide a tabular form of summarization with their methodological comaprison (Table 2 in the paper). - [here]
This is a brief summary for the categorization. Please see Section III in our survey paper for the details - [here]

[Index: Robust Architecture, Robust Regularization, Robust Loss Function, Loss Adjsutment, Sample Selection]

Robust Learning for Noisy Labels
|--- A. Robust Architecture
     |--- A.1. Noise Adaptation Layer: adding a noise adaptation layer at the top of an underlying DNN to learn label transition process
     |--- A.2. Dedicated Architecture: developing a dedicated architecture to reliably support more diverse types of label noises.
|--- B. Robust Regularization
     |--- B.1. Explicit Regularization: an explicit form that modifies the expected tarining loss, e.g., weight decay and dropout.
     |--- B.2. Implicit Regularization: an implicit form that gives the effect of stochasticity, e.g., data augmentation and mini-batch SGD.
|--- C. Robust Loss Function: designing a new loss function robust to label noise.
|--- D. Loss Adjsutment
     |--- D.1. Loss Correction: multiplying the estimated transition matrix to the prediction for all the observable labels.
     |--- D.2. Loss Reweighting: multiplying the estimated example confidence (weight) to the example loss.
     |--- D.3. Label Refurbishment: replacing the original label with other reliable one.
     |--- D.4. Meta Learning: finding an optimal adjustment rule for loss reweighing or label refurbishment.
|--- E. Sample Selection
     |--- E.1. Multi-network Learning: collaborative learning or co-training to identify clean examples from noisy data.
     |--- E.2. Multi-round Learning: refining the selected clean set through training multiple rounds.
     |--- E.3. Hybrid Leanring: combining a specific sample selection strategy with a specific semi-supervised learning model or other orthogonal directions.

In addition, there are some valuable theoretical or empirical papers for understanding the nature of noisy labels.
Go to Theoretical or Empirical Understanding.

A. Robust Architecture

A.1. Noise Adaptation Layer

Year	Venue	Title	Implementation
2015	ICCV	Webly supervised learning of convolutional networks	Official (Caffe)
2015	ICLRW	Training convolutional networks with noisy labels	Unofficial (Keras)
2016	ICDM	Learning deep networks from noisy labels with dropout regularization	Official (MATLAB)
2016	ICASSP	Training deep neural-networks based on unreliable labels	Unofficial (Chainer)
2017	ICLR	Training deep neural-networks using a noise adaptation layer	Official (Keras)

A.2. Dedicated Architecture

Year	Venue	Title	Implementation
2015	CVPR	Learning from massive noisy labeled data for image classification	Official (Caffe)
2018	NeurIPS	Masking: A new perspective of noisy supervision	Official (TensorFlow)
2018	TIP	Deep learning from noisy image labels with quality embedding	N/A
2019	ICML	Robust inference via generative classifiers for handling noisy labels	Official (PyTorch)

B. Robust Regularization

B.1. Explicit Regularization

Year	Venue	Title	Implementation
2018	ECCV	Deep bilevel learning	Official (TensorFlow)
2019	CVPR	Learning from noisy labels by regularized estimation of annotator confusion	Official (TensorFlow)
2019	ICML	Using pre-training can improve model robustness and uncertainty	Official (PyTorch)
2020	ICLR	Can gradient clipping mitigate label noise?	Unofficial (PyTorch)
2020	ICLR	Wasserstein adversarial regularization (WAR) on label noise	N/A
2021	ICLR	Robust early-learning: Hindering the memorization of noisy labels	Official (PyTorch)
2021	ICLR	When Optimizing f-Divergence is Robust with Label Noise	Official (PyTorch)
2021	ICCV	Learning with Noisy Labels via Sparse Regularization	Official (PyTorch)
2021	NeurIPS	Open-set Label Noise Can Improve Robustness Against Inherent Label Noise	Official (PyTorch)

B.2. Implicit Regularization

Year	Venue	Title	Implementation
2015	ICLR	Explaining and harnessing adversarial examples	Unofficial (PyTorch)
2017	ICLRW	Regularizing neural networks by penalizing confident output distributions	Unofficial (PyTorch)
2018	ICLR	Mixup: Beyond empirical risk minimization	Official (PyTorch)
2021	CVPR	Augmentation Strategies for Learning with Noisy Labels	Official (PyTorch)
2021	CVPR	AutoDO: Robust AutoAugment for Biased Data With Label Noise via Scalable Probabilistic Implicit Differentiation	Official (PyTorch)

C. Robust Loss Function

Year	Venue	Title	Implementation
2017	AAAI	Robust loss functions under label noise for deep neural networks	N/A
2017	ICCV	Symmetric cross entropy for robust learning with noisy labels	Official (Keras)
2018	NeurIPS	Generalized cross entropy loss for training deep neural networks with noisy labels	Unofficial (PyTorch)
2020	ICLR	Curriculum loss: Robust learning and generalization against label corruption	N/A
2020	ICML	Normalized loss functions for deep learning with noisy labels	Official (PyTorch)
2020	ICML	Peer loss functions: Learning from noisy labels without knowing noise rates	Official (PyTorch)
2021	CVPR	Learning Cross-Modal Retrieval with Noisy Labels	Official (Pytorch)
2021	CVPR	A Second-Order Approach to Learning With Instance-Dependent Label Noise	Official (PyTorch)
2022	ICLR	An Information Fusion Approach to Learning with Instance-Dependent Label Noise	N/A

D. Loss Adjustment

D.1. Loss Correction

Year	Venue	Title	Implementation
2017	CVPR	Making deep neural networks robust to label noise: A loss correction approach	Official (Keras)
2018	NeurIPS	Using trusted data to train deep networks on labels corrupted by severe noise	Official (PyTorch)
2019	NeurIPS	Are anchor points really indispensable in label-noise learning?	Official (PyTorch)
2020	NeurIPS	Dual T: Reducing estimation error for transition matrix in label-noise learning	N/A
2021	AAAI	Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model	Official (PyTorch)

D.2. Loss Reweighting

Year	Venue	Title	Implementation
2017	TNNLS	Multiclass learning with partially corrupted labels	Unofficial (PyTorch)
2017	NeurIPS	Active Bias: Training more accurate neural networks by emphasizing high variance samples	Unofficial (TensorFlow)

D.3. Label Refurbishment

Year	Venue	Title	Implementation
2015	ICLR	Training deep neural networks on noisy labels with bootstrapping	Unofficial (Keras)
2018	ICML	Dimensionality-driven learning with noisy labels	Official (Keras)
2019	ICML	Unsupervised label noise modeling and loss correction	Official (PyTorch)
2020	NeurIPS	Self-adaptive training: beyond empirical risk minimization	Official (PyTorch)
2020	ICML	Error-bounded correction of noisy labels	Official (PyTorch)
2021	AAAI	Beyond class-conditional assumption: A primary attempt to combat instancedependent label noise	Official (PyTorch)

D.4. Meta Learning

Year	Venue	Title	Implementation
2017	NeurIPSW	Learning to learn from weak supervision by full supervision	Unofficial (TensorFlow)
2017	ICCV	Learning from noisy labels with distillation	N/A
2018	ICML	Learning to reweight examples for robust deep learning	Official (TensorFlow)
2019	NeurIPS	Meta-Weight-Net: Learning an explicit mapping for sample weighting	Official (PyTorch)
2020	CVPR	Distilling effective supervision from severe label noise	Official (TensorFlow)
2021	AAAI	Meta label correction for noisy label learning	Official (PyTorch)
2021	ICCV	Adaptive Label Noise Cleaning with Meta-Supervision for Deep Face Recognition	N/A

E. Sample Selection

E.1. Multi-network Learning

Year	Venue	Title	Implementation
2017	NeurIPS	Decoupling when to update from how to update	Official (TensorFlow)
2018	ICML	MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels	Official (TensorFlow)
2018	NeurIPS	Co-teaching: Robust training of deep neural networks with extremely noisy labels	Official (PyTorch)
2019	ICML	How does disagreement help generalization against label corruption?	Official (PyTorch)
2021	CVPR	Jo-SRC: A Contrastive Approach for Combating Noisy Labels	Official (PyTorch)

E.2. Single- or Multi-round Learning

Year	Venue	Title	Implementation
2018	CVPR	Iterative learning with open-set noisy labels	Official (Keras)
2019	ICML	Learning with bad training data via iterative trimmed loss minimization	Official (GluonCV)
2019	ICML	Understanding and utilizing deep neural networks trained with noisy labels	Official (Keras)
2019	ICCV	O2U-Net: A simple noisy label detection approach for deep neural networks	Unofficial (PyTorch)
2020	ICMLW	How does early stopping can help generalization against label noise?	Official (Tensorflow)
2020	NeurIPS	A topological filter for learning with label noise	Official (PyTorch)
2021	ICLR	Learning with Instance-Dependent Label Noise: A Sample Sieve Approach	Official (PyTorch)
2021	NeurIPS	FINE Samples for Learning with Noisy Labels	Official (PyTorch)
2022	ICLR	Sample Selection with Uncertainty of Losses for Learning with Noisy Labels	N/A

E.3. Hybrid Learning

Year	Venue	Title	Implementation
2019	ICML	SELFIE: Refurbishing unclean samples for robust deep learning	Official (TensorFlow)
2020	ICLR	SELF: Learning to filter noisy labels with self-ensembling	N/A
2020	ICLR	DivideMix: Learning with noisy labels as semi-supervised learning	Official (PyTorch)
2021	ICLR	Robust curriculum learning: from clean label detection to noisy label self-correction	N/A
2021	NeurIPS	Understanding and Improving Early Stopping for Learning with Noisy Labels	Official (PyTorch)

Theoretical or Empirical Understanding

How Does a Neural Network’s Architecture Impact Its Robustness to Noisy Labels, NeurIPS 2021 [Link]
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, AAAI 2021 [Link]
Understanding Instance-Level Label Noise: Disparate Impacts and Treatments, ICML 2021 [Link]
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR 2022 [Link]

Learning from Noisy Labels towards Realistic Scenarios

There have been some studies to solve more realistic setups associated with noisy labels.

• Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries, CVPR 2022, [code]
  This paper addresses the problem of noisy labels in the online continual learning setup.
• Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion, ECCV 2022, [code]
  This paper addresses the scenario in which each data instance has multiple noisy labels from annotators (instead of a single label).