A Survey on Interpretability in Visual Recognition

Welcome to our curated repository showcasing significant works in the field of interpretable visual recognition. This collection serves as a companion to our survey paper, A Survey on Interpretability in Visual Recognition, providing insights into the evolving landscape of interpretability in this dynamic domain. We welcome contributions from the community to expand and update this repository. If you have a project or paper that you believe should be included, feel free to open an issue or submit a pull request.

Catalogue

• A Survey on Interpretability in Visual Recognition
  • Catalogue
  • Related Survey
  • Badge Introduction
  • Paper List
  • Metrics and Toolkits
  • Citation

Related Survey

We offer a selection of survey papers pertinent to this research. For more information, please visit https://vipl-vsu.github.io/xai-recognition/.

Badge Introduction

The table below introduces various badges used to categorize papers in terms of the four key dimensions of XAI recognition methods: Intent, Object, Presentation, and Methodology. The badges help to efficiently tag and identify research papers based on their interpretability approaches.

Badge	Description
	Intent is passive. Methods that explain already trained models by revealing their recognition process.
	Intent is active. Methods that integrate interpretability during model construction, making the process inherently interpretable.
	Object is local. Explanation focused on individual samples, such as diagnostic suggestions for each patient.
	Object is semilocal. Explanation that highlights common characteristics within a class of samples.
	Object is global. Explanation of the entire model's decision rules, often category-independent.
	Presentation is scalar. Explanation presented in quantitative forms, such as numerical scores.
	Presentation is attention. Used to highlight important features or regions contributing to a decision.
	Presentation is structured. Explanation involving structured representations such as graphs.
	Presentation is semantic unit. Explanation decomposed into human-understandable semantic concepts.
	Presentation is exemplar. Explanation through examples that illustrate specific model behaviors.
	Methodology is association. Methods that model correlations to show the relationships and patterns between inputs and outputs.
	Methodology is intervention. Methods predicting outcomes after making active changes to the model or its inputs.
	Methodology is conterfactual. Simulates alternative scenarios by perturbing inputs to explore the potential outcomes that could arise under different conditions.

Paper List

• Explain Any Concept: Segment Anything Meets Concept-based Explanation. NeurIPS 2024. Paper
  

• This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations. NeurIPS 2024. Paper
  

• ECLAD: Extracting Concepts with Local Aggregated Descriptors. Pattern Recognition 2024. Paper
  

• Interpretable Object Recognition by Semantic Prototype Analysis. WACV 2024. Paper Code
  

• Concept-Centric Transformers: Enhancing Model Interpretability Through Object-Centric Concept Learning Within a Shared Global Workspace. WACV 2024. Paper
  

• From Attribution Maps to Human-understandable Explanations Through Concept Relevance Propagation. Nature Machine Intelligence 2023. Paper
  

• PIP-Net: Patch-based Intuitive Prototypes for Interpretable Image Classification. CVPR 2023. Paper
  

• Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. CVPR 2023. Paper
  

• Learning Support and Trivial Prototypes for Interpretable Image Classification. ICCV 2023. Paper
  

• Optimizing Explanations by Network Canonization and Hyperparameter Search. CVPR 2023. Paper
  

• Understanding the (Extra-)ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations. arXiv 2311.16681. Paper
  

• Harsanyinet: Computing Accurate Shapley Values in a Single Forward Propagation. arXiv 2304.01811. Paper
  

• Interpretability-aware Vision Transformer. arXiv 2309.08035. Paper
  

• A Novel Neural-symbolic System under Statistical Relational Learning. arXiv 2309.08931. Paper
  

• Mitigating Bias: Enhancing Image Classification by Improving Model Explanations. arXiv 2307.01473. Paper
  

• Fixating on Attention: Integrating Human Eye Tracking into Vision Transformers. arXiv 2308.13969. Paper
  

• UFO: A Unified Method for Controlling Understandability and Faithfulness Objectives in Concept-based Explanations for CNNs. arXiv 2303.15632. Paper
  

• Distance-Aware eXplanation Based Learning. ICTAI 2023. Paper
  

• Gradient Strikes Back: How Filtering Out High Frequencies Improves Explanations. arXiv 2307.09591. Paper
  

• Concept Bottleneck Model with Additional Unsupervised Concepts. IEEE Access 2022. Paper
  

• Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification. TPAMI 2022. Paper
  

• SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation. arXiv 2204.10926. Paper
  

• ELUDE: Generating Interpretable Explanations via a Decomposition into Labelled and Unlabelled Features. arXiv 2206.07690. Paper
  

• Neural Prototype Trees for Interpretable Fine-grained Image Recognition. CVPR 2021. Paper
  

• Natural Language Descriptions of Deep Visual Features. ICLR 2021. Paper
  

• This Looks Like That, Because... Explaining Prototypes for Interpretable Image Recognition. ECML-PKDD 2021. Paper
  

• Dissect: Disentangled Simultaneous Explanations via Concept Traversals. arXiv 2105.15164. Paper
  

• Interpretable Compositional Convolutional Neural Networks. arXiv 2107.04474. Paper
  

• Best of Both Worlds: Local and Global explanations with Human-understandable Concepts. arXiv 2106.08641. Paper
  

• A Game-theoretic Taxonomy of Visual Concepts in DNNs. arXiv 2106.10938. Paper
  

• Keep Calm and Improve Visual Feature Attribution. ICCV 2021. Paper
  

• Generating Visual Explanations with Natural Language. Applied AI Letters 2021. Paper
  

• Visualizing the Emergence of Intermediate Visual Patterns in DNNs. NeurIPS 2021. Paper
  

• Concept Bottleneck Models. ICML 2020. Paper
  

• Concept Whitening for Interpretable Image Recognition. Nature Machine Intelligence 2020. Paper
  

• On Completeness-aware Concept-based Explanations in Deep Neural Networks. NeurIPS 2020. Paper
  

• Explaining Knowledge Distillation by Quantifying the Knowledge. CVPR 2020. Paper
  

• Interpretable CNNs for Object Classification. TPAMI 2020. Paper
  

• Interactive Explanations of Internal Representations of Neural Network Layers: An Exploratory Study on Outcome Prediction of Comatose Patients. KDH 2020. Paper
  

• Generating Visual and Semantic Explanations with Multi-task Network. ECCV Workshops 2020. Paper
  

• This Looks Like That: Deep Learning for Interpretable Image Recognition. NeurIPS 2019. Paper
  

• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn. Nature Communications 2019. Paper
  

• Towards Automatic Concept-based Explanations. NeurIPS 2019. Paper
  

• Explaining Classifiers with Causal Concept Effect (CaCE). arXiv 1907.07165. Paper
  

• Interpretable Image Recognition with Hierarchical Prototypes. AAAI 2019. Paper
  

• Towards Aggregating Weighted Feature Attributions. arXiv 1901.10040. Paper
  

• Deep Features Analysis with Attention Networks. arXiv 1901.10042. Paper
  

• Visualising the Training Process of Convolutional Neural Networks for Non-experts. BNAIC 2019. Paper
  

• A Universal Logic Operator for Interpretable Deep Convolution Networks. arXiv 1901.08551. Paper
  

• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML 2018. Paper
  

• Deep Learning for Case-based Reasoning through Prototypes: A Neural Network that Explains Its Predictions. AAAI 2018. Paper
  

• Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. CVPR 2018. Paper
  

• Interpreting Deep Visual Representations via Network Dissection. TPAMI 2018. Paper
  

• Interpretable Basis Decomposition for Visual Explanation. ECCV 2018. Paper
  

• Grounding Visual Explanations. ECCV 2018. Paper
  

• Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv 1806.02891. Paper
  

• Unsupervised Learning of Neural Networks to Explain Neural Networks. arXiv 1805.07468. Paper
  

• Generating Post-hoc Rationales of Deep Visual Classification Decisions. Explainable and Interpretable Models in Computer Vision and Machine Learning, 2018. Paper
  

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV 2017. Paper
  

• Axiomatic Attribution for Deep Networks. ICML 2017. Paper
  

• SmoothGrad: Removing Noise by Adding Noise. arXiv 1706.03825. Paper
  

• Network Dissection: Quantifying Interpretability of Deep Visual Representations. CVPR 2017. Paper
  

• Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition. Pattern Recognition 2017. Paper
  

• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples. arXiv 1708.05493. Paper
  

• Growing Interpretable Part Graphs on Convnets via Multi-shot Learning. AAAI 2017. Paper
  

• Learning Deep Features for Discriminative Localization. CVPR 2016. Paper
  

• Generating Visual Explanations. ECCV 2016. Paper
  

• On Pixel-wise Explanations for Non-linear Classifier Decisions by Layer-wise Relevance Propagation. PloS one 2015. Paper
  

Metrics and Toolkits

Metrics & Evaluation Related

• Incremental Residual Concept Bottleneck Models. CVPR 2024. Paper Code

• Faithful Vision-Language Interpretation via Concept Bottleneck Models. ICLR 2024. Paper Code

• Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023. Paper Code

• Learning Bottleneck Concepts in Image Classification. CVPR 2023. Paper Code

• Learning Support and Trivial Prototypes for Interpretable Image Classification. ICCV 2023. Paper Code

• Distance-Aware eXplanation Based Learning. ICTAI 2023. Paper Code

• Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations. arXiv 2310.08537. Paper Code

• A Framework for Learning Ante-Hoc Explainable Models via Concepts. CVPR 2022. Paper Code

• CLEVR-XAI: A Benchmark Dataset for the Ground Truth Evaluation of Neural Network Explanations. Information Fusion 2022. Paper Code

• How Good Is Your Explanation? Algorithmic Stability Measures to Assess the Quality of Explanations for Deep Neural Networks. WACV 2022. Paper

• Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior. CHI 2022. Paper Code

• An Experimental Study of Quantitative Evaluations on Saliency Methods. KDD 2021. Paper

• Quantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark. IUI 2021. Paper Code

• Invertible Concept-Based Explanations for CNN Models with Non-Negative Concept Activation Vectors. AAAI 2021. Paper Code

• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning. arXiv 2106.12543. Paper Code

• Explaining Knowledge Distillation by Quantifying the Knowledge. CVPR 2020. Paper Code

• Sanity Checks for Saliency Metrics. AAAI 2020. Paper

• On Completeness-Aware Concept-Based Explanations in Deep Neural Networks. NeurIPS 2020. Paper Code

• Understanding the Decisions of CNNs: An In-Model Approach. Pattern Recognition Letters 2020. Paper Code

• Evaluating CNN Interpretability on Sketch Classification. ICMV 2019. Paper

• Benchmarking Attribution Methods with Relative Feature Importance. arXiv 1907.09701. Paper Code

• Towards a Unified Evaluation of Explanation Methods Without Ground Truth. arXiv 1911.09017. Paper

• Top-Down Neural Attention by Excitation Backprop. International Journal of Computer Vision 2018. Paper Code

• RISE: Randomized Input Sampling for Explanation of Black-Box Models. arXiv 1806.07421. Paper Code

• Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems 2016. Paper

Toolkits & Libraries

• LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models. CVPRW 2024. Paper Code

• ECLAD: Extracting Concepts with Local Aggregated Descriptors. Pattern Recognition 2024. Paper Code

• Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. Journal of Machine Learning Research 2023. Paper Code

• VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers. CVPR 2022. Paper Code

• Xplique: A Deep Learning Explainability Toolbox. arXiv 2206.04394. Paper Code

• Captum: A Unified and Generic Model Interpretability Library for PyTorch. arXiv 2009.07896. Paper Code

Citation

Plain Text

Qiyang Wan, Chengzhi Gao, Ruiping Wang, Xilin Chen, "A Survey on Interpretability in Visual Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 7, pp. 8547-8566, Jul. 2026.

BibTeX

@article{wan2025survey,
  title={A Survey on Interpretability in Visual Recognition},
  author={Wan, Qiyang and Gao, Chengzhi and Wang, Ruiping and Chen, Xilin},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026},
  volume={48},
  number={7},
  pages={8547-8566},
  doi={10.1109/TPAMI.2026.3672629}
}