DL (re)training is integrated with testing and two robustness testing metrics, FOL and ZOL, are proposed.
The first source-level pre-training mutation tool based on real DL faults.
An approach to automatically generate new test inputs that can be used to augment the existing test set so that its capability to detect DL mutations increases.
A mutation testing-based tool for DNNs, DeepMutation++, which facilitates the DNN quality evaluation, supporting both feed-forward neural networks (FNNs) and stateful recurrent neural networks (RNNs).
The paper designs eight source-level mutation operators to introduce faults into the DL programming elements and propose two DL-specific mutation testing metrics.
First explicitly combines the output of a neuron and the connection weight it connects to the nextlevel neuron.
[ICSE 2020] Importance-Driven Deep Learning System Testing
The first systematic and automated testing methodology that employs the semantics of neuron influence to the DL system as a means of developing a laywer-wise functional understanding of its internal behaviour and assessing the semantic adequacy of a test set.
[ISSTA 2020] Effective white-box testing of deep neural networks with adaptive neuron-selection strategy.
This paper presents a new white-box testing technique for deep neural networks and shows that using a fixed neuron-selection strategy is a major limitation of the existing white-box approaches.
The paper conducts an empirical study to uncover the characteristics of adversarial examples(AEs) from the perspective of uncertainty.
The paper proposes SADL, a fine grained test adequacy metric that measures the surprise of an input, i.e., the difference in the behaviour of a DL system between a given input and the training data.
The paper first proposes a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation.
The paper develops coverage-guided fuzzing (CGF) methods for neural network.
The paper performs an exploratory study of combinatorial testing (CT) on DL systems.
The very first step towards the quantitative analysis of RNN-based DL systems
This is among the earliest studies to propose multi-granularity testing criteria for DL systems.
[ASE 2018] Concolic testing for deep neural networks.
This paper presents the first concolic testing approach for Deep Neural Networks (DNNs).
A systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes.
The first differential fuzzing testing framework aiming to maximize the neuron coverage and generate more adversarial inputs for a given DL system, without cross-referencing other similar DL systems or manual labeling.
The first whitebox framework for systematically testing real-world DL systems.
The paper introduces the notion of frontier of behaviours and developed DeepJanus, a search-based tool that generates frontier inputs for DL systems.