Towards Effectively Testing Machine Translation Systems from White-Box Perspectives

This repo contains the data and the source code of the tool (i.e., GRI and WALI) used in our papers:

• Towards Effectively Testing Machine Translation Systems from White-Box Perspectives. [LINK] [PDF]
• Improving QA System Testing Efficiency Through White-Box Test Prioritization

The CAT folder contains all source files and data used to reproduce the results of CAT approach. The TransRepair folder contains all source files and data used to reproduce the results of TransRepair approach. The SIT folder contains all source files and data used to reproduce the results of SIT approach. The GRI contains all source files and data used to reproduce the results of GRI approach. The WALI contains all source files and data used to reproduce the results of WALI approach. The Labeled data folder contains the samples of data used in our paper for manual evaluation.

The Transformer model used for the implementation and evaluation can be found at transformer model

Requirements and Installation

git clone https://github.com/conf2024-8888/NMT-Testing.git
cd NMT-Testing
pip install -r requirements.txt

Replicate the results

1. download transformer model at transformer model and unzip the zipped file and put the model in the root directory of the repository.

2. Go to the subdirectory TransRepair, SIT, CAT, GRI or WALI

cd GRI

3. Run the pipeline.sh :

sh pipeline.sh

4. The bash file pipeline.sh inside each subfolder will trigger all source code to run

Cite our paper

If you use our code or data, please cite our paper as follows:

@article{shao2025towards,
  title={Towards effectively testing machine translation systems from white-box perspectives},
  author={Shao, Hanying and Ding, Zishuo and Shang, Weiyi and Yang, Jinqiu and Tsantalis, Nikolaos},
  journal={Empirical Software Engineering},
  volume={30},
  number={1},
  pages={1--38},
  year={2025},
  publisher={Springer}
}