In this Guided Research, we analyse 6 commonly used Explainability methods if they show disparate fidelity across the male and female genders in their explanations. We evaluate the methods using metrics implemented from Ferret (https://ferret.readthedocs.io/en/latest/).
We have 2 experimental setups, all present in the notebooks in the Repository. The names of the notebooks contain the names of the datasets or setup type which should be explanatory enough. The Python script, pipeline.py contains the functions for each stage of the pipeline. I provide an additional notebook, 'Pipeline_test' which demonstrates the usage of the script.
The notebook 'stats_test_combined' contains the statistical test results for each Dataset in Experimental Setup 1, and for each model in Experimental Setup 2. The details of each setup are mentioned in the Report.
The FineTuned_GR zip folder contains the files for the fine-tuned TinyBERT model on the Amazon Reviews split_2 dataset as per Setup 2.
There are 2 folders as well, in addition to the notebooks in the main Repo.
- Datasets: The datafiles used in the project. Please note that, the names and paths of these files will not be consistent with the ones used in the notebooks, since I just provide the used portions of the data. The names of the files should be indicative of the dataset. Please change the names of the files and the paths in the notebooks. The report contains details of each dataset.
- Scratch: Redundant or irrelevant experiments in notebooks.
While running the experiments again, please note that the exact values obtained for the average metric scores for the datasets/models may be different from the ones reported here (in decimal places), but that is to be expected. Once the entire pipeline is run, and the statistical tests performed, those results would be similar to ours.
I have used Poetry to setup a pyproject.toml file which lists the requirements and dependancies of the packages needed in this work. The steps to install the packages using Poetry:
git clone https://gitlab.lrz.de/pratik.sen/guidedresearch.git
cd guidedresearch
curl -sSL https://install.python-poetry.org | python3 -
poetry install
Please contact me directly if you have any questions.
Pratik Sen (pratik.sen@tum.de)