BDetCLIP (ICML 2025)

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
https://arxiv.org/abs/2405.15269

Abstract: *Abstract: While multimodal contrastive learning methods (e.g., CLIP) can achieve impressive zero-shot classification performance, recent research has revealed that these methods are vulnerable to backdoor attacks. To defend against backdoor attacks on CLIP, existing defense methods focus on either the pre-training stage or the fine-tuning stage, which would unfortunately cause high computational costs due to numerous parameter updates and are not applicable in black-box settings. In this paper, we provide the first attempt at a computationally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically find that the visual representations of backdoored images are .insensitive to benign and malignant changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specifically, we first prompt a language model (e.g., GPT-4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially designed instructions. Then, the distribution difference in cosine similarity between images and the two types of class description texts can be used as the criterion to detect backdoor samples. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency. *

Licenses

You can use, redistribute, and adapt the material for non-commercial purposes, as long as you give appropriate credit by citing our paper and indicating any changes that you've made.

Requirements

• Both Linux and Windows are supported, but we strongly recommend Linux for performance and compatibility reasons.
• 64-bit Python 3.7+ installation.

Setup Environment and Install Dependencies

Conda (recommended)

Please follow the instructions at the following link to set up anaconda: Anaconda Setup The following commands create a conda environment inside the repository with the dependencies.

conda env create --prefix ./env -f environment.yml
source activate ./env

Pip

The requirements can be directly installed without creating a conda environment.

pip install -r requirements.txt

Backdoor Attack Models

For backdoor attacks on CLIP, you can refer to https://github.com/LiangSiyuan21/BadCLIP/tree/master. We provide some pre-trained backdoored model weights here.

Detection - ImageNet1K

python ./src/main.py \
  --checkpoint /path/to/your/banana_badnet_rn50.pt \
  --patch_name ./badCLIP.jpg\
  --patch_type ours_tnature \
  --patch_location middle \
  --bd_ratio 0.3 \
  --add_backdoor \
  --eval_data_type ImageNet1K \
  --eval_test_data_dir /path/to/your/imagenet/validation \
  --device cuda:0

python ./src/main.py \
  --checkpoint /path/to/your/banana_badnet_rn50.pt \
  --patch_type random \
  --patch_location random \
  --bd_ratio 0.3 \
  --add_backdoor \
  --eval_data_type ImageNet1K \
  --eval_test_data_dir /path/to/your/imagenet/validation \
  --device cuda:0

python ./src/main.py \
  --checkpoint /path/to/your/banana_blended_rn50.pt \
  --patch_type blended \
  --patch_location blended \
  --bd_ratio 0.3 \
  --add_backdoor \
  --eval_data_type ImageNet1K \
  --eval_test_data_dir /path/to/your/imagenet/validation \
  --device cuda:0

For ImageNet1K: There should be a labels.csv in the test data directory that contains 2 columns -- image, label. image should have the location to the image in the local machine.

Acknowledgements

Our work builds upon the excellent contributions from https://github.com/LiangSiyuan21/BadCLIP/tree/master and https://github.com/nishadsinghi/CleanCLIP/tree/main. We sincerely thank the authors for their outstanding work!