Question regarding evaluation of SHTech

[kennyvoo]

Hello! Great work! For the few-shot anomaly detection on the ShanghaiTech dataset. I want to verify if my understanding of the evaluation protocol is correct:

1. For each camera view, randomly selecting 4 normal frames as training data. ( 12 cameras in total)

Could you confirm if this setup is correct, and are there any important details I might be missing?"

[xcyao00]

Yes. We sample 1 frame every 5 frames to obtain normal and abnormal images (using the testing dir in ShanghaiTech). Then you can randomly select 4 normal images as the reference images.

[kennyvoo]

Thank you for your reply. I am still unable to reproduce the result for the ShTech dataset. The average image AUC I obtained is around 0.6+. Below is how I load the data, keeping the transform the same as other dataset (resize and center crop). I've sampled every 5th frame of the test set as well.

Could it be the training frames that I've sampled causing the issue? Do I have to preprocess the training frames?

shtech.txt

    def _load_data(self, class_name):
        image_paths, labels, masks = [], [], []
        phase = 'training' if self.train else 'testing'

        scenes_dir_path = os.path.join(self.root, phase, "frames")

        if not self.train:
            mask_dir = os.path.join(self.root, phase, "test_pixel_mask")
            gt_dir = os.path.join(self.root, phase, "test_frame_mask")
        else:
            mask_dir = None
            gt_dir = None

        all_scenes_dirs = sorted(os.listdir(scenes_dir_path))
        # filter the scenes of the class
        class_scenes_dirs = [v for v in all_scenes_dirs if v.startswith(class_name)]

        for class_scene_dir in class_scenes_dirs:
            images_dir = os.path.join(scenes_dir_path, class_scene_dir)
            images_list = sorted([os.path.join(images_dir, f)
                                for f in os.listdir(images_dir)
                                if f.endswith('.jpg')])
            if not self.train:
                images_list = images_list[::self.sample_rate]
            image_paths.extend(images_list)

            if self.train:
                labels.extend([0] * len(images_list))
                masks.extend([None] * len(images_list))
            else:
                gt = np.load(os.path.join(gt_dir,f"{class_scene_dir}.npy"))[::self.sample_rate]
                mask_img = np.load(os.path.join(mask_dir,f"{class_scene_dir}.npy"))[::self.sample_rate]
                labels.extend(gt)
                masks.extend(mask_img)

        class_names = [class_name] * len(image_paths)
        return image_paths, labels, masks, class_names