How do I replicate the SHREC results given in the paper?

[FrozenWolf-Cyber]

The Baidu/one drive link was provided to download the SHREC demo dataset, but why isn't it matching the contest dataset from the official shrec competition (.mrc file size and coords files, don't seem to match anywhere)? Has any prior pre-processing been done on those shared datasets? If so, can you share the details?

Also, is it possible to share the trained model checkpoint for shrec dataset, to evaluate the shrec results?

[lgl603]

Since particles only exist in [156, 356] of z-axis, we crop $z \ in [156, 356]$ for the reconstruction of contest dataset and obtain SHREC demo dataset with smaller size. Meanwhile, the z-axis coordinate of particles in coords file is updated by z_raw-156. Since all particles are included in the demo datasets, the results is same as the contest dataset.

If you want to use the contest dataset, you need to convert PDB to class index by using the following dictionary before model training. And after model training, you need to convert class index back to PDB and you can calculate the metrics using the official eval.py.

classes = ['background', '4V94', '4CR2', '1QVR', '1BXN', '3CF3', '1U6G', '3D2F', '2CG9', '3H84', '3GL1', '3QM1', '1S3X', '5MRC', 'vesicle', 'fiducial']
num2pdb = {k: v for k, v in enumerate(classes)}
pdb2num = {v: k for k, v in num2pdb.items()}

[FrozenWolf-Cyber]

Great. And also, how are ocp_diameter and label_diameter chosen, and what do these diameters signify?

[FrozenWolf-Cyber]

Since particles only exist in [156, 356] of z-axis, we crop z in[156,356] for the reconstruction of contest dataset and obtain SHREC demo dataset with smaller size.

Has this also been done for test samples for final test benchmarking as reported in the paper? Ideally, for testing, wouldn't we want the model to predict those empty regions as background to get the False positive & True negative performance? If we remove those empty regions during, then we will be ignoring the possible False positives & True negatives

[lgl603]

Great. And also, how are ocp_diameter and label_diameter chosen, and what do these diameters signify?

label_diameter could set to be 9. ocp_diameter can be set as the real diameter for each class of particles. Official shrec competition has provided the occupancy_mask.mrc, which can used directly.

Note: the Number of classes should be set as 15. Since there are total 15 classes of objects, including 13 classes of particles as well as vesicle and fiducial for shrec2021 dataset.

[lgl603]

Since particles only exist in [156, 356] of z-axis, we crop z in[156,356] for the reconstruction of contest dataset and obtain SHREC demo dataset with smaller size.

Has this also been done for test samples for final test benchmarking as reported in the paper? Ideally, for testing, wouldn't we want the model to predict those empty regions as background to get the False positive & True negative performance? If we remove those empty regions during, then we will be ignoring the possible False positives & True negatives

This has been done for test samples for the Shrec sample dataset. We have tested on the raw test sample and no particles are picked from those empty regions of background. Alternatively, you can replace the test sample of model_9.mrc with the official shrec competition. The same results will be obtained.

[pratyush-1]

How did you map your .coords generated to .txt for evalation. Because in eval the 2 classes '4V94' and 'vesicle' are removed making it 14 classes (0 to 13 indexing). But doing this and mapping back to .txt file gives keyerror 14 (indicating 15 classes 0-15).

So can you tell about how you mapped your .coords back to .txt for evaluation

[lgl603]

How did you map your .coords generated to .txt for evalation. Because in eval the 2 classes '4V94' and 'vesicle' are removed making it 14 classes (0 to 13 indexing). But doing this and mapping back to .txt file gives keyerror 14 (indicating 15 classes 0-15).

So can you tell about how you mapped your .coords back to .txt for evaluation

classes = ['background', '4V94', '4CR2', '1QVR', '1BXN', '3CF3', '1U6G', '3D2F', '2CG9', '3H84', '3GL1', '3QM1', '1S3X', '5MRC', 'vesicle', 'fiducial']
num2pdb = {k: v for k, v in enumerate(classes)}
pdb2num = {v: k for k, v in num2pdb.items()}

You can change *.coords to *.txt directly. Then using num2pdb to transform the first column of *.txt from class number to pdb. Finally, run official eval.py with the input of *.txt

[pratyush-1]

Upon following that z_raw-156 and converting the model_9/particle_locations.txt there are like 1701 - 6(vesicle particles) making it 1695 particles after the mapping. However as predicted by the deepetpicker there are a total of 1531 particles only, so eval.py script isn't working and even removing '4V94' doesn't work as the numbers don't match could you tell how did you run the eval.py for model_9

[lgl603]

Upon following that z_raw-156 and converting the model_9/particle_locations.txt there are like 1701 - 6(vesicle particles) making it 1695 particles after the mapping. However as predicted by the deepetpicker there are a total of 1531 particles only, so eval.py script isn't working and even removing '4V94' doesn't work as the numbers don't match could you tell how did you run the eval.py for model_9

In the stage of evaluation, the number of predicted particles is not required to be the same as ground truth. This is because DeepETPicker usually could not pick all particles.