Potential inconsistencies and annotation issues

[dlotfi]

First of all, thank you for making the PanTS dataset publicly available, it is an invaluable resource for the community.

While exploring the data, I came across a few issues that may be worth checking:

1. Empty lesion masks despite tumor? = 1 in metadata
45 CTs have tumor? marked as 1, but their pancreatic_lesion.nii.gz masks are empty:
PanTS_00000118, PanTS_00000291, PanTS_00000676, PanTS_00000678, PanTS_00000837, PanTS_00000876, PanTS_00002259, PanTS_00002320, PanTS_00002350, PanTS_00002537, PanTS_00002581, PanTS_00002622, PanTS_00002806, PanTS_00002996, PanTS_00003092, PanTS_00003142, PanTS_00003168, PanTS_00003361, PanTS_00003467, PanTS_00003770, PanTS_00003961, PanTS_00004288, PanTS_00004495, PanTS_00004583, PanTS_00004812, PanTS_00004831, PanTS_00005348, PanTS_00005395, PanTS_00005638, PanTS_00005667, PanTS_00005836, PanTS_00005948, PanTS_00006001, PanTS_00006005, PanTS_00006383, PanTS_00006389, PanTS_00006622, PanTS_00007037, PanTS_00007177, PanTS_00007191, PanTS_00007219, PanTS_00007241, PanTS_00007898, PanTS_00008070, PanTS_00008676

2. Unspecified CT phase
PanTS_00003188 has no ct phase information in the metadata. Was this field missed or intentionally left unspecified?

3. Missing mask
PanTS_00003188 appears to be missing a mask for the category celiac_artery.

4. Annotation discrepancies
I compared provided annotations with predictions from SuPreM and OrganSubSegmenter (both from your lab). DSC values were computed for common classes, and I observed a number of discrepancies.

Since checking all of them manually was not feasible, I reviewed a sample of cases. In some, the provided annotations were indeed correct compared to model predictions, but in others there were apparent errors that raised concerns.

The DSC results are attached for reference:
PanTS_Te_vs_SuPreM.csv
PanTS_Tr_vs_SuPreM.csv
PanTS_Te_vs_OrganSubSegmenter.csv
PanTS_Tr_vs_OrganSubSegmenter.csv

Thanks again for your invaluable work and for maintaining such a large and important dataset. I hope these notes are useful for further improving it.

[dlotfi]

While working with the PanTS dataset, I encountered a few additional issues, which I’m listing here.

5. Pancreas mask vs. pancreas components

• I computed the volumes for the pancreas mask and for its components (pancreas_head, pancreas_body, and pancreas_tail). I also calculated the fraction of each pancreatic_lesion that lies inside the pancreas mask (a value between 0 and 1), and the DSC between the pancreas mask and the union of its components (head + body + tail)
• Logically, the pancreas mask and the union of components should match closely (DSC ≈ 1), but I observed several cases with low DSC.
  Borderline cases occur when the pancreas mask is non-empty but all component masks are empty, or vice versa.
  This may indicate missing or inconsistent pancreas and pancreas component annotations in some cases.

6. Pancreas lesion inclusion
I am not certain how pancreatic_lesion was defined in the annotation protocol. I assumed lesions should normally fall within the pancreas mask. However, in many cases, the ratio of lesion voxels inside the pancreas is low.
If lesions were deliberately annotated outside the pancreas, explicitly documenting this could help avoid confusion.

The calculated volumes and other validation metrics are attached for reference:
PanTS_Te_Pancreas_Validation.csv
PanTS_Tr_Pancreas_Validation.csv

7. Affine matrix mismatches
Several mask files have affine matrices that differ from the corresponding image affine or from other class masks of the same case.
This issue appears particularly common for aorta masks and can cause misalignment when loading masks alongside the image volumes.

The lists of files with affine mismatches are attached:
PanTS_Te_Affine_Matrix_Mismatch.txt
PanTS_Tr_Affine_Matrix_Mismatch.txt

[WenxuanChelsea]

Hi @dlotfi

Thank you for your time and effort in working with the PanTS dataset. Your findings are very helpful and will guide us in improving its quality and consistency. We’ll address these issues and include the fixes in the next version on HuggingFace, and we’ll keep you updated once it’s available.

We’d also like to express our sincere thanks for your contribution — we plan to acknowledge you in the camera-ready version of the PanTS paper. If you’re interested, please email your name to wli131@jh.edu by Oct. 22, so we can include you properly 😊.

Thanks,
Wenxuan