Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery

Ming Hu^{1,2,3 *} · Zhengdi Yu *⁴ · Feilong Tang^1,2,3 · Kaiwen Chen⁵ · Yulong Li³ · Imran Razzak³ · Junjun He² · Tolga Birdal⁴ · Kaijing Zhou †⁵ · Zongyuan Ge †¹

¹Monash University · ²Shanghai AI Laboratory · ³MBZUAI · ⁴Imperial College London · ⁵Eye Hospital, Wenzhou Medical Univeristy

We introduce OphNet-3D, the first large-scale RGB-D dataset for dynamic 3D hand-instrument reconstruction in ophthalmic microsurgery, supported by an efficient multi-stage annotation pipeline, and propose novel architectures (H-Net and OH-Net) that significantly outperform existing methods in accurate hand and instrument reconstruction tasks.

News 🚩

• [2025-6-26] 🎉🎉🎉 OphNet-3D is accepted by NeurIPS 2025 as a Spotlight Paper!.
• [2025/5/26] Paper is now available. ⭐

TO DO

• Release dataset
• Release baseline experimental results and checkpoints

Data Download

• HuggingFace Mirror (optional, if you are in mainland China):

  export HF_ENDPOINT=https://hf-mirror.com

• Download:

  huggingface-cli download --repo-type dataset --resume-download xioamiyh/ophnet_3d  --revision main --local-dir ./
  

Data Structure

OphNet_3D
├── annoation
│   ├── 2024-11-25-23-05-08
│   │   ├── capsulorhexis
│   │   │   ├── -all-shot-0-0-500-hamer
│   │   │   ├── -all-shot-0-500-1500-hamer
│   │   │   ├── -all-shot-0-1500-1980-hamer
│   │   │   ├── fit_2d_concat
│   │   ├── cortex_removal
│   │   ├── ...
│   ├── 2024-11-26-17-14-13
│   ├── ...
├── data
│   ├── 2024-11-25-23-05-08
│   │   ├── capsulorhexis
│   │   │   ├── 043322072326
│   │   │   │   ├── Depth
│   │   │   │   │    ├── Depthaligned_62985251230241.png
│   │   │   │   │    ├── ...
│   │   │   │   ├── Mask
│   │   │   │   │    ├── aligned_62985251230241.png
│   │   │   │   │    ├── ...
│   │   │   │   ├── RGB
│   │   │   │   │    ├── aligned_62985251230241.png
│   │   │   │   │    ├── ...
│   │   │   ├── 043322072405
│   │   │   │   ├── Depth
│   │   │   │   ├── Mask
│   │   │   │   ├── RGB
│   │   │   ├── 044122070310
│   │   │   │   ├── ...
│   │   │   ├── 044122071255
│   │   │   │   ├── ...
│   │   │   ├── 044322070599
│   │   │   │   ├── ...
│   │   │   ├── 044322072556
│   │   │   │   ├── ...
│   │   │   ├── 233522070438
│   │   │   │   ├── ...
│   │   │   ├── 233622073339
│   │   │   │   ├── ...
│   │   ├── cortex_removal
│   │   ├── ...
│   ├── 2024-11-26-17-14-13
│   ├── ...
├── extrinsic
├── intrinsic
├── split.txt

Train

Inference

Citation

@article{hu2025towards,
  title={Towards dynamic 3d reconstruction of hand-instrument interaction in ophthalmic surgery},
  author={Hu, Ming and Yu, Zhengdi and Tang, Feilong and Chen, Kaiwen and Li, Yulong and Razzak, Imran and He, Junjun and Birdal, Tolga and Zhou, Kaijing and Ge, Zongyuan},
  journal={arXiv preprint arXiv:2505.17677},
  year={2025}
}

Contact

For any questions, please contact ming.hu@monash.edu or z.yu23@imperial.ac.uk .