DA-Fusion: Deformable Attention-based RGB-D Fusion Transformer for Unseen Object Instance Segmentation
Yesol Park1* · Hye-Jung Yoon1* · Juno Kim1* · Byoung-Tak Zhang1,2,3
1Interdisciplinary Program in AI, Seoul National University · 2AI Institute, SNU · 3Dept. of Computer Science, SNU
*Equal contribution
IEEE ICRA 2025
TL;DR: DA-Fusion fuses RGB and depth at every backbone stage with deformable self- and cross-attention, combining texture and geometry to segment unseen objects in cluttered scenes — reducing the over-segmentation of RGB-only methods and the under-segmentation of depth-only ones.
In logistics automation, precise segmentation of unseen objects is crucial for efficient robotic manipulation in cluttered environments. Tasks such as bin-picking and shelf-picking require robust perception to handle occlusions, varying object shapes, and complex spatial arrangements. Traditional RGB-based methods tend to over-segment objects due to their reliance on texture, while depth-based methods often under-segment by focusing primarily on geometric features. To address these limitations, we propose DA-Fusion, a deformable attention-based RGB-D fusion Transformer designed for unseen object instance segmentation. DA-Fusion effectively combines the strengths of both RGB and depth data, enhancing segmentation accuracy in cluttered and multi-layered object environments. We also introduce the Object Clutter Bin Dataset (OCBD), a benchmark dataset specifically tailored for evaluating bin-picking scenarios in top-down views. Extensive evaluations demonstrate that DA-Fusion outperforms state-of-the-art methods across diverse environments, making it particularly suited for real-world logistics tasks.
DA-Fusion extracts features from the RGB and depth inputs through two parallel Swin Transformer branches and fuses them at every stage using a deformable attention mechanism: Deformable Self-Attention (DS) refines each modality's features, and Deformable Cross-Attention (DC) exchanges information across modalities. The multi-scale fused features are decoded by a Mask Transformer decoder to produce class-agnostic instance masks. This dynamic, multi-level fusion integrates texture and geometry to reduce both the over-segmentation of RGB-only models and the under-segmentation of depth-only models.
Setup uses uv with Python 3.10 (PyTorch 2.4 / CUDA 12.4, with Detectron2 built from source). The full environment is scripted:
bash dafusion/scripts/setup_env.sh # creates dafusion/.venvThis installs torch==2.4.1 + torchvision==0.19.1 (cu124), Detectron2 (editable, from a source
clone), the DA-Fusion package, and the deformable-attention CUDA op. Place the Swin-L COCO
initialization checkpoint under data/checkpoints/mask2former/.
DA-Fusion is trained on UOAIS-Sim and evaluated on OCID, OSD, and our OCBD.
Datasets live under data/UOIS/ (a symlink; data/ is not tracked by git):
data/UOIS/
├── UOAIS-Sim/ # training — tabletop + bin scenes, COCO-style annotations
├── OCID-dataset/ # evaluation
├── OSD-0.20-depth/ # evaluation
└── OCBD/ # evaluation (ours)
Run from the repository root.
# Train on UOAIS-Sim (RGB-D, Swin-L); auto-runs the eval sweep after training
CONFIG=configs/dafusion_rgbd_uoais.yaml NUM_GPUS=4 bash dafusion/scripts/train.sh --eval-after
# Evaluate a single checkpoint on one benchmark
bash dafusion/scripts/eval.sh --config dafusion/configs/dafusion_rgbd_uoais.yaml \
--dataset ocid --weights <path/to/model.pth>
# Sweep every checkpoint of a run across GPUs (OCID / OSD / OCBD)
bash dafusion/scripts/eval_sweep.sh --config dafusion/configs/dafusion_rgbd_uoais.yamlDA-Fusion is evaluated with Overlap and Boundary Precision / Recall / F-measure and the percentage of segments with Overlap F ≥ 75% (F@75).
| Benchmark | Overlap P | Overlap R | Overlap F | Boundary F | F@75 |
|---|---|---|---|---|---|
| OCID | 93.2 | 92.6 | 92.1 | 90.0 | 92.7 |
| OSD | 93.5 | 92.4 | 92.9 | 88.0 | 92.9 |
| OCBD | 92.5 | 90.4 | 91.3 | 88.7 | 87.1 |
Reproduced numbers may deviate slightly from the paper due to GPU type and inherent randomness in training.
dafusion/
├── dafusion/ # package: modeling (dual-Swin + DS/DC fusion), data, engine, eval
├── configs/ # training / evaluation configs (Swin-L, RGB-D)
├── scripts/ # setup / train / eval entry points
└── third_party/ # Detectron2 (built from source)
docs/ # paper figures + PDF
data -> # datasets & checkpoints (symlink; not tracked)
If you find this work useful, please cite:
@inproceedings{park2025dafusion,
title = {DA-Fusion: Deformable Attention-based RGB-D Fusion Transformer for Unseen Object Instance Segmentation},
author = {Park, Yesol and Yoon, Hye-Jung and Kim, Juno and Zhang, Byoung-Tak},
booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
year = {2025}
}DA-Fusion builds on Mask2Former, Detectron2, Swin Transformer, Deformable Attention (DAT), and the UOAIS benchmark suite. We thank the authors for releasing their code and datasets.
Released under the MIT License. Vendored third-party components remain under their respective licenses.