[Bug] Training error occurs when using the latest compiled code for RF-DETR

[wuhaoyu010]

Description
I compiled the latest code from the repository and attempted to train the RF-DETR model, but encountered an error.
Screenshot

Training Code

from libreyolo import LibreRFDETR
def main():
    model = LibreRFDETR("weights/rf-detr-large-2026.pth")
    result = model.train(
        data="datasets",
        epochs=100,
        batch=16,
        device="4,5",
        imgsz=640,
        lr=1e-4,
        output_dir="runs/train/rfdetr_exp",
    )
    print(f"Training completed: {result}")
if __name__ == "__main__":
    main()

Additional Context
I have tried all three pre-trained models available in LibreYOLO/LibreRFDETR, but the same error occurs with all of them. Could you please help me identify what might be causing this issue? Thanks!

[imagra93]

Hi, thanks for the detailed report!

We are are working in this bug — when loading a weight file directly (e.g. rf-detr-large-2026.pth) without specifying size=, the model defaults to size="s" (3 decoder layers) instead of the correct size="l" (4 decoder layers), causing the unexpected-keys error on transformer.decoder.layers.3.*. The same mismatch applies to the medium and nano checkpoints.

A fix will be merged to dev today. After the update, LibreRFDETR("weights/rf-detr-large-2026.pth") will automatically detect the correct size from the filename — no code changes needed on your end.

In the meantime, you can work around it by passing the size explicitly:

model = LibreRFDETR("weights/rf-detr-large-2026.pth", size="l")
model = LibreRFDETR("weights/rf-detr-medium.pth", size="m")
model = LibreRFDETR("weights/rf-detr-nano.pth", size="n")

[wuhaoyu010]

Hello，你的来信已收到 

[wuhaoyu010]

Thanks for the reply. However, after applying the changes, I am still encountering the following error.
Training Code

from libreyolo import LibreRFDETR
def main():
    model = LibreRFDETR("/home/ubuntu/train-test/paddle_test/yolo/weights/LibreRFDETRl-seg.pt", size="l", task='segment')
    model.train(
        data="data_rfdetr_seg.yaml",
        epochs=200,
        batch_size=40,
        imgsz=624,
        lr=1e-4,
        device="0,1,2,3",
        output_dir="runs/train/rfdetr_exp",
    )
if __name__ == "__main__":
    main()

Additional Info

• The imgsz parameter is not set in the data_rfdetr_seg.yaml file.
• The error occurs consistently during training.
  Error Log

python train_rfdetrseg.py
2026-05-28 13:00:24 | INFO     | DDP active: rank=0/4 device=cuda:0
2026-05-28 13:00:24 | INFO     | Setting up training...
2026-05-28 13:00:24 | INFO     | Loading 2855 YOLO annotations from train/labels...
Loading YOLO annotations (train/labels): 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2855/2855 [00:05<00:00, 535.35it/s]
2026-05-28 13:00:31 | INFO     | Loaded 2855 YOLO annotations from train/labels in 6.17s
2026-05-28 13:00:31 | INFO     | Training dataset: 2855 images
2026-05-28 13:00:31 | INFO     | Iterations per epoch: 71 (batch_per_rank=10, world_size=4)
2026-05-28 13:00:31 | INFO     | Wrapped model in DDP (find_unused_parameters=True, static_graph=False, gradient_as_bucket_view=False)
2026-05-28 13:00:31 | INFO     | Using mixed precision training (AMP)
2026-05-28 13:00:31 | INFO     | Using EMA with decay=0.993, tau=100
2026-05-28 13:00:31 | INFO     | Saving to: runs/train/rfdetr_exp
2026-05-28 13:00:31 | INFO     | Starting training for 200 epochs
2026-05-28 13:00:31 | INFO     | Model: LibreRFDETR-l
2026-05-28 13:00:31 | INFO     | Batch size: 40
2026-05-28 13:00:31 | INFO     | Learning rate: 0.0001
Epoch 1/200:   0%|                                                                                                                                         | 0/71 [00:10<?, ?it/s]
W0528 13:00:47.055000 1450765 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 1452284 via signal SIGTERM
W0528 13:00:47.056000 1450765 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 1452286 via signal SIGTERM
W0528 13:00:47.056000 1450765 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 1452287 via signal SIGTERM
Traceback (most recent call last):
  File "/home/ubuntu/train-test/yolo_seg/train_rfdetrseg.py", line 16, in <module>
    main()
  File "/home/ubuntu/train-test/yolo_seg/train_rfdetrseg.py", line 5, in main
    model.train(
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/base/model.py", line 72, in wrapper
    return train_fn(self, *args, **user_kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 314, in wrapper
    return spawn_for_model(self, train_kw, len(devices), devices=devices, batch_key=batch_key)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 219, in spawn_for_model
    spawn_ddp_train(
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/distributed.py", line 354, in spawn_ddp_train
    mp.spawn(
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 211, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 87, in _wrap
    fn(i, *args)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 68, in _libreyolo_ddp_worker
    result = model.train(**train_kw)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/base/model.py", line 72, in wrapper
    return train_fn(self, *args, **user_kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 316, in wrapper
    return train_fn(self, *args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/model.py", line 680, in train
    result = trainer.train()
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/trainer.py", line 681, in train
    epoch_result = self._train_epoch(epoch)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/trainer.py", line 1099, in _train_epoch
    outputs = self.on_forward(imgs, targets, polygons=polygons)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/trainer.py", line 365, in on_forward
    outputs = self.model(imgs, targets=target_list)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1832, in forward
    else self._run_ddp_forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1657, in _run_ddp_forward
    return self.module(*inputs, **kwargs)  # type: ignore[index]
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/nn.py", line 359, in forward
    return self.model(x, targets=targets)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/lwdetr.py", line 293, in forward
    features, poss = self.backbone(samples)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/backbone.py", line 792, in forward
    x = self[0](tensor_list)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/backbone.py", line 716, in forward
    feats = self.encoder(tensor_list.tensors)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/backbone.py", line 598, in forward
    assert x.shape[2] % block_size == 0 and x.shape[3] % block_size == 0, (
AssertionError: Backbone requires input shape to be divisible by 24, but got torch.Size([10, 3, 512, 512]) 

Do you know what might be causing this? It seems that the imgsz training parameter is not taking effect?

I am using the latest code compiled from today’s repository (commit e65f632

[imagra93]

First of all, thank you for reporting this and for using LibreRFDETR — this kind of detailed bug report (with logs, a minimal repro, and the exact commit) makes it much easier to track down the root cause quickly. Really appreciated.

A proper fix is being tracked in #294.

Until then, you can unblock training with two small changes:

1 — Disable multi-scale jitter by passing multi_scale=False to model.train(). This bypasses _multi_scale_scales() entirely so the wrong divisor is never computed and the backbone assertion never fires.

2 — Pin imgsz in your data yaml because the imgsz you pass to model.train() is currently overridden by the model's default (e.g. 504 for l-seg). Until the fix lands, add this line to data_rfdetr_seg.yaml:

imgsz: 624
Your training call then becomes:

model.train(
    data="data_rfdetr_seg.yaml",
    epochs=200,
    batch_size=40,
    imgsz=624,          # will be honoured once PR #294 is merged
    lr=1e-4,
    device="0,1,2,3",
    output_dir="runs/train/rfdetr_exp",
    multi_scale=False,  # workaround for the DDP patch_size bug
)

or

Train at the model's native resolution (504) if you don't strictly need 624. This requires no workarounds — just omit imgsz entirely and let the model use its default.

[wuhaoyu010]

Thanks for your response. However, after applying the changes mentioned above, I encountered the following error during retraining. Could this be an issue with the official RF-DETR repository? How can I bypass this bug?
Error Log:

python train_rfdetrseg.py
2026-05-28 14:08:58 | INFO     | DDP active: rank=0/4 device=cuda:0
2026-05-28 14:08:58 | INFO     | Setting up training...
2026-05-28 14:08:58 | INFO     | Loading 2855 YOLO annotations from train/labels...
Loading YOLO annotations (train/labels): 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2855/2855 [00:02<00:00, 1088.29it/s]
2026-05-28 14:09:01 | INFO     | Loaded 2855 YOLO annotations from train/labels in 2.81s
2026-05-28 14:09:01 | INFO     | Training dataset: 2855 images
2026-05-28 14:09:01 | INFO     | Iterations per epoch: 71 (batch_per_rank=10, world_size=4)
2026-05-28 14:09:02 | INFO     | Wrapped model in DDP (find_unused_parameters=True, static_graph=False, gradient_as_bucket_view=False)
2026-05-28 14:09:02 | INFO     | Using mixed precision training (AMP)
2026-05-28 14:09:02 | INFO     | Using EMA with decay=0.993, tau=100
2026-05-28 14:09:02 | INFO     | Saving to: runs/train/rfdetr_exp
2026-05-28 14:09:02 | INFO     | Starting training for 200 epochs
2026-05-28 14:09:02 | INFO     | Model: LibreRFDETR-l
2026-05-28 14:09:02 | INFO     | Batch size: 40
2026-05-28 14:09:02 | INFO     | Learning rate: 0.0001
Epoch 1/200:   0%|                                                                                                                                         | 0/71 [00:00<?, ?it/s][transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
Epoch 1/200:   0%|                                                                                                                                         | 0/71 [00:09<?, ?it/s]
W0528 14:09:14.425000 2004120 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 2004343 via signal SIGTERM
W0528 14:09:14.425000 2004120 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 2004345 via signal SIGTERM
W0528 14:09:14.425000 2004120 site-packages/torch/multiprocessing/spawn.py:165] Terminating process 2004346 via signal SIGTERM
Traceback (most recent call last):
  File "/home/ubuntu/train-test/yolo_seg/train_rfdetrseg.py", line 16, in <module>
    main()
  File "/home/ubuntu/train-test/yolo_seg/train_rfdetrseg.py", line 5, in main
    model.train(
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/base/model.py", line 72, in wrapper
    return train_fn(self, *args, **user_kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 314, in wrapper
    return spawn_for_model(self, train_kw, len(devices), devices=devices, batch_key=batch_key)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 219, in spawn_for_model
    spawn_ddp_train(
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/distributed.py", line 354, in spawn_ddp_train
    mp.spawn(
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 211, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 87, in _wrap
    fn(i, *args)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 68, in _libreyolo_ddp_worker
    result = model.train(**train_kw)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/base/model.py", line 72, in wrapper
    return train_fn(self, *args, **user_kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/ddp_spawn.py", line 316, in wrapper
    return train_fn(self, *args, **kwargs)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/models/rfdetr/model.py", line 680, in train
    result = trainer.train()
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/trainer.py", line 681, in train
    epoch_result = self._train_epoch(epoch)
  File "/home/ubuntu/train-test/libreyolo/libreyolo/training/trainer.py", line 1103, in _train_epoch
    self.scaler.scale(loss).backward()
  File "/usr/local/lib/python3.10/site-packages/torch/_tensor.py", line 631, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/site-packages/torch/autograd/__init__.py", line 379, in backward
    _engine_run_backward(
  File "/usr/local/lib/python3.10/site-packages/torch/autograd/graph.py", line 882, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 280 with name model.segmentation_head.bias has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.

[imagra93]

Thanks for the follow-up and the detailed logs — this confirms a third bug on top of the two we already fixed.

The new error (Expected to mark a variable ready only once) is a DDP reducer crash unrelated to imgsz. The RF-DETR segmentation head runs in two branches per forward pass, so the same parameters receive gradients twice. With find_unused_parameters=True DDP registers a hook on each parameter that fires both times, which is illegal.

The fix is to initialise DDP with static_graph=True instead — this locks the reducer graph after the first iteration and handles the two-branch pattern correctly. It's already in the pipeline and will be included in the next merge to dev.

In the meantime, the safest workaround is to train on a single GPU and scale batch size accordingly. Multi-GPU training with multi_scale=False still hits this crash because the issue is in the backward pass, not the scale computation.

[wuhaoyu010]

Issue: DDP Training Crash with Unlabeled Images & VRAM Explosion During Evaluation with High-Resolution Images

Environment

• libreyolo: 1.2.0
• torch: 2.6.0+cu126
• torchmetrics: 1.9.0
• torchvision: 0.21.0+cu126
• Python: 3.10

---

Issue 1: DDP RuntimeError Caused by Unlabeled Images in Segmentation Training

Description

When training LibreRFDETR with the segmentation option enabled, the training process crashes at the first iteration with a DDP RuntimeError. The root cause appears to be the presence of unlabeled images (images without corresponding annotation files) in the training dataset, which leads to unused parameters in certain iterations — incompatible with static_graph=True in DDP.
The same dataset trains without issues using the official RF-DETR segmentation code.

Reproduction

from libreyolo import LibreRFDETR
def main():
    model = LibreRFDETR(
        "/home/ubuntu/train-test/paddle_test/yolo/weights/LibreRFDETRn-seg.pt",
        size="n",
        segmentation=True,
    )
    model.train(
        data="data_rfdetr_seg.yaml",
        epochs=100,
        batch_size=48,
        lr=1e-4,
        device="0,1,2,3",
        output_dir="runs/train/rfdetr_exp",
    )
if __name__ == "__main__":
    main()

Error Log

2026-06-02 07:50:58 | INFO     | DDP active: rank=0/4 device=cuda:0
2026-06-02 07:50:58 | INFO     | Setting up training...
2026-06-02 07:50:58 | INFO     | Loading 2855 YOLO annotations from train/labels...
Loading YOLO annotations (train/labels): 100%|██████████| 2855/2855 [00:01<00:00, 2816.58it/s]
2026-06-02 07:50:59 | INFO     | Loaded 2855 YOLO annotations from train/labels in 1.05s
2026-06-02 07:50:59 | INFO     | Training dataset: 2855 images
2026-06-02 07:50:59 | INFO     | Iterations per epoch: 356 (batch_per_rank=2, world_size=4)
2026-06-02 07:51:00 | INFO     | Wrapped model in DDP (find_unused_parameters=False, static_graph=True, gradient_as_bucket_view=False)
2026-06-02 07:51:00 | INFO     | Using mixed precision training (AMP)
2026-06-02 07:51:00 | INFO     | Using EMA with decay=0.993, tau=100
2026-06-02 07:51:00 | INFO     | Saving to: runs/train/rfdetr_exp
2026-06-02 07:51:00 | INFO     | Starting training for 100 epochs
2026-06-02 07:51:00 | INFO     | Model: LibreRFDETR-n
2026-06-02 07:51:00 | INFO     | Batch size: 8
2026-06-02 07:51:00 | INFO     | Learning rate: 0.0001
Epoch 1/100:   0%|  | 0/356 [00:00<?, ?it/s]
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
[rank3]:[W602 07:51:02.632853313 reducer.cpp:327] Warning: Grad strides do not match bucket view strides. ...
[rank1]:[W602 07:51:02.681756946 reducer.cpp:327] Warning: Grad strides do not match bucket view strides. ...
[rank0]:[W602 07:51:02.761718747 reducer.cpp:327] Warning: Grad strides do not match bucket view strides. ...
Epoch 1/100:   0%|  | 1/356 [00:02<12:20, 2.09s/it, loss=97.0014, lr=0.000100, ce=12.9352, bbox=1.3107, giou=3.6105, mask_ce=1.5664, mask_dice=2.1438]
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.
This error indicates that your training graph has changed in this iteration, e.g., one parameter is used in first iteration, but then got unused in the second iteration.
This is not compatible with static_graph set to True.
Parameter indices which did not receive grad for rank 2: 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

Workaround

Removing unlabeled images from the dataset resolves the crash. After cleaning up the dataset, training proceeds normally.

Issue 2: VRAM Overflow During Evaluation with High-Resolution Images & Extremely Slow Evaluation

Description

When training with drone-captured high-resolution images (e.g., 4000×3000), the evaluation phase causes a VRAM overflow (OOM). Even after reducing the batch size to avoid OOM, the evaluation is extremely slow — e.g., ~2 minutes of training but ~20 minutes of evaluation on only 130 validation images (with 2000+ training images).
This issue also exists in the official RF-DETR codebase. It does not occur with YOLOv8-seg, which handles evaluation quickly and without VRAM issues.

Workaround

Pre-processing images by resizing them to 640px (matching YOLO's training resolution) before training resolves both the VRAM overflow and the slow evaluation.

Questions / Discussion

I'm unsure whether these two preprocessing-related fixes should be incorporated into libreyolo itself, or whether they are the responsibility of the user to handle before training:

1. Auto-filtering unlabeled images: Should the dataloader automatically skip images without corresponding labels, or should this be a user responsibility?
2. Auto-resizing high-resolution images: Should the library impose a max input resolution limit or auto-resize during evaluation to prevent OOM, or is this purely a user-side data preparation concern?
   Any guidance on the intended design philosophy here would be greatly appreciated. Thank you!

[imagra93]

Thanks for the detailed report and the clear reproduction steps — very helpful!

Issue 1 — DDP crash with unlabeled images in segmentation training

This is fixed in PR #323. There were two cooperating root causes:

Loss gradient disconnect: when a batch has zero Hungarian matches, loss_masks was returning a disconnected torch.tensor([]) with no grad_fn, silently dropping the mask-head parameters from the backward graph and breaking static_graph=True. Fixed by routing the zero losses through the actual mask-head tensors so every parameter still receives a (zero) gradient.

Unlabeled images reaching the dataloader: fixed by automatically filtering zero-annotation images at dataset construction time when segmentation is enabled, with a warning logged. This removes the root cause rather than just handling it in the loss.

Issue 2 — VRAM OOM / slow evaluation with high-resolution images

This is a known upstream RF-DETR limitation. Pre-resizing images to your training resolution (as you already found) is the right workaround for now. My inclination is to leave this as a user-side data preparation step for the time being, but happy to hear other opinions — feel free to keep the discussion going here.

Thanks again for writing this up so thoroughly!