RADIO Metrics

[mranzinger]

Hello,

Thank you so much for this paper, as well as OmniProbe. I'm working on integrating it into my tooling. I like SOC as a metric.

One thing I've noticed: I'm pretty sure that your Table 2 is reporting metrics for C-RADIOv3-B, not the -H. Table r5 shows the identical metrics, but correctly lists "c_radio_3_b" as the model that produced it. However, at least Table r10 seems to suggest that those results are for the -H model (not the -B).

Section A.6 seems to suggest that C-RADIOv4 will be present in Table r5, however, I only see "c_radio_3_b" and "radio". I'm not sure what "radio" refers to.

Using your repo, this is what I get, including replications for 4 tested models:

Model	My SOC / CC	Paper SOC / CC
dinov2_b14	60.43 / 78.91	60.4 / 78.9 ✓
dinov2_l14	62.41 / 80.69	62.4 / 80.7 ✓
clip_b16	16.06 / 24.93	16.1 / 24.9 ✓
c_radio_3_b	51.05 / 69.01	51.1 / 69.0 ✓ (paper's "C-RADIOv3")
c_radio_3_l	58.75 / 76.76	—
c_radio_3_h	60.92 / 77.94	—
radio (paper Table r5)	—	55.54 / 72.53
c_radio_4_h	61.68 / 78.21	—

[odunkel]

Hey,

Thanks a lot for reaching out!

Yes, you are right: wherever possible, we report results for the base model versions in the main paper.

Regarding radio, apologies for the confusion.
We originally evaluated AM-RADIOv2 in our pipeline and therefore also included it in the supplementary material. However, in OmniProbe, we initially observed unexpectedly low performance for AM-RADIOv2 in the linear-probing setting for ADE20K semantic segmentation. We were able to resolve this by adding batch normalization to the linear segmentation head. However, since we did not observe this issue for other models, we decided, for consistency, to leave AM-RADIOv2 out of the initial OmniProbe release. Maybe you have an explanation for why this adjustment is specifically important for AM-RADIOv2?

In any case, we are currently working on additional functionality and potential refactoring of the codebase. As part of this process, we are also re-evaluating the models using the updated implementation. This may result in slightly different numbers from the ones reported in the current paper.
Based on those evaluations, we will also document and maintain up-to-date model evaluations in this repository.

Other than that, we would also be, of course, happy to hear any feedback or suggestions on how we could improve the framework and make probing comparisons across different backbones as fair as possible.

Once our rework is complete, we will update the arXiv version of the paper and, of course, clarify the RADIO model naming to prevent further confusion.

I hope this addresses your concerns.

Thanks again for reaching out and all the best,
Olaf

[mranzinger]

Thanks for clarifying. I think it's mostly just a problem then with Table r10, which is incorrectly stating that you evaluated our -H sized model, not the -B.

AM-RADIOv2 had the mode switching problem that we outlined in the RADIOv2.5 paper. So, my educated guess is that you're seeing that. Perhaps given that AM-RADIOv2 is quite old, I would advocate to just remove it from the analysis. You've already covered what we think is our best -B sized model, which is C-RADIOv3-B.

One suggestion I have: I think it would be good to control for the number of patches that the vision encoder processes, as opposed to the resolution. I noticed this because you're evaluating both patch-14 and patch-16 models. If you increase the resolution of the patch-16 models to match the resulting number of patches, at least C-RADIOv3-B and DINOv3 improve on their tasks. It still doesn't resolve the other quite surprising result of your paper, which is that DINOv2 is quite a bit better than DINOv3, but it does close the gap slightly.

[odunkel]

That is true. We will update it. Thank you for the careful check.

That is a fair point as well. We analyzed that effect internally (eg correspondence_soco.yaml supports a fixed_patched_size). But it might, indeed, valuable more clearly presenting this.

Regarding DINOv2 vs. DINOv3: Table r5 shows that DINOv3-L is actually pretty close to DINOv2-L, even in the fixed resolution setting, which is different to the distilled base variants.

[mranzinger]

Oh, wow, yeah, you're right. There's a big jump for DINOv3-L. Still a pretty big gap (IMO) on CC, but yeah, SOC is much closer. Curious indeed.