Performance variance in top-down (text-prompted) patch selection with custom images

[huhanwj]

Hi everyone,

First, thank you for this incredible work on PS3! I'm exploring the model's capabilities, particularly the top-down selection feature.

I've noticed a significant difference in the precision of text-prompted patch selection when using my own images compared to the dock.jpg example provided in the README.md.

While the demo image shows a very clear and accurate shift in the attention heatmap based on the text prompt, the model's selection on my images is often diffuse or doesn't strongly correlate with the prompt, even for seemingly simple queries.

My main question is: Is this performance variance expected for images that might be "out-of-distribution" compared to the (currently unreleased) pre-training dataset?

The train/README.md mentions the training data is not yet public, so I suspect this might be the core reason. Could you provide any insights on the types of images or scenes the model is most proficient with?

[bfshi]

Hi @huhanwj, yeah it's normal that the patch selection results are sometimes not accurate enough. The patch selection module is a pretty light-weight one to maintain the efficiency of the whole model, so the localization performance might not be as good as models that are trained for localization only.

The training data contains both natural images from SA-1B and DataComp, as well as document images from IDL and PDFA. The diversity of the data should be large enough for regular natural images but the capacity of the localization module as well as the localization supervision might be limiting the localization performance.

That being said, you can always choose to select more high-res patches is PS3 to reduce the probability of missing the regions that are relevant to the text prompt. You can do this by setting num_look_close and num_token_look_close in the forward function of PS3.