Low quality of the visible mask images in BOP format

[Jing-lun]

Hi all, it's me again.

Thanks for your previous help!

Recently I made up the annotations for the HOPE-video dataset to generate a BOP format dataset for our research.
However, after I used the bop_toolkit to obtain the mask and visible-mask images of each view, I found that the quality of the visible images was low; meanwhile, the mask images looked nice and sound. Below is an example of RGB image, visible mask image, and mask image.

RGB Image (Orange Juice Box)

Visible Mask Image (Orange Juice Box)

Mask Image (Orange Juice Box)

It's clear to see that there are some erosions on the side in the visible mask image of the orange juice box.
So I wonder if you have ever met this issue or not.

[swtyree]

Hi, sorry for the delayed response. My toddler was home sick last week, and we all caught it in the end. 😷

My understanding of these two mask images is that the second mask only considers the known objects in the scene but the first mask incorporates the depth map to estimate the effect of unlabeled occluders in the scene. Since the depth image is noisy, the masks are lower quality. I think there is a threshold in the BOP script that can filter out some of that noise. But if I recall correctly, HOPE-Video doesn't have any external occluders (unlike HOPE-Image where objects are in bags or boxes, etc.) so the second version of the masks should be reliable.

[swtyree]

Also, @Jing-lun, as I was correcting the files per the other issue, I noticed that we didn't filter HOPE-Video annotations by visibility. Would you be willing to share the BOP-style annotations that you've generated? I'd be happy to include them in this repository (HOPE-Video won't be in the official BOP datasets) and credit you for generating them.

[Jing-lun]

Hi @swtyree, I hope you are well now.

Now it makes sense that the noisy depth image causes the low quality of the visible mask image. I'll try to increase the tolerance parameter in the BOP script to enhance the visibility of the mask images.

I would like to share the BOP-style HOPE-Video dataset! Please check the link below:
https://www.dropbox.com/s/yc4hl319jse60gw/HOPE-Video.zip?dl=0

P.S. When I made up the BOP-style annotations (scene_gt.json files for each scene), I found that the original pose information provided in each single annotation file (such as 0000.json/0001.json/...) must be modified. Specifically, the order of rotations vectors in each pose needs to be switched from [nx, ny, nz] to [nz, nx, ny] in order to make up a 'scene_gt.json' file.

[swtyree]

Thank you! I'll take a look later this afternoon. By the way, for these annotations, did you assume the meshes from this repo or the meshes from BOP? They are slightly different in both rotation and sometimes a slight translation. (BOP assumes origin is the center of the bounding box, while we originally align the axes with object symmetry. I think this only applies to the cans, which have the small protrusion on the top. We centered the origin on the vertical axis of the cylinder.) I have the transforms saved, and can apply those if needed.

[Jing-lun]

Thank you! I'll take a look later this afternoon. By the way, for these annotations, did you assume the meshes from this repo or the meshes from BOP? They are slightly different in both rotation and sometimes a slight translation. (BOP assumes origin is the center of the bounding box, while tried to align the axes with object symmetry. I think this only applies to the cans, which have the small protrusion on the top. We centered the origin on the vertical axis of the cylinder.) I have the transforms saved, and can apply those if needed.

I used the meshes from this repo because of the rich features it contains, so I have to re-center the axes to match with BOP-style.

[swtyree]

Hi @Jing-lun, I'm circling back around to some of the things I haven't cleaned up here. I'd like to upload a copy of the HOPE-Video dataset in BOP format. Do you still have the annotations that you generated last year? (The Dropbox link has expired.)

[Jing-lun]

Hi @Jing-lun, I'm circling back around to some of the things I haven't cleaned up here. I'd like to upload a copy of the HOPE-Video dataset in BOP format. Do you still have the annotations that you generated last year? (The Dropbox link has expired.)

Not a problem @swtyree! Please check this link for the dataset I generated!

[AnasIbrahim]

Hi @swtyree,

I converted and regenerated the HOPE-Image dataset in the BOP format. I wanted to regenerate the masks but without the BOP toolkit. I needed better visible masks because I'm benchmarking with an RGB-only object detector and the current masks are noisy. This is a reason because the BOP toolkit uses the depth image for the generation and the depth images as usual are noisy. So I used Open3D directly to reconstruct the 3D scene and capture fake depth images. This works without problem as the current data has no occlusion from the environment and the occlusion is only from other objects. The segmentation mask accuracy got much better with regenerated masks. Actually, the quality of the 6D annotations is still not perfect, but at least the visible masks are better.

Here is a link to the regenrated data: https://drive.google.com/file/d/1WDKrinmxjf5uzhO1EHK_SZQdrBSuZrwr/view?usp=drive_link
Here is the code I wrote to regenerate it: https://github.com/AnasIbrahim/hope_2_bop_format