About the Dataset Annotation of 3D objects

[kasteric]

Hi, thanks for your inspiring work! I have a question regarding the mismatch between the 3d affordance annotation and the actual affordance depicted in the 2d interaction image. As introduced, a 2d interaction image and a 3d object is paired up in training set, the problem is: when 2d interaction images of the same affordance type (i.e. grasp) actually have two different contact regions as shown below, how can we make sure that they have the correct 3D affordance ground-truth of shape (2048,) given that the 3D affordace gts are collected from 3D affordance Net?

[yyvhang]

Hi, In order to learn a consistent affordance distribution, the annotated regions in 3D-AffordanceNet are almost uniform. If you want to align with the image, perhaps you can consider manually annotating some data, the annotation method is introduced in another work: LEMON, please refer to its supp.

[kasteric]

Thanks for your response! I went through the supplementary of LEMON about the affordance annotation, which mostly follows 3D-AffordanceNet. One thing that confuses me is the label propagation, where it seems that the recorded coordinates of keypoints were not used, but only the parts that were associated with the affordance type were used (one-hot label of shape N*18). Is my understanding correct?

[yyvhang]

Yeah, we refer to the annotation pipeline in 3D-AffordanceNet but there are also differences. The affordance value of key points is Initialized to 1 so that the affordance can propagate to specific parts. The CAD objects used in 3D-AffordanceNet have part annotation (the propagable region), but some of our 3D data can not segment specific parts. Therefore, we first outline a region as key points, and then outline a propagable region for propagation. I will consider releasing the code for calculating the affordance propagation next month. Stay tuned!