Figure Dip Bending Problem

[LyingMoon]

I noticed that when I fully stretch my hand (i.e., all joint angles are at 0), the retargeted values still cause the DIP joints to bend to around 90 degrees. Since the FK model seems correct, I suspect there's something in the retargeting network training that's preventing the robot hand from achieving a fully stretched pose.

The following is running on the allegro model in this repo, it is still OK (In this case I didn't use your PID controller and directly map the retarget joints to the robot model (no feedback control used)). But you still can see the DIP joints are still tricky

Screencast.from.07-24-2025.12.37.41.PM.webm

I am curious if this can be resolved because I have tuned a lot of hyper parameters but that doesn't seem to work

[zhaohengyin]

Thanks for the issue! Can you please check whether the corresponding joint produced by GeoRT is changing (e.g. via print)? I just noticed that Sapien may have some simulation issue with customized URDFs. This will affect the visualization (last joint of each finger becomes frozen). #12

[LyingMoon]

Hi Zhaoheng, thanks for your reply. The following is a video demonstration of this problem (using replay_evaluation.py and I add some functionality like raw data visualization and FK pinochino/FK network comparison for debug)

https://github.com/user-attachments/assets/489e2eb9-c5d1-46b5-9159-28107e4e9d38
https://github.com/user-attachments/assets/e08d0937-13b5-47fd-b994-6fcc470098e8

dip_joint_indices = [3, 7, 11] # Index, Middle, Ring DIP joints (thumb has no DIP)

You can see that the retargeting is giving a value that lead to figure bending
I feel like the problem lies within the trained retargeting network. Especially when the dataset contains the finger pose that is hypertension (out of the joint limit of the robot model). I validate the FK network with my joint state publisher gui and I feel like that is not the problem

[zhaohengyin]

Thanks for the reply! I see your point now.

I believe this issue may stem from a corner case in the loss function that hasn’t been fully addressed yet. Specifically, there's a tricky situation that arises near the boundary of the Cartesian human configuration space (C-space).

In our current implementation, we randomly sample a point x from the human C-space and a random direction vector d. We then enforce a directional consistency loss: f(x + d) - f(x) should be parallel to d. The goal is for the network to preserve the intended motion direction in the mapped robot C-space.

However, when x is at the boundary of the human C-space, the perturbed point x + d can easily fall outside the valid human C-space. This causes the problem. If the network maps this boundary point x to another boundary point f(x) in the robot C-space, it may not be able to find a valid f(x + d) that preserves the direction d (because this will require f(x+d) out of robot C-space!). As a result, the network may instead map x to a point deeper inside the robot C-space to satisfy the directional constraint—leading to the bending behavior we’re seeing.

[zhaohengyin]

To solve this, we need to mask out bad direction d at boundary point x (i.e. x+d out of human C-space). However, since this did not affect our teleoperation experience so far, we have not implemented this fix yet.

[goodfor7]

To solve this, we need to mask out bad direction d at boundary point x (i.e. x+d out of human C-space). However, since this did not affect our teleoperation experience so far, we have not implemented this fix yet.

Hi @zhaohengyin ,

What about the bending issue mentioned earlier when mapping in reality rather than in simulation? For example, when I extend all five fingers, does the dexterous hand in reality also display all five fingers extended? Could you provide a video for reference?