Eval issue on `lift_box`

[cosdeneb]

Hi, thank you for releasing the Tabletop-Sim codebases.

I am trying to reproduce the Tabletop-Sim evaluation protocol, and I noticed a potential issue with the Lift Box task. When I replay/evaluate the provided dataset demonstrations for lift_box using the current environment success criterion, the success rate appears to be 0%, even though the box is visibly lifted in the rollout videos.

The same replay/evaluation check works as expected for the other Tabletop tasks, so this seems specific to Lift Box. I was wondering whether this behavior is expected, or whether there may be a mismatch between the provided Lift Box demonstrations and the current success condition/evaluation code.

In particular, it seems possible that the box position is correct, but the orientation-related success condition is not satisfied. Could you please confirm the intended success criterion for Lift Box, and whether the released dataset demonstrations should achieve non-zero success when replayed in the current environment?

Thank you very much for your help.

[jellyho]

Hey, thanks for the great catch — you nailed it, the problem is the orientation success condition, not the demos.

Here's what happened: I recently fixed the quaternion convention in the Lift Box reward (MuJoCo stores quats as wxyz but SciPy's from_quat expects xyzw, so I added scalar_first=True). That part was fine — but in the same change I messed up the rotation threshold. The success criterion should be "box lifted, upright, front facing the camera" → roll/pitch/yaw all ≈ 0, but I accidentally made it require yaw ≈ 180

So, I believe now this should work as intended. Could you please check?

Thanks for your contribution :)