Clarification on success criterion and evaluation scripts for UGR / H / D in Table 1

[mori0umi]

Hi, thanks for releasing the official implementation.

I’m trying to reproduce the 12 Contacts results in the main table of the paper, and I’d like to confirm the exact evaluation protocol used for UGR, H, and D.

I found that the IsaacLab evaluation already logs per-grasp Trials, Successes, and the three entropy terms, but I could not find the final aggregation script for the paper table, especially for D.

Could you please clarify:

1. What is the exact success criterion for a grasp?

   • Is a grasp successful when at least half of the evaluation trials succeed?
   • Or only when all trials succeed?

2. How is D computed?

   • Is it the mean maximum penetration over all grasps, only successful grasps, or something else?

If possible, could you also share the exact evaluation/aggregation script used to produce the Table 1 metrics? That would help a lot with faithful reproduction.

Thanks a lot.

[renezurbruegg]

Hi, thanks for reaching out, and thanks for your interest in reproducing the results.

To clarify the evaluation protocol:

Success criterion:

• A grasp is considered successful if at least half of the evaluation trials succeed. It does not need to succeed in all trials.
  Computation of D:
• D is computed as the average maximum penetration depth, averaged over all grasps per object and all objects. The maximum penetration is computed with respect to the object surface points.

I uploaded the relevant metric script here:
https://github.com/leggedrobotics/graspqp/blob/fix/eval/scripts/isaaclab/calc_metrics.py

A few additional notes: for the original paper, we used IsaacLab 4.2. Results obtained with IsaacLab 5.1 may differ slightly, since the PhysX backend has changed. We also observed some non-determinism with respect to environment origins and the ordering of grasps. For example, evaluating grasp 1 at origin 0 versus grasp 1 at origin 1 can lead to slightly different results. This is also one of the reasons why we use the “at least half of the trials successful” criterion. We discussed this behavior with the PhysX team, and it seems to be expected.

Finally, while working on a follow-up paper, we noticed an issue in the evaluation metrics for UGR and related quantities, where the discretization arguments were swapped. We fixed this in our latest version and pushed the corresponding changes to the fix/eval branch:
https://github.com/leggedrobotics/graspqp/tree/fix/eval

For new work, I would recommend using the updated metrics from that branch. Sorry for the inconvenience!

[mori0umi]

Hi, thanks for reaching out, and thanks for your interest in reproducing the results.

To clarify the evaluation protocol:

Success criterion:

• A grasp is considered successful if at least half of the evaluation trials succeed. It does not need to succeed in all trials.
  Computation of D:
• D is computed as the average maximum penetration depth, averaged over all grasps per object and all objects. The maximum penetration is computed with respect to the object surface points.

I uploaded the relevant metric script here: https://github.com/leggedrobotics/graspqp/blob/fix/eval/scripts/isaaclab/calc_metrics.py

A few additional notes: for the original paper, we used IsaacLab 4.2. Results obtained with IsaacLab 5.1 may differ slightly, since the PhysX backend has changed. We also observed some non-determinism with respect to environment origins and the ordering of grasps. For example, evaluating grasp 1 at origin 0 versus grasp 1 at origin 1 can lead to slightly different results. This is also one of the reasons why we use the “at least half of the trials successful” criterion. We discussed this behavior with the PhysX team, and it seems to be expected.

Finally, while working on a follow-up paper, we noticed an issue in the evaluation metrics for UGR and related quantities, where the discretization arguments were swapped. We fixed this in our latest version and pushed the corresponding changes to the fix/eval branch: https://github.com/leggedrobotics/graspqp/tree/fix/eval

For new work, I would recommend using the updated metrics from that branch. Sorry for the inconvenience!

Thanks for your patient and detailed reply. I have one more detail I need to confirm with you. Can I assume that for a certain object grasp generation task, the UGR in the paper is good_grasps_2cm_45deg_45deg / total_grasps, and D is mean(e_pen_max), which occur in the output of calc_metrics.py?

[renezurbruegg]

Yes, D is mean(e_pen_max).

For UGR, in DexEvolve (https://arxiv.org/abs/2602.15201) the USG@threshold is exactly this.

For the GraspQP paper it should be the output of https://github.com/leggedrobotics/graspqp/blob/main/graspqp_isaaclab/src/graspqp_isaaclab/agents/eval.py#L172-L181 when using calc_unique_grasps(joint_positions.clone(),hand_poses joint_positions.clone(), 0.02, 1.15° * 3.1415/180°, 4° * 3.1418/180°) / total_grasps.
However, I have since noticed that the argument for joint_pos was flipped in the old eval script (which got fixed in my later commit), so if possible i would strongly recommend to use the USG@threshold and newer calc-metrics script.