The metrics of retrained cogact on calvin are always 0

[NaNaoiSong]

Hi Authors, thank you for the well-organized VLA codebase.

I have tested the official cogact_calvin checkpoint and was able to reproduce the reported results. However, after retraining the model myself, the evaluation metrics consistently remain at 0. I would greatly appreciate any guidance on what might be causing this issue or how to resolve it. Thank you very much for your time and support, and I look forward to your response.

[yuzhms]

Could you please provide more detailed testing information, such as evaluation videos, predicted actions, and the logs from both the model and the simulator during inference?

When the evaluation result is consistently 0, it is usually caused by a configuration mismatch. We recommend checking whether the model’s normalization is correctly loaded and whether the simulator inference configuration matches the training setup.

[NaNaoiSong]

I think my mistake is the same as #58. I also trained some exps on libero, and only PI can be evaluated normally.

I noticed a difference in the data transformation pipeline between pi0 and cogact (or memvla). Specifically, the order of DeltaAction and AddTrajectory is swapped, which results in different action supervision—one being relative to the first frame and the other relative to the preceding frame.

Could this discrepancy be the cause of the training failure?

[EmberNoWeither]

Yeah, only pi works.

[yuzhms]

On the DexBotic Benchmark, due to format differences between models, Pi and other models were evaluated using different test configurations. For example, CogACT used the following config:

👉 https://github.com/dexmal/dexbotic-benchmark/blob/main/evaluation/configs/libero/libero_10.yaml

[EmberNoWeither]

I noticed another issue: why does the pi0 model require post-processing during inference to convert the output into absolute actions, while CogACT directly uses the de-normalized action outputs? In the dexbotic-benchmark implementation there is indeed a use_delta parameter, but in the LIBERO evaluation this parameter is not used—instead, it directly uses the action returned by the model (regardless of whether it represents delta motion or an absolute position). In most inference setups, shouldn’t the model generally output delta EEF actions for control?

Finally, thank you for your outstanding open-source work and for your patience in answering!