Abnormal Too Large Grad Norm

[HowCCC]

Thank you for your great work; it has been incredibly helpful!

I have a question regarding training on your released dataset. During training, the gradient norm is mostly restricted to 0.5. However, at certain steps, the gradient spikes significantly, as shown in the attached image. This disrupts the training process and even triggers a PyTorch watchdog interrupt.

Could you please provide some guidance or advice on how to resolve this issue?

[hyuanChen]

It’s normal since the data isn’t 100% clean.
We already skip such steps based on the gradient norm, so it won’t affect the final training results.

[HowCCC]

It’s normal since the data isn’t 100% clean. We already skip such steps based on the gradient norm, so it won’t affect the final training results.

Thanks for your reply, by the way, could you please clariy for me that the reason of unclean data? Maybe I could resolve it by myself.

[hyuanChen]

It’s normal since the data isn’t 100% clean. We already skip such steps based on the gradient norm, so it won’t affect the final training results.

Thanks for your reply, by the way, could you please clariy for me that the reason of unclean data? Maybe I could resolve it by myself.

The cause is the image-to-point-cloud misalignment during rendering, as well as errors in the initial point cloud acquisition.
You may need to record the corrupted data separately and exclude them from the training set.

[HowCCC]

It’s normal since the data isn’t 100% clean. We already skip such steps based on the gradient norm, so it won’t affect the final training results.

Thanks for your reply, by the way, could you please clariy for me that the reason of unclean data? Maybe I could resolve it by myself.

The cause is the image-to-point-cloud misalignment during rendering, as well as errors in the initial point cloud acquisition. You may need to record the corrupted data separately and exclude them from the training set.

Thanks a lot