Didn't match in peg insertion task, why is that?

[pussinmood]

Thanks for your sharing spirit and remarkable work!
I tried to train the arp+ model with two 4090, but when I evaluated it, the success rate just reached 79% total， and peg insertion was just 40%， just in the middle of the rvt2 and the arp. I want to know where I got wrong and how could I improve the rate.
Really hope to get your advices.

[mlzxy]

Have you tried evaluating multiple checkpoints? The last checkpoint sometimes does not give you the best result. What I do is that I evaluate all checkpoints from 60000 steps, and evaluate at least three times for each checkpoint.

Also, consider running another train to see any changes if possible. If this issue persists, I will try to retrain myself and compare with some old codebases to try figuring out.

[mlzxy]

BTW, what batch size do you use? I trained this model with 4x 40GB A100s. Using 2x 4090s may mean a smaller batch size?

[mlzxy]

By default, I use 4 GPUs and each GPU has a batch size of 48. The learning rate is the config is per sample as shown in here. Therefore, if you use a smaller batch size, you need to increase the learning rate accordingly. For example, if you use 2 GPUs with batch size of 24 on each, you need to multiply the current learning rate by 4.

[pussinmood]

Thanks a lot for your advice. I will try this out. Have a good day

[Walter0807]

Thanks for your sharing spirit and remarkable work! I tried to train the arp+ model with two 4090, but when I evaluated it, the success rate just reached 79% total， and peg insertion was just 40%， just in the middle of the rvt2 and the arp. I want to know where I got wrong and how could I improve the rate. Really hope to get your advices.

I encountered the same situation.

I ran the following command on two 80G GPUs:

python3 train.py config=./configs/arp_plus.yaml hydra.job.name=arp_plus train.num_gpus=2 train.bs=96

The performance was still below expectations, especially on the peg insertion task.

Additionally, I tried using six 32G GPUs with the following command:

python3 train.py config=./configs/arp_plus.yaml hydra.job.name=arp_plus train.num_gpus=6 train.bs=32

The results were better but still slightly lower than the released checkpoint.

Any suggestions or thoughts would be greatly appreciated!

[Walter0807]

Thanks for your sharing spirit and remarkable work! I tried to train the arp+ model with two 4090, but when I evaluated it, the success rate just reached 79% total， and peg insertion was just 40%， just in the middle of the rvt2 and the arp. I want to know where I got wrong and how could I improve the rate. Really hope to get your advices.

I encountered the same situation.

I ran the following command on two 80G GPUs:

python3 train.py config=./configs/arp_plus.yaml hydra.job.name=arp_plus train.num_gpus=2 train.bs=96
The performance was still below expectations, especially on the peg insertion task.

Additionally, I tried using six 32G GPUs with the following command:

python3 train.py config=./configs/arp_plus.yaml hydra.job.name=arp_plus train.num_gpus=6 train.bs=32
The results were better but still slightly lower than the released checkpoint.

Any suggestions or thoughts would be greatly appreciated!

I can reproduce the performance now, thanks!