GRPO Loss remains 0 during training

[adityavavre]

Hi,

I am using your code as is i.e. trying to train Qwen 2 VL 2B model on two GPUs with batch size 1. I notice that the GRPO loss starts at 0 and remains zero through atleast 72 steps after which I stopped training. Is this an issue?

[ByungKwanLee]

Could you clarify more detail. Did you use my original code?

I didnt experience that. You mean just 0? Not 0.03 like that?

[adityavavre]

Yes I am using your original code as is. Did not change the model or the dataset or any part of the training code. Just restricted the training to 2 GPUs. One for training and another for VLLM as specified.
The loss stays at 0.00.

[adityavavre]

Here is a screenshot of training for reference.

[ByungKwanLee]

this is my GRPO loss. I didn't understand why it is not changed

[ByungKwanLee]

could you share the environment setting i should explore it

[adityavavre]

I increased the batch size from 1 to 2, which resolved the issue. Is this the expected behavior, or could it be a bug?

[ByungKwanLee]

It shouldn't be. Thanks for reporting it. I will explore why it happens.

[johnsonkao0213]

Find the same issue when changing batch size to 1.

[liuweixue001]

Find the same issue when changing batch size to 1.

If only sample once, the advantage is zero, and the loss is zero.

[ByungKwanLee]

@adityavavre @liuweixue001 @johnsonkao0213
Thanks for your wait. I got to catch why it is happening. Qwen2-VL's decoding hyperparameter is the problem. You can increase the top_p to 0.9 and top_k to 50. It will be making more diversify the generated texts. Before it, the generated texts are not pretty diverse. That's the problem

[johnsonkao0213]

@ByungKwanLee Thanks for your reply. Will you report some experiments in README that you did on this repo and this dataset, so we can make sure the result is expected? For example, the training curve, the evaluation dataset you used, performance, the response length, and the analysis of aha moment... etc.

[ByungKwanLee]

@ByungKwanLee Thanks for your reply. Will you report some experiments in README that you did on this repo and this dataset, so we can make sure the result is expected? For example, the training curve, the evaluation dataset you used, performance, the response length, and the analysis of aha moment... etc.

I will have plans to do it. Let me update in readme later