I was trying to do some analysis with the train.json profiling and was unable to find the the backpropagation of weighted element-wise sum of moe outputs, which is the backpropagation of this calculation shown in the DeepSeek-V3 Technical Report:
My question is: is this neglected from the profiling on purpose or hidden somewhere in the deepEP operators?
My real motivation for this issue is to find out the data format for the MOE outputs, the activations saved for the gradient calculation of g_i,t shown in the equation above. Is it FP8, FP16/BF16, or even FP32?
I was trying to do some analysis with the train.json profiling and was unable to find the the backpropagation of weighted element-wise sum of moe outputs, which is the backpropagation of this calculation shown in the DeepSeek-V3 Technical Report:
My question is: is this neglected from the profiling on purpose or hidden somewhere in the deepEP operators?
My real motivation for this issue is to find out the data format for the MOE outputs, the activations saved for the gradient calculation of g_i,t shown in the equation above. Is it FP8, FP16/BF16, or even FP32?