What is the motivation behind replacing the fp16 Neumann inverse with an 8x8 fp32 forward substitution

[zilanlia]

Hi, I noticed that matrix inversion in kernel1 has changed from Neumann inverse to forward substitution. There appears to be little difference in data precision error between these two methods. May I ask what the motivation behind this change is?

Does forward substitution offer better performance?

[asp0ex]

We found that the Neumann-series approach can have worse numerical accuracy in some cases, e.g. with 16 consecutive identical tokens. So we switched back to forward substitution for better numerical robustness. This comes with roughly a 1% end-to-end performance regression.