[Recommendation] StoSignSGD: Sign-based Low-precision LLM Optimizer

[lumeilevel]

Blog title

StoSignSGD: Sign-based Low-precision LLM Optimizer

Canonical URL

https://lumeilevel.github.io/StoSignSGD/

Are you the author?

Yes, I wrote this post

Author / Lab / Organization

Dingzhi Yu, Rui Pan, Yuxing Liu, Difan Zou, Tong Zhang

Source name

No response

Suggested BlogrXiv category

Efficient AI

Suggested tags

Low-precision training, LLMs Optimizer, FP4, FP8

Short summary

Trouble training LLMs in FP4 or FP8? Try StoSignSGD!

Why is this technically valuable?

StoSignSGD uses structured stochasticity to remove the bias of deterministic sign updates, fixing SignSGD’s nonsmooth convergence issue with optimal theoretical convergence. At the same time, its sign-based design plus numerical friendly max op avoids fragile second-moment normalization, making it especially robust for aggressive FP8/FP4 training.

Related paper / code, optional

https://arxiv.org/abs/2604.15416

Submission confirmation

• The submitted URL is publicly accessible.
• If I am the author, I confirm that I have the right to submit this post.
• I understand that BlogrXiv may accept, reject, summarize, categorize, or request revisions.

[hq-King]

Thanks for sharing, we will add this to blogrxiv next week

[lumeilevel]

Thank you very much for your time and your service to BlogrXiv!