Request for 2-bit, 3-bit, and 4-bit quantized versions of LLaMA 3.1-70B and LLaMA 2-70B

[Hittopu]

First of all, thank you for the great work on this project!

I was wondering if it might be possible to share or release the 2-bit, 3-bit, and 4-bit quantized versions of the LLaMA 3.1-70B and LLaMA 2-70B models. Having these available would be incredibly helpful for my usage.

Thank you very much for your time and for sharing your excellent work with the community!

[liang2kl]

Thank you for you support. We will quantize and release these two models in 4 bits; however bit widths under 4 bit are not currently supported due to lack of kernels.

[Hittopu]

Thank you for you support. We will quantize and release these two models in 4 bits; however bit widths under 4 bit are not currently supported due to lack of kernels.

Thank you so much for the update and for planning to release the 4-bit models! We are really looking forward to them for efficient deployment.

We completely understand the kernel limitations for sub 4-bit models. If it isn't too much trouble, providing the fake quantized models under 2-bit and 3-bit would also be valuable for us to reproduce and benchmarking.

Thank you again for your time and excellent work!