Questions Regarding the Use of QuIP#

[MoleSir]

Hi, and thanks for providing such an excellent paper and releasing the Caldera code.

[MoleSir]

The paper states:“This is achieved by first applying SVD-LLM to the LLM under a 40% compression ratio, and then applying QuIP# for 2-bit quantization on the compressed model generated from SVD-LLM.”

However, based on my understanding, QuIP# operates on the original weight matrix W ($m \times n$) and the Hessian ($n \times n$). After applying SVD-LLM, W ($m \times n$) is decomposed and compressed into U ($m \times r$) and V ($r \times n$). Therefore, I would like to ask how QuIP# is applied to U and V for further compression.