I'm applying this framework to the 3D domain, but I've encountered a problem. I've customized an nn.Module class, using the language portion of qwen2.5-vl-7b, deleting the visual portion, and defining my own 3D encoder and connector. The code runs successfully, but it runs out of VRAM when I try to roll 8 times. I tried enabling deepspeed-zero3, but it throws an error.
[rank1]: RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [0] and normalized_shape = [3584]
I've done some exploration and experimentation, and found that in the original qwen2.5-vl model, after applying Zero3, the size of the weights in both the visual and language parts became 0. I suspect this is because Zero3 performs a slicing operation. However, after directly replacing my model, I found that only the language part's weights became 0; the visual part's weights didn't become 0. I suspect this is due to a problem with slice management. After using with deepspeed.zero.Init(dtype = torch.bfloat16): to force slice management, the visual part's weights in my model also became 0, but I still get this error. How can I solve this?
If I use zero2, it will throw an error.
RuntimeError: torch.cat(): expected a non-empty list of Tensors
I'm applying this framework to the 3D domain, but I've encountered a problem. I've customized an
nn.Moduleclass, using the language portion ofqwen2.5-vl-7b, deleting the visual portion, and defining my own 3D encoder and connector. The code runs successfully, but it runs out of VRAM when I try to roll 8 times. I tried enabling deepspeed-zero3, but it throws an error.[rank1]: RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [0] and normalized_shape = [3584]
I've done some exploration and experimentation, and found that in the original qwen2.5-vl model, after applying Zero3, the size of the weights in both the visual and language parts became 0. I suspect this is because Zero3 performs a slicing operation. However, after directly replacing my model, I found that only the language part's weights became 0; the visual part's weights didn't become 0. I suspect this is due to a problem with slice management. After using
with deepspeed.zero.Init(dtype = torch.bfloat16):to force slice management, the visual part's weights in my model also became 0, but I still get this error. How can I solve this?If I use zero2, it will throw an error.
RuntimeError: torch.cat(): expected a non-empty list of Tensors