BUG

[Huster-Hq]

Traceback (most recent call last):
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1863, in _get_module
return importlib.import_module("." + module_name, self.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 38, in
from ...modeling_utils import PreTrainedModel
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/modeling_utils.py", line 50, in
from .integrations.flash_attention import flash_attention_forward
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/integrations/flash_attention.py", line 5, in
from ..modeling_flash_attention_utils import _flash_attention_forward
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/modeling_flash_attention_utils.py", line 30, in
from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/flash_attn/init.py", line 3, in
from flash_attn.flash_attn_interface import (
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/flash_attn/flash_attn_interface.py", line 15, in
import flash_attn_2_cuda as flash_attn_gpu
ImportError: /data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1011StorageImpl27throw_data_ptr_access_errorEv

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data/huqiang/code/VLM/R1-V/src/r1-v/src/open_r1/grpo.py", line 22, in
from transformers import Qwen2VLForConditionalGeneration
File "", line 1229, in _handle_fromlist
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1852, in getattr
value = getattr(module, name)
^^^^^^^^^^^^^^^^^^^^^
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1851, in getattr
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1865, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.qwen2_vl.modeling_qwen2_vl because of the following error (look up to see its traceback):
/data/huqiang/anaconda3/envs/r1-v/lib/python3.11/site-packages/flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1011StorageImpl27throw_data_ptr_access_errorEv

[a1wj1]

flash_attn_2的版本和cuda和python版本没对上，我是参考这个解决的：
https://blog.csdn.net/a61022706/article/details/141570792

[lezhang7]

flash_attn_2的版本和cuda和python版本没对上，我是参考这个解决的： https://blog.csdn.net/a61022706/article/details/141570792

我是python 3.11和cuda 12.1加上pytorch 2.5.1，安装flash_attn-2.7.4.post1+cu12torch2.5cxx11abiTRUE-cp311-cp311-linux_x86_64.whl还是遇到这个问题了

[lezhang7]

flash_attn_2的版本和cuda和python版本没对上，我是参考这个解决的： https://blog.csdn.net/a61022706/article/details/141570792

我是python 3.11和cuda 12.1加上pytorch 2.5.1，安装flash_attn-2.7.4.post1+cu12torch2.5cxx11abiTRUE-cp311-cp311-linux_x86_64.whl还是遇到这个问题了

fix this issue by install abiFALSE verision