I managed to install the docker images for testing and I get the correct result. However, when I try to run a simple bert model, I get the wrong answer. The iGPU is detected and used during the inference time, however, it generated completely wrong answers. Here is a simple example for reproduction:
from transformers import pipeline
unmasker = pipeline('fill-mask', model='google-bert/bert-base-uncased', device_map="cuda")
print(unmasker("Hello I'm a [MASK] model."))To investigate this problem, I also used another demo code (see below), and I found that the when I set HSA_OVERRIDE_GFX_VERSION to 11.5.0, the code will raise error:
HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
When I set HSA_OVERRIDE_GFX_VERSION to 11.0.0, the code runs successfully, but the result is wrong. I checked the generated ids and I found that all of them are set to zeros (including the input_ids), which is not expected.
My environment is:
- torch 2.7.1+rocm6.3
- transformers 4.53.1
Below is the demo code:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Qwen/Qwen3-0.6B"
# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
print(model.device)
# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# conduct text completion
generated_ids = model.generate(
**model_inputs,
max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
# parsing thinking content
try:
# rindex finding 151668 (</think>)
index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
index = 0
thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
print("thinking content:", thinking_content)
print("content:", content)
I managed to install the docker images for testing and I get the correct result. However, when I try to run a simple bert model, I get the wrong answer. The iGPU is detected and used during the inference time, however, it generated completely wrong answers. Here is a simple example for reproduction:
To investigate this problem, I also used another demo code (see below), and I found that the when I set
HSA_OVERRIDE_GFX_VERSIONto 11.5.0, the code will raise error:When I set
HSA_OVERRIDE_GFX_VERSIONto 11.0.0, the code runs successfully, but the result is wrong. I checked the generated ids and I found that all of them are set to zeros (including the input_ids), which is not expected.My environment is:
Below is the demo code: