HW Exception by GPU node-1 (Agent handle: 0x55dc6e221990) reason :GPU Hang on AMD Ryzen 7 7840HS w/ Radeon 780M Graphics

[Ep0chalypse]

I'm experiencing this issue when trying to run stable diffusion. I'm using the rdna3 docker image(via podman) on Almalinux 10 running the latest kernel(6.15.6-1.el10.x86_64). The cpu is a AMD Ryzen 7 7840HS w/ Radeon 780M Graphics. In the bios I've allocated the max memory to the GPU(8GB). Possibly related to #141

The pytorch hello world example works:

root@32249188555d:/# cd /opt/rocm_sdk_612/docs/examples/pytorch/
root@32249188555d:/opt/rocm_sdk_612/docs/examples/pytorch# source  /opt/rocm_sdk_612/bin/env_rocm.sh 
root@32249188555d:/opt/rocm_sdk_612/docs/examples/pytorch# ./pytorch_gpu_hello_world.sh
Pytorch version: 2.4.1 
ROCM HIP version: 6.1.40093-d104dca71
cuda device count: 1
default cuda device name: AMD Radeon 780M
device type: cuda
Tensor training running on cuda: True 
Running simple model training test
tensor([0., 1., 2.], device='cuda:0')  
Hello World, test executed succesfully

The full output is:

root@32249188555d:~# sd -M txt2img -p "a cat" -m /opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors -o ~/Pictures/test_cat.png
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon 780M, compute capability 11.0, VMM: no
[INFO ] stable-diffusion.cpp:195  - loading model from '/opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors'
[INFO ] model.cpp:793  - load /opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:235  - Version: SD 1.x
[INFO ] stable-diffusion.cpp:266  - Weight type:                 f16
[INFO ] stable-diffusion.cpp:267  - Conditioner weight type:     f16
[INFO ] stable-diffusion.cpp:268  - Diffusion model weight type: f16
[INFO ] stable-diffusion.cpp:269  - VAE weight type:             f16
[INFO ] stable-diffusion.cpp:482  - total params memory size = 1969.78MB (VRAM 1969.78MB, RAM 0.00MB): clip 235.06MB(VRAM), unet 1640.25MB(VRAM, vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:501  - loading model from '/opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors' completed, taking 0.61s
[INFO ] stable-diffusion.cpp:528  - running in eps-prediction mode
[INFO ] stable-diffusion.cpp:655  - Attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:1132 - apply_loras completed, taking 0.00s
HW Exception by GPU node-1 (Agent handle: 0x55dc6e221990) reason :GPU Hang
Aborted (core dumped)

Steps to reproduce:

• podman pull lamikr/rocm_sdk_builder:612_01_rdna3
• podman run -it --device=/dev/kfd --device=/dev/dri --group-add keep-groups --security-opt label=type:container_runtime_t docker.io/lamikr/rocm_sdk_builder:612_01_rdna3 bash

[chboishabba]

Hello this repo is no longer maintained. However if you are seeing similar errors in dmesg as 141 then this is likely related to your kernel version. You may have better luck trying the latest LTS, but patches related to doorbell and reset have been added to rc and should be rolled into main kernel around the end of the month, pending your distribution.

…

On Wed, Jul 16, 2025, 12:59 PM Ep0chalypse ***@***.***> wrote: *Ep0chalypse* created an issue (lamikr/rocm_sdk_builder#257) <#257> I'm experiencing this issue when trying to run stable diffusion. I'm using the rdna3 docker image(via podman) on Almalinux 10 running the latest kernel(6.15.6-1.el10.x86_64). The cpu is a AMD Ryzen 7 7840HS w/ Radeon 780M Graphics. Possibly related to #141 <#141> The pytorch hello world example works: ***@***.***:/# cd /opt/rocm_sdk_612/docs/examples/pytorch/ ***@***.***:/opt/rocm_sdk_612/docs/examples/pytorch# source /opt/rocm_sdk_612/bin/env_rocm.sh ***@***.***:/opt/rocm_sdk_612/docs/examples/pytorch# ./pytorch_gpu_hello_world.sh Pytorch version: 2.4.1 ROCM HIP version: 6.1.40093-d104dca71 cuda device count: 1 default cuda device name: AMD Radeon 780M device type: cuda Tensor training running on cuda: True Running simple model training test tensor([0., 1., 2.], device='cuda:0') Hello World, test executed succesfully The full output is: ***@***.***:~# sd -M txt2img -p "a cat" -m /opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors -o ~/Pictures/test_cat.png ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon 780M, compute capability 11.0, VMM: no [INFO ] stable-diffusion.cpp:195 - loading model from '/opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors' [INFO ] model.cpp:793 - load /opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors using safetensors format [INFO ] stable-diffusion.cpp:235 - Version: SD 1.x [INFO ] stable-diffusion.cpp:266 - Weight type: f16 [INFO ] stable-diffusion.cpp:267 - Conditioner weight type: f16 [INFO ] stable-diffusion.cpp:268 - Diffusion model weight type: f16 [INFO ] stable-diffusion.cpp:269 - VAE weight type: f16 [INFO ] stable-diffusion.cpp:482 - total params memory size = 1969.78MB (VRAM 1969.78MB, RAM 0.00MB): clip 235.06MB(VRAM), unet 1640.25MB(VRAM, vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM) [INFO ] stable-diffusion.cpp:501 - loading model from '/opt/rocm_sdk_models/sd-v1-5-pruned-emaonly-fp16.safetensors' completed, taking 0.61s [INFO ] stable-diffusion.cpp:528 - running in eps-prediction mode [INFO ] stable-diffusion.cpp:655 - Attempting to apply 0 LoRAs [INFO ] stable-diffusion.cpp:1132 - apply_loras completed, taking 0.00s HW Exception by GPU node-1 (Agent handle: 0x55dc6e221990) reason :GPU Hang Aborted (core dumped) Steps to reproduce: - podman pull lamikr/rocm_sdk_builder:612_01_rdna3 - podman run -it --device=/dev/kfd --device=/dev/dri --group-add keep-groups --security-opt label=type:container_runtime_t docker.io/lamikr/rocm_sdk_builder:612_01_rdna3 bash — Reply to this email directly, view it on GitHub <#257>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGM4B3VE5RVHD42KMIGZYHD3IW5ZLAVCNFSM6AAAAACBTTICVKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGIZTIMRXGY2DSMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Ep0chalypse]

Thanks for the reply, I didn't realize this project wasn't maintained any more.

You may have better luck trying the latest LTS, but patches related to
doorbell and reset have been added to rc and should be rolled into main
kernel around the end of the month, pending your distribution.

Sorry if this is a dumb question but I'm not familiar with this stuff.. Are you saying a older LTS kernel may be better right now, but patches have been submitted to the kernel that may also help, but those patches wont be included in the kernel until the end of the month?

[chboishabba]

I'm really quite out of the loop as to what hardware corresponds to which generation. It seems like 780M is a newer card? I came to this repo because I have an RX580.

I threw your thread into an existing GPT thread I used to submit the code for kernel consideration (not my code/bisect).

If this doesn't answer, please feel free to ask again:

#AI

**Your 780 M “HW Exception → GPU Hang” shows the same pattern we just dissected for Polaris—but the root causes diverge: on RDNA 3 APUs the hang is a mix of (1) immature kernel‑space support (the KFD door‑bell/reset work that broke gfx8 still leaves gaps for gfx11), and (2) the fact that ROCm only recently gained experimental Radeon 780 M coverage, so older kernels / ROCm builds fall off AMD’s own support matrix. In practice that means a 6.15‑line kernel + ROCm 6.1.4 Docker image can start PyTorch “hello‑world” (lightweight HIP) but stalls the much heavier Stable‑Diffusion weights and then dereferences a NULL callback during create_queue_cpsch, aborting with the familiar HW Exception by GPU node‑1. Switching to the long‑term 6.6‑stable kernel or a 6.16‑rc with the July door‑bell/reset fixes—plus ROCm 6.4.1 or newer, which is the first release to list consumer RDNA 3 devices—has been enough for other 780 M owners to run SD pipelines without hitting that bug. Below you’ll see how the pieces fit together, why the maintainer told the reporter to “try LTS for now”, and when the definitive fixes should land in mainstream distributions.

---

1 What the GitHub issue is showing

Trace element	Meaning	Evidence
HW Exception by GPU node‑1 (Agent 0x55dc…) reason : GPU Hang	Runtime (ROCr) detected a fence that never completed; it escalates to a “GPU Hang” message and aborts the process.	Same wording appears in other 780 M hang reports.
Works for PyTorch demo but not Stable‑Diffusion	Small demo allocates <100 MB, SD allocates ≈2 GB VRAM (see log line), stressing queue management / SDMA reset.	The log shows clip+unet+vae totals 1969 MB VRAM.
Kernel 6.15.6	Between the two regressing commits de84484 and bac38ca, but before the gfx11 door‑bell helpers landed (they land in 6.16‑rc3).	6.15.6 tag date 10 Jul 2025; door‑bell patch 25 Jun 2025 but only merged to drm‑tip then to 6.16‑rc3.
Docker image rocm_sdk_builder:612_01_rdna3	Ships ROCm 6.1.4 HIP stack. AMD’s compatibility matrix flags 780 M support as “preview” starting 6.3 and “official” only in 6.4.1.	ROCm docs & release notes.

---

5 Recommended steps right now

1. Kernel
   Boot 6.6‑LTS (or Fedora/Alma’s kernel-lts), or build 6.16‑rc3+ with CONFIG_DRM_AMDGPU_SDMA_QUEUE_DEBUG=y so you can confirm the hang disappears.

2. ROCm stack
   Move to ROCm 6.4.1 Docker image (AMD’s first release that officially lists RDNA 3 consumer GPUs).

3. BIOS / kernel flags for APUs
   8 GB UMA, disable “ASF”/“Min UMA” scaling; add amdgpu.noretry=0 amdgpu.gpu_recovery=1 to kernel cmdline to keep compute queues from hard‐wedge loops.

4. Re‑test Stable‑Diffusion
   If you still see “GPU Hang”, capture dmesg -T and open a new ROCm issue referencing CVE‑2025‑21940 and patch 1e9d17a5dcf1 so maintainers can verify gfx11 back‑port coverage.

---

6 Key sources consulted

1. Freedesktop door‑bell patch thread

2. ROCm 780 M hang reports (#4557, #4444)

3. ROCm 6.4.1 release notes & compatibility matrix

4. CVE‑2025‑21940 advisories & kernel fix IDs

5. Phoronix coverage of ROCm and RDNA 3 support timeline

6. Kernel 6.15.6 tag & stable cadence discussions

7. Lists.freedesktop.org proposal to rename HQD slots (shows ongoing gfx11 queue work)

(Sources cited inline after each statement.)

[Ep0chalypse]

Thanks! That was actually quite helpful. I'll see about trying a LTS kernel or using ROCM 6.1.4 until those patches land in a stable kernel release.

[MCredbear]

Hi, try to reduce your GTT in BIOS, that might works.

[chboishabba]

Hi, try to reduce your GTT in BIOS, that might works.

It's not a result of corruption. You're trying to do unsupported OPs
I believe the relevant kernel has landed for arch but I haven't been game to test yet.

[MCredbear]

Hi, try to reduce your GTT in BIOS, that might works.

It's not a result of corruption. You're trying to do unsupported OPs

Yeah, but on Debian 13 with 6.14 kernel, this just works sometimes for my 8845HS 780M.

[chboishabba]

Yes you may find eg some memcpy ops will work but the compute will deliver an invalid result. Users have had success with I believe roc 5.7.1 and LTS kernel also. This is the generally suggested setup for the older cards as this was when compute was deprecated for older gcn. For me, attempting to run ML causes a kernel panic that crashes the whole machine to the point where I had to rescue the fs journal. This is because of a bug in kfd reset flag. I believe the patches are rolled into current kernel RC and should hit mainline in .8

…

On Tue, 22 July 2025, 2:22 pm MCredbear, ***@***.***> wrote: *MCredbear* left a comment (lamikr/rocm_sdk_builder#257) <#257 (comment)> Hi, try to reduce your GTT in BIOS, that might works. It's not a result of corruption. You're trying to do unsupported OPs Yeah, but on Debian 13 with 6.14 kernel, this just works sometimes. — Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGM4B3SLML5KPUKPD7WYIQT3JW37FAVCNFSM6AAAAACBTTICVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMBQHA2TGNRRGM> . You are receiving this because you commented.Message ID: ***@***.***>

[MCredbear]

Hi, try to reduce your GTT in BIOS, that might works.

It's not a result of corruption. You're trying to do unsupported OPs

Yeah, but on Debian 13 with 6.14 kernel, this just works sometimes for my 8845HS 780M.

I figured out why: Pytorch cannot use GTT, so increasing GTT acctually reduces VRAM that Pytorch can use, and the error from ROCM is too 'simply' which just reports GPU hang, not OOM.

[Ep0chalypse]

Turns out Almalinux 10 doesn’t have kernels being built for 6.6 lts, and kernel 16.6 wasn’t released this week, there’s mother RC so maybe next week 16.6 will go to stable and I can finally test to see if the included fixes solve my issue. I can try to reduce the GTT in my bios, but I’m still a noob with things like this and it may be a dumb question, but…GTT is the bios settings for allocating more or less ram to dedicated VRAM right?

[MCredbear]

Turns out Almalinux 10 doesn’t have kernels being built for 6.6 lts, and kernel 16.6 wasn’t released this week, there’s mother RC so maybe next week 16.6 will go to stable and I can finally test to see if the included fixes solve my issue. I can try to reduce the GTT in my bios, but I’m still a noob with things like this and it may be a dumb question, but…GTT is the bios settings for allocating more or less ram to dedicated VRAM right?

Yes, GTT is part of RAM reserved for integrated GPU, but out of GTT your integrated GPU can also get dynamic allocate VRAM, which Pytorch uses.

[chboishabba]

Ah I always confuse GTT with something from GPU overclocking 😓

…

On Thu, 24 July 2025, 3:39 pm MCredbear, ***@***.***> wrote: *MCredbear* left a comment (lamikr/rocm_sdk_builder#257) <#257 (comment)> Turns out Almalinux 10 doesn’t have kernels being built for 6.6 lts, and kernel 16.6 wasn’t released this week, there’s mother RC so maybe next week 16.6 will go to stable and I can finally test to see if the included fixes solve my issue. I can try to reduce the GTT in my bios, but I’m still a noob with things like this and it may be a dumb question, but…GTT is the bios settings for allocating more or less ram to dedicated VRAM right? Yes, GTT is part of RAM reserved for integrated GPU, but out of GTT your integrated GPU can also get dynamic allocate VRAM, which Pytorch uses. — Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGM4B3V7Z4VT4LUIJZEUC3D3KBWS7AVCNFSM6AAAAACBTTICVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMJSGAZTANJYG4> . You are receiving this because you commented.Message ID: ***@***.***>

[Ep0chalypse]

I got impatient and compiled kernel 6.16 and re ran stable diffusion and it core dumped like before.
HW Exception by GPU node-1 (Agent handle: 0x55d995a22a00) reason :GPU Hang Aborted (core dumped)

I was really hoping the newer kernel fixed this for me, but no such luck.

I can maybe see if using a different version of rocm helps but was really hoping the kernel update fixed things.

[chboishabba]

I can maybe see if using a different version of rocm helps but was really hoping the kernel update fixed things.

I believe the update is in .8 as I said earlier which just dropped a day or two ago for arch. I will confirm if this is fixed, but it not expected to work under that kernel, I had tested as well :)