cudaGetDeviceCount fails with error 801 on Orin Nano 8GB (J3011), custom wrynose build — CUDA 13.2/TensorRT 10.16, kernel loads fine, no dmesg activity

[Hi5808]

"## cudaGetDeviceCount fails with error 801 on Orin Nano 8GB (J3011), custom wrynose build \u2014 CUDA 13.2/TensorRT 10.16, kernel loads fine, no dmesg activity\n\nHardware: Seeed reComputer J401 carrier + Jetson Orin Nano 8GB (SKU 0003)\nMACHINE: p3509-a02-p3767-0000, TEGRA_BOARDSKU = \"0003\"\nBranch: wrynose\nKernel: linux-noble-nvidia-tegra 6.8.12-l4t-r39.2.0-1021.21 (built via meta-tegra's own recipe, PREFERRED_PROVIDER_virtual/kernel = \"linux-noble-nvidia-tegra\")\nCUDA: 13.2 (cuda-toolkit/cuda-toolkit-dev)\nTensorRT: 10.16.2.10-1 (tensorrt-core/tensorrt-core-dev)\n\n### Symptom\n\nAny CUDA program \u2014 including a minimal standalone cudaGetDeviceCount() test, not just TensorRT-Edge-LLM which is what surfaced this \u2014 fails immediately:\n\n\ncudaGetDeviceCount: err=801 (operation not supported), count=0\n\n\n### What's confirmed working (ruling out the obvious suspects)\n\n- nvgpu kernel module loads cleanly at boot (ga10b_init_hal etc. in dmesg, no errors)\n- /usr/lib/libcuda.so.1.1 is the real 91MB driver library (not the 74KB stub), loads from the correct path (confirmed via LD_DEBUG=libs)\n- GPU firmware present and non-empty (/lib/firmware/nvidia/ga10b/*)\n- /dev/nvhost-gpu, /dev/nvmap, /dev/host1x-fence all present with correct permissions (root)\n- tegrastats reports the GPU normally (temps ~55\u00b0C, GR3D_FREQ tracking, normal idle power draw) \u2014 hardware/driver-level GPU state looks completely healthy\n- Persists across a full reboot (ruled out transient/session state)\n- No dmesg output at all when the failing call happens \u2014 the failure occurs entirely in userspace, before any ioctl reaches the kernel driver\n\n### Why I suspect a kernel/userland pairing issue, not something exotic\n\nThis MACHINE profile (p3509-a02-p3767-0000) has already surfaced two other Orin-NX-vs-Orin-Nano config mismatches earlier in the same build effort:\n\n1. TEGRA_BOARDSKU defaults to 0000 (Orin NX 16GB) rather than the actual module SKU \u2014 required an explicit override.\n2. NVPMODEL defaults (via orin-nx.inc, which this MACHINE requires) reference nvpmodel_p3767_0000_super (Orin NX: 8 CPU cores, has a PVA) \u2014 our actual hardware only has 6 cores and no PVA, causing nvpmodel.service to fail outright until overridden to nvpmodel_p3767_0003_super.\n\nGiven that pattern, I suspect there's a third gap somewhere in the CUDA/GPU userland-vs-kernel pairing for this exact MACHINE+SKU combination, but I don't have visibility into NVIDIA's internal version pinning to pin it down further myself. I found discussion #1004 which has the same top-level symptom (cudaGetDeviceCount failing on a custom kernel build) and was resolved by using a properly-matched kernel+userland branch pairing rather than mixing versions \u2014 wanted to flag this as a possible instance of the same class of issue, on much newer hardware/software (Orin Nano, JetPack 7.2/CUDA 13.2) than that one.\n\n### What I haven't been able to check\n\nI don't have a clean local way to verify whether our built kernel's nvgpu ABI/ioctl version actually matches what the prebuilt libcuda.so (fetched via l4t_deb_pkgfeed from NVIDIA's official binary feed) expects \u2014 no version marker I could find exposes this directly for comparison. If there's a known-good way to check this (or a specific patch/version pin this MACHINE should be carrying for CUDA+Orin-Nano that it isn't), I'd appreciate a pointer.\n\nHappy to provide bitbake -e output, exact package versions, or anything else that would help narrow this down.\n"

[Hi5808]

[madisongh]

Please just start over with using the correct machine configuration for your hardware, as I suggested here. You'll save yourself a bunch of hassle that way, rather than hacking and slashing your way through trying to work around that initial mistake. The machine configuration determines a bunch of settings for the boot firmware and, as you discovered, the kernel device tree, that are SKU-specific. It's really important that you start with a configuration that is correct for the particular module you are using.

[Hi5808]

"Update: this goes deeper than a static file selection issue, and I think it rules out the fixes I was trying.\n\nWorth restating for context: this is all on a custom, minimal Yocto core-image-minimal build (not stock JetPack) \u2014 the whole point of this project is a stripped-down OS with the smallest possible resource footprint, specifically to leave maximum RAM/storage headroom for running an LLM workload on the 8GB module. That's why I'm chasing this at the OS-build level rather than just reflashing stock JetPack (which would very likely sidestep the issue entirely, but isn't an option for this use case).\n\nI traced the actual flash package (NVIDIA's official l4t_initrd_flash.sh + a Seeed-provided BSP for their J401 carrier, used as a --flash-only workaround for the t23x_minimal hang described in my earlier posts) all the way down:\n\n- flash.idx correctly references kernel_tegra234-j401-p3768-0000+p3767-0003-recomputer.dtb for both A/B kernel-dtb slots\n- That file's actual compiled content, decompiled with dtc, correctly has compatible = \"nvidia,p3768-0000+p3767-0003-super\", \"nvidia,p3767-0003\", \"nvidia,tegra234\" (only the cosmetic model string is stale, still says "Orin NX")\n- The corresponding BPMP DTB (tegra234-bpmp-3767-0003-3768-super_with_odm_sigheader.dtb.encrypt) and DCE firmware blob are also correctly SKU-0003-named\n- I also tried overriding KERNEL_DEVICETREE in our own meta-tegra Yocto build to the equivalent tegra234-p3768-0000+p3767-0003-nv-super.dtb (which was already being built, just not selected) \u2014 no effect, which now makes sense given the above, since that's not even the DTB source this flashing path uses\n\nDespite all of that being correct and freshly reflashed (confirmed via a completely clean --flash-only --massflash 5 run), the running kernel still reports:\n\n\n# cat /proc/device-tree/compatible\nnvidia,p3768-0000+p3767-0000-super\nnvidia,p3767-0000\nnvidia,tegra234\n# cat /proc/device-tree/model\nNVIDIA Jetson Orin NX Engineering Reference Developer Kit Super\n\n\nGiven the static partition content is verifiably correct but gets overridden anyway, this looks like UEFI's runtime Plugin Manager (EEPROM/hardware-based dynamic device tree construction, independent of what's statically in the kernel-dtb partition) is where the actual SKU 0000-vs-0003 decision is really happening \u2014 not anything file-selection-related that I can fix from the OS-build side.\n\nGiven this is on Seeed's J401 carrier board specifically, I'm wondering if this is a carrier-board EEPROM programming issue (their EEPROM not correctly identifying the module as Orin Nano to UEFI's runtime detection) rather than anything meta-tegra-side. If anyone has pointers on how UEFI's Plugin Manager/EEPROM-based detection works for this SKU pairing, or a way to check/query what UEFI thinks the detected SKU is (independent of the static DTB content), I'd appreciate it \u2014 this is getting into territory I don't have visibility into from the OS-build side alone.\n"

[kekiefer]

It looks like you are pasting a llm response. What’s the question? What did you try? Did you try to wipe out your tmp directory and start over again with the correct machine configuration like Matt suggested?