nvhost-ctrl (/dev/nvhost-ctrl) not available with OOT modules on mainline kernel (kernel 6.18, Jetson Orin NX)

[schwankner]

Summary

When running the OE4T OOT driver stack (linux-nvgpu + linux-nv-oot) on a mainline-based kernel (kernel 6.18, Talos Linux), /dev/nvhost-ctrl is not created. The JetPack 6 CUDA runtime (libnvrm_host1x.so) requires this device for interrupt-driven cudaStreamSynchronize — without it, CUDA falls back to CPU polling, causing a significant throughput regression on inference workloads (~7 tok/s vs. ~60 tok/s for qwen3:4b on Jetson Orin NX 16GB).

Background

On a standard JetPack installation, /dev/nvhost-ctrl is provided by NVIDIA's downstream kernel (nvidia-t23x-kernel, in-tree nvhost). With OOT modules + mainline kernel, the device does not exist.

The OOT nvgpu.ko driver integrates with the OE4T host1x subsystem for GPU compute, but the /dev/nvhost-ctrl character device with the NVHOST_IOCTL_CTRL_SYNCPT_WAITMEX ioctl is not part of the OOT package.

ioctls used by libnvrm_host1x.so

The JetPack 6 CUDA runtime calls these 8 ioctls on /dev/nvhost-ctrl:

• NVHOST_IOCTL_CTRL_GET_VERSION
• NVHOST_IOCTL_CTRL_SYNCPT_READ
• NVHOST_IOCTL_CTRL_SYNCPT_READ_MAX
• NVHOST_IOCTL_CTRL_SYNCPT_WAITMEX (interrupt-driven via dma_fence)
• NVHOST_IOCTL_CTRL_SYNC_FENCE_CREATE
• NVHOST_IOCTL_CTRL_GET_CHARACTERISTICS
• NVHOST_IOCTL_CTRL_POLL_FD_CREATE
• NVHOST_IOCTL_CTRL_SYNC_FILE_EXTRACT

The most critical is SYNCPT_WAITMEX: it implements interrupt-driven syncpoint waiting using dma_fence / host1x_fence. Without it, the CUDA runtime polls via CPU semaphore, causing the throughput regression.

Current workaround

We developed a small shim module (nvhost_ctrl_shim.c) that creates /dev/nvhost-ctrl and implements these 8 ioctls backed by the OE4T host1x syncpoint API. This is currently included in a Talos Linux kernel package PR: siderolabs/pkgs#1518.

The siderolabs maintainer (@dsseng) suggested getting community review from OE4T before merging — hence this issue.

Questions

1. Is there a roadmap for supporting /dev/nvhost-ctrl in the OOT module stack for mainline kernel deployments? Would it make sense to integrate this functionality into linux-nvgpu or as a separate OOT module?

2. Does the shim approach (implementing the ioctls via OE4T host1x API) look reasonable? Are there known issues with this interface on GA10B / Tegra234?

3. Is there a preferred upstream path for this functionality?

We're happy to contribute — whether that means contributing the shim code to OE4T, integrating into an existing module, or following a different approach.

Related

Also tracking: OE4T's conftest infrastructure (linux-nv-oot) does not yet detect kernel 6.18 APIs correctly when building with LLVM=1 / Clang. Currently worked around in siderolabs/pkgs#1518 via forced macro overrides in fixup.sh. Would appreciate guidance on where to upstream the kernel 6.18 conftest fixes (since linux-nv-oot has issues disabled).

[kekiefer]

If I'm not mistaken, all the device hierarchy has been changing here for quite some time, in favor of a new heirarchy, for example the gpu drivers moving from /dev/nvhost-ctrl-gpu to /dev/nvgpu/igpu0/ctrl, and no central /dev/nvhost-ctrl device is supposed to exist at all.

Can you help me locate where in the downstream kernel sources this device is being set up? If you move to the 5.15 kernel from jetpack (mirrored here https://github.com/OE4T/linux-jammy-nvidia-tegra, PREFERRED_PROVIDER_virtual/kernel = "linux-jammy-nvidia-tegra") is this situation any different?

[kekiefer]

I just realized I have a Jetpack 6 flashed orin nx available to me, and I checked the device nodes there:

nvidia@orin-nx:~$ ls /dev/nvhost*
/dev/nvhost-as-gpu        /dev/nvhost-ctxsw-gpu
/dev/nvhost-ctrl-gpu      /dev/nvhost-dbg-gpu
/dev/nvhost-ctrl-isp      /dev/nvhost-gpu
/dev/nvhost-ctrl-isp-thi  /dev/nvhost-nvsched_ctrl_fifo-gpu
/dev/nvhost-ctrl-nvcsi    /dev/nvhost-nvsched-gpu
/dev/nvhost-ctrl-nvdla0   /dev/nvhost-power-gpu
/dev/nvhost-ctrl-nvdla1   /dev/nvhost-prof-ctx-gpu
/dev/nvhost-ctrl-pva0     /dev/nvhost-prof-dev-gpu
/dev/nvhost-ctrl-vi0      /dev/nvhost-prof-gpu
/dev/nvhost-ctrl-vi0-thi  /dev/nvhost-sched-gpu
/dev/nvhost-ctrl-vi1      /dev/nvhost-tsg-gpu
/dev/nvhost-ctrl-vi1-thi

I think this issue has some measure of AI hallucinations going on with it. The syncpt interface for Jetpack 6 (provided by NVIDIA's out of tree gpu driver, the same one we're building in OE4T) has been moved to /dev/host1x-fence and has a new polling API associated with it. CUDA should know how to deal with this. And if it doesn't, it's not something we can really address here.

[madisongh]

"OOT" just stands for "out-of-tree".. the driver code is all NVIDIA's, and the repos you are pulling the code from are simply work-in-progress trees we used for integration into our OpenEmbedded/Yocto builds. The definitive sources, and current patches, are enumerated in the bitbake recipes here. NVIDIA has not yet published sources for their out-of-tree drivers that are compatible with recent kernels yet, so the extra 60+ patches we apply on top of the R36.5 sources are combinations of back-ports from their R38.4 out-of-tree drivers, and community contributions to deal with build and runtime issues we've discovered.

I have zero knowledge of Talos Linux, how it builds, or how it integrates the Jetson Linux runtime packages, kernel, etc. All I can suggest is to use Jetson Linux directly and refer to the Bring Your Own Kernel documentation that NVIDIA provides there, or to try some Yocto builds using our demo distro (our master and wrynose branches default to using the 6.18 linux-yocto kernel). You'll at least have something that works to compare against, and then you can do some side-by-side analysis of where you got your integration wrong.

[madisongh]

I think this issue has some measure of AI hallucinations going on with it.

Oh, is that why the problem statement sounded so authoritative and yet was so wrong? I should have picked up on that.

[kekiefer]

I went ahead and converted this issue to a discussion. It's fair to ask questions here and get help debugging an issue with the application (at least to the extent we can assist with that here), but at this point I don't see anything implicating the work specific to building OE4T.

[schwankner]

Thanks for the detailed responses. I should clarify the setup, and then provide concrete data rather than assertions.

Setup: Talos Linux (immutable OS, mainline kernel 6.18.24) + OE4T OOT modules (linux-nvgpu + linux-nv-oot, branch wip-r36.5-take-2) + JetPack 6.2 CUDA runtime libraries. This is not a JetPack kernel setup — it is mainline + OOT. The CUDA libraries come from JetPack 6 but the kernel is stock mainline.

On the "AI hallucinations" comment: I understand the skepticism, but the claims in the issue are based on empirical observation on real hardware. Here is the evidence:

1. /dev/nvhost-ctrl is hardcoded in libnvrm_host1x.so (JetPack 6.2):

$ grep -ao "/dev/nvhost[a-z_-]*" /usr/lib/aarch64-linux-gnu/nvidia/libnvrm_host1x.so | sort -u
/dev/nvhost-ctrl
/dev/nvhost-ctrl-gpu
/dev/nvhost-display
/dev/nvhost-gpu
...

2. The library actively opens it during inference:

Two ollama runner processes (qwen3:4b and qwen2.5:0.5b) each hold two open file descriptors to /dev/nvhost-ctrl during GPU inference. No /dev/host1x-fence is opened — the library has both paths in its binary, but in practice opens /dev/nvhost-ctrl.

3. Performance delta is real:

• Without shim (CPU polling fallback): ~7 tok/s
• With shim (interrupt-driven via host1x dma_fence): 63.7 tok/s (qwen2.5:0.5b on Jetson Orin NX 16GB)

---

On kekiefer's JetPack 6 device listing: /dev/nvhost-ctrl (without suffix) was not in the listing — but the JetPack downstream kernel likely provides the syncpoint interface via a different mechanism than the OOT modules. On a mainline+OOT setup, host1x.ko does not create /dev/nvhost-ctrl. Hence the shim.

On madisongh's suggestion to use BYOK or demo-distro for comparison: Fair point. I am aware the OE4T repos are WIP Yocto trees. The reason I am using them is that they are the only publicly available kernel 6.18 source with GA10B support.

I am not asking OE4T to support Talos Linux. The question is narrower: is /dev/nvhost-ctrl an appropriate interface to implement as an OOT module? If so, is there interest in upstreaming the shim into linux-nvgpu? If not, what is the recommended path for JetPack 6 CUDA on mainline+OOT?

[kekiefer]

No, there's not interest in this. This is the old interface. Support for it used to exist but has been removed in Jetpack releases in favor of the current implementation.

When you run an strace on a number of the tools, you will most likely find they will try older methods first before going to the new one. I suspect this is one of those cases.

And please, next time you post, don't just use your LLM. We are real humans here, working to support this project. We all use these tools as well, but this forum is for human developers to collaborate, not robots.

[kekiefer]

Actually, let me make one clarification, because I shouldn't so hasty.

"No there's not interest in this" UNLESS I'm wrong about the current roadmap and state of the art. I could be wrong, and will gladly change tune if there's some evidence that there's something being mis-built here compared to what should be there.

As far as I can tell though, this is a problem with other libraries that are not open source, and so that's why I'm implying that a workaround for those would not be appropriate here.

[madisongh]

I am not asking OE4T to support Talos Linux. The question is narrower: is /dev/nvhost-ctrl an appropriate interface to implement as an OOT module?

This is a technical question that I don't think we can give a definitive answer to, since it's about how NVIDIA's closed-source libraries work. I think @kekiefer 's analysis is correct, though - it looks like that was an older interface, now obsolete.

If so, is there interest in upstreaming the shim into linux-nvgpu?

If you want to pursue this particular question further, you should probably post a question in the NVIDIA developer forum.

If not, what is the recommended path for JetPack 6 CUDA on mainline+OOT?

CUDA support for the Jetsons isn't just about kernel and the added out-of-tree drivers. They have to be configured and built just so, and they have to be integrated with the userland libraries in the L4T BSP just so. And you have to be installing a version of the CUDA libraries that is known to work on the Jetson hardware, too. That's what I meant by it being an integration issue, and recommended that you compare against a known working combination as a starting point.

[schwankner]

Thanks to both of you for the time and feedback!
I will ask in the NVIDIA dev forum what the supposed way is.

If anyone wants to know what my technical findings are: My FD-Analysis shows that /dev/nvgpu/igpu0/ctrl and /dev/nvhost-ctrl are opened concurrently by the CUDA-process. /dev/nvgpu/igpu0/ctrl for channel-allocation and /dev/nvhost-ctrl for syncpoint-wait (POLL_FD_CREATE, SYNC_FENCE_CREATE). Without nvhost-ctrl the library silently falls back to CPU polling which gives 7 tok/s instead of 63 tok/s with nvhost-ctrl in ollama. If that is cause of the closed NVIDIA libs or what this is can hopefully answer somone at the NVIDIA Forums.

Thank you again!