[Bug]: Batch renderer ERROR 4 in nvvmAddNVVMContainerToProgram

[alt250]

Bug Description

Error with batch renderer, example output:

$ uv run examples/rigid/single_franka_batch_render.py
[...]
CUDA linking Failed!
ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library

Error at /project/src/mw/cuda_exec.cpp:1197 in auto madrona::buildBVHKernels(const CompileConfig &, int32_t, std::pair<int, int>, CudaBatchRenderConfig::RenderMode)::(anonymous class)::operator()(nvJitLinkResult) const
nvJitLink error: Internal error

Steps to Reproduce

This can be reproduced on several internal machines and with different programs using the gs-madrona batch renderer.

Here is how to reproduce on a Vast.ai instance and Genesis single_franka_batch_render example for convenience:

• Create an instance
  • Select template NVIDIA CUDA
  • Edit template to change tag to cuda-12.8.1-auto
  • Select an instance with Max CUDA: 13.0 and click Rent
• ssh connect to the instance and run the following commands:

git clone https://github.com/Genesis-Embodied-AI/genesis-world.git
cd genesis-world
uv add torch
uv run examples/rigid/single_franka_batch_render.py

Expected Behavior

No error

Screenshots/Videos

No response

Relevant log output

(main) root@C.37205212:/workspace/genesis-world$ uv run examples/rigid/single_franka_batch_render.py                    
[Genesis] [08:05:09] [INFO] ╭─────────────────────────────────────────────────────────────────────────────────────╮
[Genesis] [08:05:09] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉│                                                                                                                           
[Genesis] [08:05:09] [INFO] ╰─────────────────────────────────────────────────────────────────────────────────────╯
[Genesis] [08:05:10] [INFO] Running on [NVIDIA GeForce RTX 2070 SUPER] with backend gs.cuda. Device memory: 7.60 GB.
[Genesis] [08:05:10] [INFO] 🚀 Genesis initialized. 🔖 version: 0.4.7, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO                                                              
[Genesis] [08:05:13] [INFO] Scene <6df7fe9> created.    
[Genesis] [08:05:15] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 0, uid: <ce178ec>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.                                                                                        
[Genesis] [08:05:20] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 1, uid: <3a51d9f>, morph: <gs.morphs.MJCF(file='/workspace/genesis-world/genesis/assets/xml/franka_emika_panda/panda.xml')>, material: <gs.materials.Rigid>.
[Genesis] [08:05:21] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint1`                                                                                                                                              
[Genesis] [08:05:21] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint2`                                                                                                                                              
[Genesis] [08:05:23] [INFO] Building scene <6df7fe9>...
[Genesis] [08:05:26] [WARNING] Neutral robot position (qpos0) exceeds joint limits.
[Genesis] [08:05:27] [WARNING] Constraint solver time constant should be greater than 2*substep_dt. timeconst is changed from `0.005` to `0.02`). Decrease simulation timestep or increase timeconst to avoid altering the original value.
[Genesis] [08:05:32] [WARNING] Filtered out geometry pairs causing self-collision for the neutral configuration (qpos0): (6, 11), (7, 11), (14, 21), (15, 20), (16, 23), (17, 22). Consider tuning Morph option 'decompose_robot_error_thresho
ld' or specify dedicated collision meshes. This behavior can be disabled by setting Morph option 'enable_neutral_collision=True'.
[Genesis] [08:05:33] [INFO] Compiling simulation kernels... 
[Genesis] [08:07:00] [INFO] Building visualizer...
[Genesis] [08:07:05] [INFO] Raytracer and Rasterizer requires binding to the camera with a specific environment index. Defaulting to 'rendered_envs_idx[0]'. Please specify 'env_idx' if necessary.
CUDA linking Failed!
ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library


Error at /project/src/mw/cuda_exec.cpp:1197 in auto madrona::buildBVHKernels(const CompileConfig &, int32_t, std::pair<int, int>, CudaBatchRenderConfig::RenderMode)::(anonymous class)::operator()(nvJitLinkResult) const
nvJitLink error: Internal error

Environment

(main) root@C.37205212:/workspace/genesis-world$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.4 LTS"
(main) root@C.37205212:/workspace/genesis-world$ uname -a
Linux 1f98a53d6b8a 6.8.0-110-generic #110~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 27 12:43:08 UTC  x86_64 x86_64 x86_64 GNU/Linux
(main) root@C.37205212:/workspace/genesis-world$ nvidia-smi 
Thu May 21 08:30:12 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 2070 ...    On  |   00000000:65:00.0 Off |                  N/A |
|  0%   57C    P8             25W /  215W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
(main) root@C.37205212:/workspace/genesis-world$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
(main) root@C.37205212:/workspace/genesis-world$ 

Release version or Commit ID

ommit 723a20eca31433db37a440bbd7df2d181cb4758d (HEAD -> main, origin/main, origin/HEAD)
Author: Alexis DUBURCQ <alexis.duburcq@gmail.com>
Date:   Thu May 21 00:12:03 2026 +0200

    [BUG FIX] Fix collision non-convex vs any for watertight geometries. (#2812)

Additional Context

$ uv tree --package torch
Resolved 240 packages in 1ms
torch v2.12.0
├── cuda-bindings v13.2.0
│   └── cuda-pathfinder v1.5.4
├── cuda-toolkit[cudart, cufft, cufile, cupti, curand, cusolver, cusparse, nvjitlink, nvrtc, nvtx] v13.0.2
│   ├── nvidia-cuda-runtime v13.0.96 (extra: cudart)
│   ├── nvidia-cufft v12.0.0.61 (extra: cufft)
│   │   └── nvidia-nvjitlink v13.0.88
│   ├── nvidia-cufile v1.15.1.6 (extra: cufile)
│   ├── nvidia-cuda-cupti v13.0.85 (extra: cupti)
│   ├── nvidia-curand v10.4.0.35 (extra: curand)
│   ├── nvidia-cusolver v12.0.4.66 (extra: cusolver)
│   │   ├── nvidia-cublas v13.1.1.3
│   │   │   └── nvidia-cuda-nvrtc v13.0.88
│   │   ├── nvidia-cusparse v12.6.3.3
│   │   │   └── nvidia-nvjitlink v13.0.88
│   │   └── nvidia-nvjitlink v13.0.88
│   ├── nvidia-cusparse v12.6.3.3 (extra: cusparse) (*)
│   ├── nvidia-nvjitlink v13.0.88 (extra: nvjitlink)
│   ├── nvidia-cuda-nvrtc v13.0.88 (extra: nvrtc)
│   └── nvidia-nvtx v13.0.85 (extra: nvtx)
├── filelock v3.29.0
├── fsspec v2026.4.0
├── jinja2 v3.1.6
│   └── markupsafe v3.0.3
├── networkx v3.6.1
├── nvidia-cublas v13.1.1.3 (*)
├── nvidia-cudnn-cu13 v9.20.0.48
│   └── nvidia-cublas v13.1.1.3 (*)
├── nvidia-cusparselt-cu13 v0.8.1
├── nvidia-nccl-cu13 v2.29.7
├── nvidia-nvshmem-cu13 v3.4.5
├── setuptools v81.0.0
├── sympy v1.14.0
│   └── mpmath v1.3.0
├── triton v3.7.0
└── typing-extensions v4.15.0
(*) Package tree already displayed

nv_libs.sh

import ctypes
from importlib.metadata import distribution, PackageNotFoundError

for package_name, lib_name in (
    ("nvidia_cuda_nvrtc_cu12", "libnvrtc.so.1"), 
    ("nvidia_cuda_nvrtc", "libnvrtc.so.1"), 
    ("nvidia_nvjitlink_cu12", "libnvJitLink.so.12"),
    ("nvidia_nvjitlink", "libnvJitLink.so.13"),
):
    try:
        dist = distribution(package_name)
        for file in dist.files:
            if file.name.startswith(lib_name):
                print(ctypes.CDLL(dist.locate_file(file), ctypes.RTLD_LOCAL))
                break
    except PackageNotFoundError as e:
        print(e)
        pass

(main) root@C.37205212:/workspace/genesis-world$ uv run nv_libs.py 
No package metadata was found for nvidia_cuda_nvrtc_cu12
<CDLL '/workspace/genesis-world/.venv/lib/python3.11/site-packages/nvidia/cu13/lib/libnvrtc.so.13', handle 593f2dbcb580 at 0x72a66dce4c10>
No package metadata was found for nvidia_nvjitlink_cu12
<CDLL '/workspace/genesis-world/.venv/lib/python3.11/site-packages/nvidia/cu13/lib/libnvJitLink.so.13', handle 593f2ddb47b0 at 0x72a66dc61690>

[alt250]

Note there is no error using a Vast.ai instance with Max CUDA 12.8.
This is a workaround, but such pre-CUDA 13 machines tend to become rare

Vast.ai template NVIDIA CUDA tag=cuda-12.8.1-auto

git clone https://github.com/Genesis-Embodied-AI/genesis-world.git
cd genesis-world

append to pyproject.toml:

[[tool.uv.index]]
name = "pytorch-cu128"
url = "https://download.pytorch.org/whl/cu128"
explicit = true

[tool.uv.sources]
torch = [
  { index = "pytorch-cu128", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]
torchvision = [
  { index = "pytorch-cu128", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]

uv add torch
uv run examples/rigid/single_franka_batch_render.py

examples/rigid/single_franka_batch_render.py output:

[Genesis] [09:00:44] [INFO] ╭─────────────────────────────────────────────────────────────────────────────────────╮
[Genesis] [09:00:44] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉│
[Genesis] [09:00:44] [INFO] ╰─────────────────────────────────────────────────────────────────────────────────────╯
[Genesis] [09:00:45] [INFO] Running on [NVIDIA GeForce RTX 3060] with backend gs.cuda. Device memory: 11.64 GB.
[Genesis] [09:00:45] [INFO] 🚀 Genesis initialized. 🔖 version: 0.4.7, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO
[Genesis] [09:00:50] [INFO] Scene <b669f07> created.
[Genesis] [09:00:52] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 0, uid: <0184d77>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.
[Genesis] [09:00:58] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 1, uid: <37a041a>, morph: <gs.morphs.MJCF(file='/workspace/genesis-world/genesis/assets/xml/franka_emika_panda/panda.xml')>, material: <gs.materials.Rigid>.
[Genesis] [09:00:59] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint1`
[Genesis] [09:00:59] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint2`
[Genesis] [09:01:02] [INFO] Building scene <b669f07>...
[Genesis] [09:01:05] [WARNING] Neutral robot position (qpos0) exceeds joint limits.
[Genesis] [09:01:06] [WARNING] Constraint solver time constant should be greater than 2*substep_dt. timeconst is changed from `0.005` to `0.02`). Decrease simulation timestep or increase timeconst to avoid altering the original value.
[Genesis] [09:01:13] [WARNING] Filtered out geometry pairs causing self-collision for the neutral configuration (qpos0): (6, 11), (7, 11), (14, 21), (15, 20), (16, 23), (17, 22). Consider tuning Morph option 'decompose_robot_error_threshold' or specify dedicated collision meshes. This behavior can be disabled by setting Morph option 'enable_neutral_collision=True'.
[Genesis] [09:01:15] [INFO] Compiling simulation kernels...
[Genesis] [09:03:21] [INFO] Building visualizer...
[Genesis] [09:03:24] [INFO] Raytracer and Rasterizer requires binding to the camera with a specific environment index. Defaulting to 'rendered_envs_idx[0]'. Please specify 'env_idx' if necessary.

[Genesis] [09:06:14] [INFO] Exported 6 frame(s) (3 rgb, 3 depth) from camera 1 at step 0 to data/test
[Genesis] [09:06:14] [INFO] Running at 8.00 FPS (2.67 FPS per env, 3 envs).
[Genesis] [09:06:14] [INFO] Exported 6 frame(s) (3 rgb, 3 depth) from camera 1 at step 1 to data/test
[Genesis] [09:06:14] [INFO] 💤 Exiting Genesis and caching compiled kernels...

(main) root@C.37209564:/workspace$ nvidia-smi                                                                                                                                                                                                 
Thu May 21 09:04:23 2026                                                                                                                                                                                                                      
+-----------------------------------------------------------------------------------------+                                                                                                                                                   
| NVIDIA-SMI 570.211.01             Driver Version: 570.211.01     CUDA Version: 12.8     |                                                                                                                                                   
|-----------------------------------------+------------------------+----------------------+                                                                                                                                                   
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |                                                                                                                                                   
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |                                                                                                                                                   
|                                         |                        |               MIG M. |                                                                                                                                                   
|=========================================+========================+======================|                                                                                                                                                   
|   0  NVIDIA GeForce RTX 3060        On  |   00000000:25:00.0 Off |                  N/A |
| 30%   37C    P8             12W /  170W |    1251MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                          
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
(main) root@C.37209564:/workspace$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
(main) root@C.37209564:/workspace$ cd genesis-world/
(main) root@C.37209564:/workspace/genesis-world$ uv tree --package torch
Resolved 241 packages in 2ms
torch v2.11.0+cu128
├── cuda-bindings v12.9.6
│   └── cuda-pathfinder v1.5.4
├── cuda-toolkit[cublas, cudart, cufft, cufile, cupti, curand, cusolver, cusparse, nvjitlink, nvrtc, nvtx] v12.8.1
│   ├── nvidia-cublas-cu12 v12.8.4.1 (extra: cublas)
│   ├── nvidia-cuda-runtime-cu12 v12.8.90 (extra: cudart)
│   ├── nvidia-cufft-cu12 v11.3.3.83 (extra: cufft)
│   │   └── nvidia-nvjitlink-cu12 v12.8.93
│   ├── nvidia-cufile-cu12 v1.13.1.3 (extra: cufile)
│   ├── nvidia-cuda-cupti-cu12 v12.8.90 (extra: cupti)
│   ├── nvidia-curand-cu12 v10.3.9.90 (extra: curand)
│   ├── nvidia-cusolver-cu12 v11.7.3.90 (extra: cusolver)
│   │   ├── nvidia-cublas-cu12 v12.8.4.1
│   │   ├── nvidia-cusparse-cu12 v12.5.8.93
│   │   │   └── nvidia-nvjitlink-cu12 v12.8.93
│   │   └── nvidia-nvjitlink-cu12 v12.8.93
│   ├── nvidia-cusparse-cu12 v12.5.8.93 (extra: cusparse) (*)
│   ├── nvidia-nvjitlink-cu12 v12.8.93 (extra: nvjitlink)
│   ├── nvidia-cuda-nvrtc-cu12 v12.8.93 (extra: nvrtc)
│   └── nvidia-nvtx-cu12 v12.8.90 (extra: nvtx)
├── filelock v3.29.0
├── fsspec v2026.4.0
├── jinja2 v3.1.6
│   └── markupsafe v3.0.3
├── networkx v3.6.1
├── nvidia-cudnn-cu12 v9.19.0.56
│   └── nvidia-cublas-cu12 v12.8.4.1
├── nvidia-cusparselt-cu12 v0.7.1
├── nvidia-nccl-cu12 v2.28.9
├── nvidia-nvshmem-cu12 v3.4.5
├── setuptools v81.0.0
├── sympy v1.14.0
│   └── mpmath v1.3.0
├── triton v3.6.0
└── typing-extensions v4.15.0
(*) Package tree already displayed