Problem
rdma.ome.io/auto-inject (profile oci-roce, pkg/webhook/admission/pod/rdma_injector.go) sets privileged: true on the target container. Privileged containers ignore NVIDIA_VISIBLE_DEVICES, so the pod sees the node's raw GPUs instead of its device-plugin-allocated ones.
On any node shared with other GPU workloads this collides immediately. Live repro (moirai-dev, TP2 vLLM decoder allocated 2 GPUs on a node where GPU 1 was held by a CI job):
ValueError: Free memory on device cuda:1 (7.16/79.18 GiB) on startup is less than desired GPU memory utilization (0.9, 71.26 GiB).
And when it doesn't crash, it's worse: the workload silently squats on GPUs the device plugin has allocated to another pod.
Suggested fix
The profile should not need privileged: RDMA device access can come from an RDMA device plugin resource (e.g. rdma/hca_shared_devices_a) or explicit device mounts + IPC_LOCK/CAP_NET_RAW caps. Workaround we're using: skip auto-inject, request the shared-HCA resource, declare dshm + IPC_LOCK in the runtime.
Problem
rdma.ome.io/auto-inject(profileoci-roce, pkg/webhook/admission/pod/rdma_injector.go) setsprivileged: trueon the target container. Privileged containers ignoreNVIDIA_VISIBLE_DEVICES, so the pod sees the node's raw GPUs instead of its device-plugin-allocated ones.On any node shared with other GPU workloads this collides immediately. Live repro (moirai-dev, TP2 vLLM decoder allocated 2 GPUs on a node where GPU 1 was held by a CI job):
And when it doesn't crash, it's worse: the workload silently squats on GPUs the device plugin has allocated to another pod.
Suggested fix
The profile should not need
privileged: RDMA device access can come from an RDMA device plugin resource (e.g.rdma/hca_shared_devices_a) or explicit device mounts +IPC_LOCK/CAP_NET_RAWcaps. Workaround we're using: skip auto-inject, request the shared-HCA resource, declaredshm+IPC_LOCKin the runtime.