[Bug]: nvidia-container-cli fails intermittently with "nvcgo rpc error: timed out" (hard-coded 10s RPC timeout in libnvidia-container)

[happyai-365]

Describe the bug

nvidia-container-cli intermittently fails during container startup with:

nvidia-container-cli: initialization error: nvcgo rpc error: timed out

The failure happens when the node is under CPU/IO pressure (e.g. a burst of GPU containers starting at the same time). The error is produced by the hard-coded 10 second RPC client timeout in libnvidia-container:

• src/rpc.c (setup_client):

struct timeval timeout = {10, 0};
...
clnt_control(rpc->clt, CLSET_TIMEOUT, (char *)&timeout);

During nvc_init(), nvcgo_init() (src/nvcgo.c) forks a child process that serves an ONC RPC over a unix socketpair. The server-side handler nvcgo_init_1_svc dlopen()s the Go c-shared library libnvcgo.so (with RTLD_NOW), which boots a full Go runtime. On a loaded or slow node (cold page cache, CPU throttling, IO contention, container start storms), fork + dlopen + Go runtime init can exceed 10 s, so clnt_call() returns RPC_TIMEDOUT and the whole container start fails.

The timeout is hard-coded: it is not configurable via the config file, an environment variable, or a CLI flag, and there is no retry.

Note this is a different failure flavor from the well-known driver rpc error: timed out (#237, #202), which is caused by slow NVML initialization when persistence mode is off. The nvcgo RPC program only loads the Go cgroup helper library and does not touch NVML, so enabling persistence mode does not help here.

To Reproduce

1. Use a node under heavy CPU/IO load (or artificially slow down process spawn / file reads, e.g. cold page cache + stress-ng).
2. Start a GPU container (legacy mode), e.g. docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi, or launch a burst of GPU pods on Kubernetes with the legacy runtime mode.
3. Intermittently, container creation fails with nvidia-container-cli: initialization error: nvcgo rpc error: timed out.

Expected behavior

The nvcgo RPC initialization should be resilient to slow nodes:

• the RPC timeout should be configurable (e.g. via nvidia-container-cli config / env var), and/or
• the default should be more tolerant, and/or
• the call should be retried before failing the container start.

Environment (please provide the following information):

• nvidia-container-toolkit version: v1.17.8 (also verified in the latest release v1.20.0, see below)
• NVIDIA Driver Version: 550.x
• Host OS: Ubuntu 22.04
• Kernel Version: 5.15-generic
• Container Runtime Version: containerd 1.7.x
• CPU Architecture: x86_64
• Kubernetes Distro and Version: K8s

Additional context

I verified that the issue is still present in the latest release:

• nvidia-container-toolkit v1.20.0 pins third_party/libnvidia-container at 3e428194 (libnvidia-container v1.20.0).
• In libnvidia-container v1.20.0 (and current main), src/rpc.c still contains the same hard-coded struct timeval timeout = {10, 0}; there were no timeout-related changes to src/rpc.c between v1.17.8 and v1.20.0.
• nvcgo_init() is called unconditionally from nvc_init() in builds with WITH_NVCGO=yes (the default for official packages), so every nvidia-container-cli configure invocation pays this fork+dlopen cost and is exposed to the 10 s budget.

Workarounds we are considering on our side: switch the runtime to CDI mode (which does not invoke nvidia-container-cli), reduce node-level start bursts. A configurable timeout would still be appreciated.

@elezar @klueska

[kvnloo]

The ownership boundary here looks important: the hard-coded timeout is in libnvidia-container, not the Go toolkit layer.

I would avoid adding a retry in nvidia-container-toolkit around a lower-level initialization timeout because that could repeat partially completed setup without the RPC layer knowing why.

A smaller contract at the libnvidia-container boundary would be:

• preserve the current timeout as the default;
• make the nvcgo RPC deadline configurable;
• parse/validate the value once during CLI/context initialization;
• include the effective timeout in debug diagnostics;
• keep timeout failure distinct from connection/setup failure.

A deterministic test could make the child-side RPC handler intentionally exceed a short configured deadline, then verify both timeout and success with a longer deadline.

That would solve the loaded-node case without globally weakening failure detection.

If maintainers agree, I think the implementation should originate in libnvidia-container and then be exposed by the toolkit configuration surface.