Skip to content

fix: [apply-cuda-memory-limits] exit gracefully on oci spec load failure - #2103

Merged
tariq1890 merged 1 commit into
mainfrom
cuda-mem-limits-silent-fail
Sep 16, 2026
Merged

tariq1890 merged 1 commit into
mainfrom
cuda-mem-limits-silent-fail

Conversation

@tariq1890

Copy link
Copy Markdown
Contributor

With this fix, containers started with "usens --nomap" no longer fail as the OCI spec load operation is no longer non-blocking. This is required as the containers with "users --nomap" fail even if they don't set the cuda memory limits environment variables.

@coveralls

coveralls commented Sep 16, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 35126854839

Coverage decreased (-0.003%) to 44.087%

Details

  • Coverage decreased (-0.003%) from the base build.
  • Patch coverage: 2 uncovered changes across 1 file (0 of 2 lines covered, 0.0%).
  • 1 coverage regression across 1 file.

Uncovered Changes

File Changed Covered %
cmd/nvidia-cdi-hook/apply-cuda-memory-limits/apply-cuda-memory-limits.go 2 0 0.0%

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File Lines Losing Coverage Coverage
cmd/nvidia-cdi-hook/apply-cuda-memory-limits/apply-cuda-memory-limits.go 1 6.84%

Coverage Stats

Coverage Status
Relevant Lines: 13682
Covered Lines: 6032
Line Coverage: 44.09%
Coverage Strength: 0.44 hits per line

💛 - Coveralls

With this fix, containers started with "userns --nomap" no longer fail as
the OCI spec load operation is no longer non-blocking. This is required as
the containers with "userns --nomap" fail even if they don't set the cuda
memory limits environment variables.

Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>
@tariq1890
tariq1890 force-pushed the cuda-mem-limits-silent-fail branch from 8d00324 to 1e5fb48 Compare September 16, 2026 17:13
@tariq1890

Copy link
Copy Markdown
Contributor Author

@cdesiniotis Thanks for the approve. Just fixed some typos in the commit description.

@tariq1890
tariq1890 enabled auto-merge September 16, 2026 17:14
@tariq1890

Copy link
Copy Markdown
Contributor Author

/cherry-pick release-1.20

@henry118 henry118 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious what led this change? previously we didn't expect the permission issue...

@tariq1890

Copy link
Copy Markdown
Contributor Author

After running a podman container with --userns nomap, we observed this error. The failure is especially bad because the container does not set the NVIDIA_GPU_MEMORY_REQUEST|LIMIT env vars.

~$ podman run --rm --security-opt=label=disable --device=nvidia.com/gpu=all --userns nomap ubuntu nvidia-smi
Error: OCI runtime error: crun: error executing hook `/usr/bin/nvidia-cdi-hook` (exit code: 1)

~$ cat nvidia-cdi-hook.log
time="2026-09-16T16:51:39Z" level=error msg="failed to load OCI container spec: error opening OCI specification file: open /config.json: no such file or directory"

@henry118

henry118 commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

open /config.json: no such file or directory"

This appears to be a wrong path rather than permission issues?

@tariq1890
tariq1890 merged commit 3a6c050 into main Sep 16, 2026
21 checks passed
@tariq1890
tariq1890 deleted the cuda-mem-limits-silent-fail branch September 16, 2026 17:55
@github-actions

Copy link
Copy Markdown

🤖 Backport PR created for release-1.20: #2104 ✅

@henry118

Copy link
Copy Markdown
Member

Opened PR containers/crun#2261 which should address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants