feat: add new CDI hook to set CUDA memory limits - #2075
Merged
Merged
Conversation
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
2 times, most recently
from
September 10, 2026 23:56
e74a378 to
33c5844
Compare
Coverage Report for CI Build 34788198846Coverage decreased (-0.2%) to 43.787%Details
Uncovered Changes
Coverage Regressions4 previously-covered lines in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
2 times, most recently
from
September 11, 2026 03:40
18696a4 to
fcca9a9
Compare
cdesiniotis
reviewed
Sep 11, 2026
cdesiniotis
reviewed
Sep 11, 2026
henry118
reviewed
Sep 11, 2026
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
4 times, most recently
from
September 12, 2026 00:09
465ea21 to
d07ebd9
Compare
cdesiniotis
reviewed
Sep 12, 2026
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
4 times, most recently
from
September 12, 2026 00:42
8c66f16 to
567826e
Compare
cdesiniotis
approved these changes
Sep 12, 2026
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
8 times, most recently
from
September 13, 2026 22:55
6e8b4d1 to
544986e
Compare
tariq1890
commented
Sep 14, 2026
tariq1890
commented
Sep 14, 2026
Comment on lines
+124
to
+126
| if !hasRequest && !hasLimit { | ||
| return nil | ||
| } |
Contributor
Author
There was a problem hiding this comment.
@cdesiniotis this logic has been updated since the last approve. In the latest iteration, setting both the request and limits are no longer mandatory; either one env var being set is enough for the the CDI hook to call the NVML Memory Limits API.
The new behaviour is as follows
i) If both env vars are set, it continues to work as before
ii) If only the request env var is set, the limit is set to the max-possible uint64 values
iii) If only the limit env var is set, then the request is set to the same value as the limit.
Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com> get cgroup path from procfs instead Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>
tariq1890
force-pushed
the
cuda-memory-limits-impl
branch
from
September 14, 2026 16:45
544986e to
2d76167
Compare
cdesiniotis
approved these changes
Sep 14, 2026
henry118
approved these changes
Sep 14, 2026
Contributor
|
/cherry-pick release-1.20 |
|
🤖 Backport PR created for |
Contributor
Author
|
/cherry-pick release-1.20 |
|
🤖 Backport PR created for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces a new subcommand of the nvidia CDI hook called apply-cuda-memory-limits. The purpose of this CDI hook is to apply the desired memory limits to a GPU device.
It reads the following envars from a container spec to fetch the input parameters for the CDI hook
Under the covers, the CDI hooks invokes the new NVML
SetMemoryLimits_v1method (introduced in CUDA 13.4) to apply the desired CUDA memory limit values to the cgroup of the GPU container in question