From 64364875849683ce70e93439b9226dc279260012 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 22 Sep 2026 13:04:09 -0400 Subject: [PATCH 1/7] feat: GPU Op 26.7.1 Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 36 ++++++++++++++++++++---------- gpu-operator/release-notes.rst | 27 ++++++++++++++++++++++ repo.toml | 2 +- 3 files changed, 52 insertions(+), 13 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index cfe5df751..f01d8df54 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -88,65 +88,77 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - GPU Operator Version + - :cspan:`1` GPU Operator Version * - v26.7.0 + - v26.7.1 * - NVIDIA GPU Driver - | `610.57.04 `_ | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ | `535.309.01 `_ + - | `615.71.09 `_ + | `610.57.04 `_ + | `595.91.07 `_ (**R**, **D**) + | `580.173.02 `_ + | `535.309.01 `_ * - NVIDIA Driver Manager for Kubernetes - - `v0.12.0 `__ + - :cspan:`1` `v0.12.0 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ + - `1.20.1 `__ * - NVIDIA Device Plugin for Kubernetes - - `0.20.0 `__ + - :cspan:`1` `0.20.0 `__ * - DRA Driver for NVIDIA GPUs - - `v${dra_version} `__ + - :cspan:`1` `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ + - `v4.6.1-4.8.4 `__ * - Node Feature Discovery - - `v0.19.0 `__ + - :cspan:`1` `v0.19.0 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - - `0.20.0 `__ + - :cspan:`1` `0.20.0 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ + - `0.15.1 `__ * - DCGM - `4.6.0-1 `__ + - `4.6.1-1 `__ * - Validator for NVIDIA GPU Operator - - ${version} + - v26.7.0 + - v26.7.1 * - NVIDIA KubeVirt GPU Device Plugin - - `v1.6.0 `__ + - :cspan:`1` `v1.6.0 `__ * - NVIDIA vGPU Device Manager - - `v0.5.0 `__ + - :cspan:`1` `v0.5.0 `__ * - NVIDIA GDS Driver |gds|_ - - `2.29.4 `__ + - :cspan:`1` `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ + - `v0.4.4 `__ * - NVIDIA GDRCopy Driver - - `v2.6 `__ + - :cspan:`1` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - - `v0.0.5 `__ + - :cspan:`1` `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 319fec2db..961ab3a42 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,33 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.1: + +26.7.1 +====== + +New Features +------------ + +* Updated software component versions: + + - NVIDIA Container Toolkit v1.20.1 + - NVIDIA DCGM Exporter v4.6.1-4.8.4 + - NVIDIA DCGM 4.6.1-1 + - NVIDIA MIG Manager for Kubernetes v0.15.1 + - NVIDIA Confidential Computing Manager for Kubernetes v0.4.4 + +* Added support for NVIDIA Data Center GPU Driver 615.71.09. + +Known Issues +------------ + +* On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting can prevent subsequent MIG reconfiguration. + Deleting a MIG GPU instance fails with ``NVML ERROR_UNKNOWN``, and the driver logs Xid 119. + Reboot the node after upgrading the driver and before changing the MIG configuration. + +---- + .. _v26.7.0: 26.7.0 diff --git a/repo.toml b/repo.toml index f1b1986f1..1532ebce1 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.0", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.1", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst", From fac52cb0d14ed8459887f8160323295b456a446b Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 22 Sep 2026 15:16:58 -0400 Subject: [PATCH 2/7] docs: Add R615 GPU reset workaround Signed-off-by: Mike McKiernan --- gpu-operator/release-notes.rst | 51 ++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 961ab3a42..a9ccccb86 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -54,9 +54,56 @@ New Features Known Issues ------------ -* On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting can prevent subsequent MIG reconfiguration. +* On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting or resetting the GPUs can prevent subsequent MIG reconfiguration. Deleting a MIG GPU instance fails with ``NVML ERROR_UNKNOWN``, and the driver logs Xid 119. - Reboot the node after upgrading the driver and before changing the MIG configuration. + + As a workaround, reboot the node after upgrading the driver and before changing the MIG configuration. + Alternatively, perform the following steps for each affected node to reset the GPUs without rebooting the node: + + #. Before upgrading the driver, disable MIG and wait for the ``nvidia.com/mig.config.state`` node label to report ``success``: + + .. code-block:: console + + $ kubectl label node nvidia.com/mig.config=all-disabled --overwrite + + If disabling MIG fails, reboot the node instead of continuing with these steps. + + #. Pause the GPU operands that hold GPU handles: + + .. code-block:: console + + $ kubectl label node --overwrite \ + nvidia.com/gpu.deploy.device-plugin=false \ + nvidia.com/gpu.deploy.gpu-feature-discovery=false \ + nvidia.com/gpu.deploy.dcgm=false \ + nvidia.com/gpu.deploy.dcgm-exporter=false \ + nvidia.com/gpu.deploy.mig-manager=false + + #. Change the ``driver.version`` value to ``615.71.09`` and wait for the driver pod on the node to report ``Running``. + + #. Reset the GPUs and verify that they are detected: + + .. code-block:: console + + $ kubectl -n gpu-operator exec -c nvidia-driver-ctr -- nvidia-smi -r + $ kubectl -n gpu-operator exec -c nvidia-driver-ctr -- nvidia-smi -L + + #. Resume the GPU operands: + + .. code-block:: console + + $ kubectl label node --overwrite \ + nvidia.com/gpu.deploy.device-plugin=true \ + nvidia.com/gpu.deploy.gpu-feature-discovery=true \ + nvidia.com/gpu.deploy.dcgm=true \ + nvidia.com/gpu.deploy.dcgm-exporter=true \ + nvidia.com/gpu.deploy.mig-manager=true + + #. Restore the required MIG profile: + + .. code-block:: console + + $ kubectl label node nvidia.com/mig.config= --overwrite ---- From c05b692517ade63b0bafdcd9b05571568a1bd215 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 22 Sep 2026 15:36:01 -0400 Subject: [PATCH 3/7] docs: Update vGPU Device Manager for 26.7.1 Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 3 ++- gpu-operator/release-notes.rst | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index f01d8df54..44209f141 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -144,7 +144,8 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. - :cspan:`1` `v1.6.0 `__ * - NVIDIA vGPU Device Manager - - :cspan:`1` `v0.5.0 `__ + - `v0.5.0 `__ + - `v0.5.1 `__ * - NVIDIA GDS Driver |gds|_ - :cspan:`1` `2.29.4 `__ diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index a9ccccb86..0e5c2f50c 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -47,6 +47,7 @@ New Features - NVIDIA DCGM Exporter v4.6.1-4.8.4 - NVIDIA DCGM 4.6.1-1 - NVIDIA MIG Manager for Kubernetes v0.15.1 + - NVIDIA vGPU Device Manager v0.5.1 - NVIDIA Confidential Computing Manager for Kubernetes v0.4.4 * Added support for NVIDIA Data Center GPU Driver 615.71.09. From fa21afadb7f8c65ed0f8cc6b8125f70b02941763 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 22 Sep 2026 19:37:31 -0400 Subject: [PATCH 4/7] docs: Update late 26.7.1 component versions Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 9 ++++++--- gpu-operator/release-notes.rst | 3 +++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 44209f141..cca3e3de2 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -105,14 +105,16 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. | `535.309.01 `_ * - NVIDIA Driver Manager for Kubernetes - - :cspan:`1` `v0.12.0 `__ + - `v0.12.0 `__ + - `v0.12.1 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ - `1.20.1 `__ * - NVIDIA Device Plugin for Kubernetes - - :cspan:`1` `0.20.0 `__ + - `0.20.0 `__ + - `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - :cspan:`1` `v${dra_version} `__ @@ -126,7 +128,8 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. * - | NVIDIA GPU Feature Discovery | for Kubernetes - - :cspan:`1` `0.20.0 `__ + - `0.20.0 `__ + - `0.20.1 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 0e5c2f50c..17a5e19dd 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -43,10 +43,13 @@ New Features * Updated software component versions: + - NVIDIA Driver Manager for Kubernetes v0.12.1 - NVIDIA Container Toolkit v1.20.1 + - NVIDIA Device Plugin for Kubernetes v0.20.1 - NVIDIA DCGM Exporter v4.6.1-4.8.4 - NVIDIA DCGM 4.6.1-1 - NVIDIA MIG Manager for Kubernetes v0.15.1 + - NVIDIA GPU Feature Discovery for Kubernetes v0.20.1 - NVIDIA vGPU Device Manager v0.5.1 - NVIDIA Confidential Computing Manager for Kubernetes v0.4.4 From da46a88bc19fd9953d1557263b9aff7efe051fb1 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 23 Sep 2026 11:22:06 -0400 Subject: [PATCH 5/7] docs: Remove EOL R535 driver from 26.7 matrix Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index cca3e3de2..73f26e11e 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -97,12 +97,10 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. - | `610.57.04 `_ | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ - | `535.309.01 `_ - | `615.71.09 `_ | `610.57.04 `_ | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ - | `535.309.01 `_ * - NVIDIA Driver Manager for Kubernetes - `v0.12.0 `__ From 5205318fb32a4557f339bcd0a28b84a9c5a325f7 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 23 Sep 2026 12:29:43 -0400 Subject: [PATCH 6/7] docs: Add 26.7.1 release details Signed-off-by: Mike McKiernan --- gpu-operator/release-notes.rst | 77 ++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 17a5e19dd..67206cd18 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -55,6 +55,83 @@ New Features * Added support for NVIDIA Data Center GPU Driver 615.71.09. +* Helm installations can now configure the leader-election lease renewal deadline by setting the ``operator.leaderElection.renewDeadline`` Helm value. + This matches the configuration available with OLM installations. + (`GPU Operator PR #2902 `__) + +* Containers can now enforce soft and hard per-GPU CUDA memory limits. + Set the ``NVIDIA_GPU_MEMORY_REQUEST`` and ``NVIDIA_GPU_MEMORY_LIMIT`` environment variables to specify the limits in MiB. + This feature requires an R615 or later driver. + (`Container Toolkit PR #2093 `__) + +Improvements +------------ + +* The deprecated ``spec.numNodes`` field is no longer required in a ``ComputeDomain`` resource. + When omitted, the field defaults to ``0``. + (`GPU Operator PR #2863 `__) + +* The release image list now includes images used only by the OLM bundle, helping administrators mirror or allowlist every image required for an OLM installation. + (`GPU Operator PR #2927 `__) + +* Containers using R615 or later drivers now automatically receive the GPU firmware files required at runtime when CDI is used. + (`Container Toolkit PR #2096 `__) + +Fixed Issues +------------ + +* Fixed an issue where GPU Feature Discovery could not update a ``NodeFeature`` resource because its role did not permit deleting ``NodeFeature`` resources. + The role now includes the required delete permission. + (`GPU Operator PR #2926 `__, `Issue #2914 `__) + +* Fixed an issue where GPU operands could start after the driver pod restarted but before the driver libraries became available. + Affected operands could fail, and NVIDIA MIG Manager could report ``ERROR_LIBRARY_NOT_FOUND``. + GPU operands now wait for driver validation to complete. + (`GPU Operator PR #2896 `__, `Issue #2874 `__) + +* Fixed an issue where the Operator could continue reconciliation after NVIDIA driver DaemonSet cleanup failed. + The Operator now remains in the current state and reports the cleanup error against the correct state. + (`GPU Operator PR #2793 `__) + +* Fixed an issue where an ``NVIDIADriver``-managed driver pod did not mount the host ``/lib/modules`` directory when using precompiled drivers on SUSE nodes. + (`GPU Operator PR #2791 `__, `Issue #2776 `__) + +* Fixed false ``GPUOperatorNodeDeploymentFailed`` alerts when the NVIDIA Device Plugin was intentionally disabled in the ``ClusterPolicy``. + The NVIDIA Node Status Exporter now skips Device Plugin validation when ``devicePlugin.enabled=false``. + (`GPU Operator PR #2818 `__, `Issue #2237 `__) + +* Fixed an issue where GPU Feature Discovery could panic while parsing a malformed PCI vendor-specific capability length. + Invalid capability entries are now skipped. + (`k8s-device-plugin PR #2027 `__, `Issue #1891 `__) + +* Fixed an issue where NVIDIA MIG Manager could leave GPU operands paused indefinitely when waiting for GPU client pods to stop or when restarting the validator failed. + NVIDIA MIG Manager now restores the operand labels and reports the MIG configuration state as ``failed``. + (`MIG Manager PR #484 `__, `Issue #485 `__) + +* Fixed an issue where mounting ``/etc/nvidia`` as read-only caused the ``update-application-profile`` hook to fail container creation. + The hook now ignores read-only file system errors when application profiles cannot be written. + (`Container Toolkit PR #2028 `__, `Issue #2026 `__) + +* Fixed an issue where enabling CDI with ``nvidia-ctk runtime configure`` could overwrite existing Docker feature flags, such as ``containerd-snapshotter``. + Existing Docker feature flags are now preserved. + (`Container Toolkit PR #2094 `__) + +* Fixed NVIDIA graphics and display injection on hosts where X.Org and GLVND configuration files are installed outside the canonical container paths. + NVIDIA Container Toolkit now mounts the files in canonical locations and injects the EGL X11 libraries and ``nvidia-xconfig``. + (`Container Toolkit PR #1980 `__, `Issue #1477 `__, `Issue #563 `__) + +* Fixed an issue where generated CDI specifications could omit NVIDIA control device nodes and cause Vulkan or Wayland applications to fail. + The ``nvidia-cdi-refresh`` service now creates missing control device nodes before generating CDI specifications. + (`Container Toolkit PR #1979 `__, `Issue #1982 `__) + +* Fixed an issue where CDI hooks could fail with Podman ``--userns nomap`` or ``--userns auto`` because they could not read the container bundle. + With ``crun`` 1.27 or later, the hooks now use the container root path supplied by the runtime. + (`Container Toolkit PR #1971 `__, `Issue #648 `__) + +* Improved handling of transient Kubernetes API errors during driver operations that could leave a node cordoned. + NVIDIA Driver Manager now retries cordon and uncordon requests with bounded exponential backoff. + (`Driver Manager PR #210 `__, `Issue #190 `__) + Known Issues ------------ From f5028ad18837fbf7f9de9b98608b1d49d57cbf73 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 23 Sep 2026 17:58:22 -0400 Subject: [PATCH 7/7] docs: Clarify GFD fixed issue scope Signed-off-by: Mike McKiernan --- gpu-operator/release-notes.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 67206cd18..bf9192693 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -80,8 +80,8 @@ Improvements Fixed Issues ------------ -* Fixed an issue where GPU Feature Discovery could not update a ``NodeFeature`` resource because its role did not permit deleting ``NodeFeature`` resources. - The role now includes the required delete permission. +* Fixed an issue where GPU Feature Discovery could not update a ``NodeFeature`` resource when the ``OwnerReferencesPermissionEnforcement`` admission controller was enabled. + The GPU Feature Discovery role now permits deleting ``NodeFeature`` resources. (`GPU Operator PR #2926 `__, `Issue #2914 `__) * Fixed an issue where GPU operands could start after the driver pod restarted but before the driver libraries became available.