diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index cfe5df751..73f26e11e 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -88,65 +88,79 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - GPU Operator Version + - :cspan:`1` GPU Operator Version * - v26.7.0 + - v26.7.1 * - NVIDIA GPU Driver - | `610.57.04 `_ | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ - | `535.309.01 `_ + - | `615.71.09 `_ + | `610.57.04 `_ + | `595.91.07 `_ (**R**, **D**) + | `580.173.02 `_ * - NVIDIA Driver Manager for Kubernetes - `v0.12.0 `__ + - `v0.12.1 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ + - `1.20.1 `__ * - NVIDIA Device Plugin for Kubernetes - `0.20.0 `__ + - `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - - `v${dra_version} `__ + - :cspan:`1` `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ + - `v4.6.1-4.8.4 `__ * - Node Feature Discovery - - `v0.19.0 `__ + - :cspan:`1` `v0.19.0 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - `0.20.0 `__ + - `0.20.1 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ + - `0.15.1 `__ * - DCGM - `4.6.0-1 `__ + - `4.6.1-1 `__ * - Validator for NVIDIA GPU Operator - - ${version} + - v26.7.0 + - v26.7.1 * - NVIDIA KubeVirt GPU Device Plugin - - `v1.6.0 `__ + - :cspan:`1` `v1.6.0 `__ * - NVIDIA vGPU Device Manager - `v0.5.0 `__ + - `v0.5.1 `__ * - NVIDIA GDS Driver |gds|_ - - `2.29.4 `__ + - :cspan:`1` `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ + - `v0.4.4 `__ * - NVIDIA GDRCopy Driver - - `v2.6 `__ + - :cspan:`1` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - - `v0.0.5 `__ + - :cspan:`1` `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 319fec2db..bf9192693 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,161 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.1: + +26.7.1 +====== + +New Features +------------ + +* Updated software component versions: + + - NVIDIA Driver Manager for Kubernetes v0.12.1 + - NVIDIA Container Toolkit v1.20.1 + - NVIDIA Device Plugin for Kubernetes v0.20.1 + - NVIDIA DCGM Exporter v4.6.1-4.8.4 + - NVIDIA DCGM 4.6.1-1 + - NVIDIA MIG Manager for Kubernetes v0.15.1 + - NVIDIA GPU Feature Discovery for Kubernetes v0.20.1 + - NVIDIA vGPU Device Manager v0.5.1 + - NVIDIA Confidential Computing Manager for Kubernetes v0.4.4 + +* Added support for NVIDIA Data Center GPU Driver 615.71.09. + +* Helm installations can now configure the leader-election lease renewal deadline by setting the ``operator.leaderElection.renewDeadline`` Helm value. + This matches the configuration available with OLM installations. + (`GPU Operator PR #2902 `__) + +* Containers can now enforce soft and hard per-GPU CUDA memory limits. + Set the ``NVIDIA_GPU_MEMORY_REQUEST`` and ``NVIDIA_GPU_MEMORY_LIMIT`` environment variables to specify the limits in MiB. + This feature requires an R615 or later driver. + (`Container Toolkit PR #2093 `__) + +Improvements +------------ + +* The deprecated ``spec.numNodes`` field is no longer required in a ``ComputeDomain`` resource. + When omitted, the field defaults to ``0``. + (`GPU Operator PR #2863 `__) + +* The release image list now includes images used only by the OLM bundle, helping administrators mirror or allowlist every image required for an OLM installation. + (`GPU Operator PR #2927 `__) + +* Containers using R615 or later drivers now automatically receive the GPU firmware files required at runtime when CDI is used. + (`Container Toolkit PR #2096 `__) + +Fixed Issues +------------ + +* Fixed an issue where GPU Feature Discovery could not update a ``NodeFeature`` resource when the ``OwnerReferencesPermissionEnforcement`` admission controller was enabled. + The GPU Feature Discovery role now permits deleting ``NodeFeature`` resources. + (`GPU Operator PR #2926 `__, `Issue #2914 `__) + +* Fixed an issue where GPU operands could start after the driver pod restarted but before the driver libraries became available. + Affected operands could fail, and NVIDIA MIG Manager could report ``ERROR_LIBRARY_NOT_FOUND``. + GPU operands now wait for driver validation to complete. + (`GPU Operator PR #2896 `__, `Issue #2874 `__) + +* Fixed an issue where the Operator could continue reconciliation after NVIDIA driver DaemonSet cleanup failed. + The Operator now remains in the current state and reports the cleanup error against the correct state. + (`GPU Operator PR #2793 `__) + +* Fixed an issue where an ``NVIDIADriver``-managed driver pod did not mount the host ``/lib/modules`` directory when using precompiled drivers on SUSE nodes. + (`GPU Operator PR #2791 `__, `Issue #2776 `__) + +* Fixed false ``GPUOperatorNodeDeploymentFailed`` alerts when the NVIDIA Device Plugin was intentionally disabled in the ``ClusterPolicy``. + The NVIDIA Node Status Exporter now skips Device Plugin validation when ``devicePlugin.enabled=false``. + (`GPU Operator PR #2818 `__, `Issue #2237 `__) + +* Fixed an issue where GPU Feature Discovery could panic while parsing a malformed PCI vendor-specific capability length. + Invalid capability entries are now skipped. + (`k8s-device-plugin PR #2027 `__, `Issue #1891 `__) + +* Fixed an issue where NVIDIA MIG Manager could leave GPU operands paused indefinitely when waiting for GPU client pods to stop or when restarting the validator failed. + NVIDIA MIG Manager now restores the operand labels and reports the MIG configuration state as ``failed``. + (`MIG Manager PR #484 `__, `Issue #485 `__) + +* Fixed an issue where mounting ``/etc/nvidia`` as read-only caused the ``update-application-profile`` hook to fail container creation. + The hook now ignores read-only file system errors when application profiles cannot be written. + (`Container Toolkit PR #2028 `__, `Issue #2026 `__) + +* Fixed an issue where enabling CDI with ``nvidia-ctk runtime configure`` could overwrite existing Docker feature flags, such as ``containerd-snapshotter``. + Existing Docker feature flags are now preserved. + (`Container Toolkit PR #2094 `__) + +* Fixed NVIDIA graphics and display injection on hosts where X.Org and GLVND configuration files are installed outside the canonical container paths. + NVIDIA Container Toolkit now mounts the files in canonical locations and injects the EGL X11 libraries and ``nvidia-xconfig``. + (`Container Toolkit PR #1980 `__, `Issue #1477 `__, `Issue #563 `__) + +* Fixed an issue where generated CDI specifications could omit NVIDIA control device nodes and cause Vulkan or Wayland applications to fail. + The ``nvidia-cdi-refresh`` service now creates missing control device nodes before generating CDI specifications. + (`Container Toolkit PR #1979 `__, `Issue #1982 `__) + +* Fixed an issue where CDI hooks could fail with Podman ``--userns nomap`` or ``--userns auto`` because they could not read the container bundle. + With ``crun`` 1.27 or later, the hooks now use the container root path supplied by the runtime. + (`Container Toolkit PR #1971 `__, `Issue #648 `__) + +* Improved handling of transient Kubernetes API errors during driver operations that could leave a node cordoned. + NVIDIA Driver Manager now retries cordon and uncordon requests with bounded exponential backoff. + (`Driver Manager PR #210 `__, `Issue #190 `__) + +Known Issues +------------ + +* On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting or resetting the GPUs can prevent subsequent MIG reconfiguration. + Deleting a MIG GPU instance fails with ``NVML ERROR_UNKNOWN``, and the driver logs Xid 119. + + As a workaround, reboot the node after upgrading the driver and before changing the MIG configuration. + Alternatively, perform the following steps for each affected node to reset the GPUs without rebooting the node: + + #. Before upgrading the driver, disable MIG and wait for the ``nvidia.com/mig.config.state`` node label to report ``success``: + + .. code-block:: console + + $ kubectl label node nvidia.com/mig.config=all-disabled --overwrite + + If disabling MIG fails, reboot the node instead of continuing with these steps. + + #. Pause the GPU operands that hold GPU handles: + + .. code-block:: console + + $ kubectl label node --overwrite \ + nvidia.com/gpu.deploy.device-plugin=false \ + nvidia.com/gpu.deploy.gpu-feature-discovery=false \ + nvidia.com/gpu.deploy.dcgm=false \ + nvidia.com/gpu.deploy.dcgm-exporter=false \ + nvidia.com/gpu.deploy.mig-manager=false + + #. Change the ``driver.version`` value to ``615.71.09`` and wait for the driver pod on the node to report ``Running``. + + #. Reset the GPUs and verify that they are detected: + + .. code-block:: console + + $ kubectl -n gpu-operator exec -c nvidia-driver-ctr -- nvidia-smi -r + $ kubectl -n gpu-operator exec -c nvidia-driver-ctr -- nvidia-smi -L + + #. Resume the GPU operands: + + .. code-block:: console + + $ kubectl label node --overwrite \ + nvidia.com/gpu.deploy.device-plugin=true \ + nvidia.com/gpu.deploy.gpu-feature-discovery=true \ + nvidia.com/gpu.deploy.dcgm=true \ + nvidia.com/gpu.deploy.dcgm-exporter=true \ + nvidia.com/gpu.deploy.mig-manager=true + + #. Restore the required MIG profile: + + .. code-block:: console + + $ kubectl label node nvidia.com/mig.config= --overwrite + +---- + .. _v26.7.0: 26.7.0 diff --git a/repo.toml b/repo.toml index f1b1986f1..1532ebce1 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.0", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.1", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst",