fix(rm): fall back to an unaligned allocation when device links are unavailable

[sirzzang]

Description

Closes #2038.

On a node that exposes whole GPUs, GetPreferredAllocation takes the aligned path, and that path builds the link information for all physical devices before the available list is applied. A device that has fallen off the bus (Xid 79) therefore makes the call fail even though the health check has already marked it unhealthy and kubelet no longer offers it in available. The error becomes an admission failure for every GPU pod on the node, including single-device allocations that the remaining healthy devices could satisfy, and pods owned by a controller are recreated indefinitely: on the cluster where we hit this, one Deployment produced roughly a thousand UnexpectedAdmissionError pods every five minutes for three days.

This turns that failure into a degraded allocation instead. When link discovery fails, log a warning and select devices without considering the topology. unalignedAlloc selects only from available, always includes required, de-duplicates both, and still returns an error when the request cannot be satisfied — a negative size, more distinct required devices than the allocation size, a required device that is not available, or not enough distinct available devices.

The change is deliberately narrow and orthogonal to health reporting: no device is marked healthy or unhealthy, NVML initialisation policy is unchanged, and the aligned path is untouched whenever link discovery succeeds.

#2038 also proposes the more complete fix — enumerating only the candidate devices in go-gpuallocator so that a lost device is never touched. The two compose rather than compete: once that lands, this class of failure no longer reaches the fallback, which then only covers other reasons link discovery can fail. I am happy to follow whichever direction you prefer, including going straight to that one instead.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Unit tests passing (make test)
• Go mod artifacts in-sync (make check-modules)
• Third-party notices in-sync (make check-third-party-notices)
• Test cases are added for new code paths

Testing

Unit tests only — the hardware fault is not reproducible on demand.

internal/rm/nvml_manager_fallback_test.go injects a minimal nvml.Interface and nvml.Device test double, in the same style as fakeNvmlLib in internal/rm/health_test.go and without adding a dependency. It reports four devices but returns ERROR_GPU_IS_LOST for the handle of one of them, and only the three healthy IDs are passed as available, which is the state kubelet is in once the health check has removed the lost device.

This is also the first test to execute the aligned path: TestFullGPUNodeIgnoresAllocationPolicy currently notes that it can only check the branch condition "since alignedAlloc requires NVML".

I checked that the tests actually discriminate the change:

1. Tests added, fallback not yet implemented — the behavioural cases fail with the production error:

--- FAIL: TestAlignedAllocFallsBackWhenDeviceLinkInfoUnavailable/a_single_device_is_allocated_from_the_healthy_devices
    unable to get device link information: failed to get devices: error getting device handle for index '2': ERROR_GPU_IS_LOST

2. With the fallback — all cases pass: five subtests for the allocation path (single device, required devices included, the lost device is never allocated, the first device having fallen off the bus, allocating more devices than are available still fails) and ten table cases for unalignedAlloc.

3. Reverting only the call site while keeping the helper brings the failures back, and the boundary case ("allocating more devices than are available still fails") passes in both directions.

Local results on this branch:

make test                        # all packages pass, no failures
make lint                        # 0 issues
make check-modules               # clean
make check-third-party-notices   # clean
gofmt -l internal/rm/            # no output

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[myeolenv]

Thanks for the contribution. We will review this PR soon.