Bottlerocket 1.48.0 EKS 1.34 image will not mount EBS volumes over nvme15

[johnjeffers]

Image I'm using:
EKS 1.34 Bottlerocket AMI 1.48.0 (arm64)

What I expected to happen:
All volumes attached to EC2 are properly mounted and available for kubernetes persistent volumes

What actually happened:

• Volumes with an index greater than 15 are not mounted
• Kubernetes pods that require the associated PV will not start
• This issue wasn't present in EKS 1.33 AMIs, and only started after upgrading to EKS 1.34 AMIs.
• Upgrading to Bottlerocket 1.49.0 did not resolve the issue

The EC2 instance shows the EBS volumes attached in the EC2 console, but SSHing into the instance and running lsblk shows that volumes with indexes greater than 15 are not properly mounted.

The following errors occur when trying to mount /dev/xvdao and subsequent devices.

Oct 23 06:32:49 ip-10-128-14-122.ec2.internal kubelet[2201]: E1023 06:32:49.753731 2201 csi_attacher.go:401] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = NotFound desc = Failed to find device path /dev/xvdap. no device path for device "/dev/xvdap" volume "vol-08078f8696e14b62c" found

Kernel logs showed NVMe device timeout issues for devices after nvme15 (nvme16, nvme17, etc.)

[ 1000.337730] nvme nvme16: I/O tag 25 (1019) QID 1 timeout, completion polled
[ 1030.397337] nvme nvme16: I/O tag 26 (101a) QID 1 timeout, completion polled
[ 1032.307305] nvme nvme17: I/O tag 0 (1000) QID 0 timeout, completion polled

How to reproduce the problem:
In an EKS 1.34 cluster, attempt to run more than 15 pods with persistent volumes on a single node

I have opened a case with AWS support regarding this issue but have seen no traction on it, so raising it here for additional awareness. The agent assigned to the case was able to reproduce the issue.

I have worked around the problem by running smaller nodes, so that we don't hit the 15-volume limit because fewer pods are able to schedule on the smaller nodes. However, this is costing us a lot of money as our observability vendor charges us by the node, and we have had to double our node count.

[gthao313]

@johnjeffers Thanks for opening this ticket.
I successfully reproduced the issue following your steps. We've confirmed this occurs on EKS 1.34 Bottlerocket AMI specifically due to kernel version 6.12. Earlier Kubernetes versions (pre-1.33) are unaffected as they use kernel 6.1. Our team is currently investigating the kernel-related issue and will provide an update soon. Thank you for your patience.

[johnjeffers]

@gthao313 Thank you! Do you have any idea of when this might be fixed? It's costing us several thousand dollars per month to have to run smaller nodes to avoid this problem.

[gthao313]

@johnjeffers. The timeline is still being determined. We'll provide updates when we have more concrete figures. Thanks for your patient.

This issue wasn't present in EKS 1.33 AMIs, and only started after upgrading to EKS 1.34 AMIs.

Could you please share which EKS 1.33 AMIs you were using? Our investigation has shown that this issue only occurs with ARM-based EKS images for versions 1.33 and 1.34.