kata container is failing on ARM64

[KarstenB]

root@arm64:/host/usr/local/bin# file containerd-shim-kata-v2
containerd-shim-kata-v2: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, BuildID[sha1]=728e7a9dfef80031e9c2715c782b254771657cd3, with debug_info, not stripped

root@arm64:/host/usr/local/bin# ldd containerd-shim-kata-v2
	/lib/ld-musl-aarch64.so.1 => /lib/ld-linux-aarch64.so.1 (0x0000ffffa51f0000)
	linux-vdso.so.1 (0x0000ffffa522c000)
	libc.so => not found

ls /host/lib/ld-linux-aarch64.so.1
ls: cannot access '/host/lib/ld-linux-aarch64.so.1': No such file or directory

This is unlike the AMD64 executable:

root@amd64:/host/usr/local/bin# file containerd-shim-kata-v2
containerd-shim-kata-v2: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, BuildID[sha1]=601037634f963e7085b2200dee14e8596110d6e6, with debug_info, not stripped
root@amd64:/host/usr/local/bin# ldd containerd-shim-kata-v2
	linux-vdso.so.1 (0x00007ff72e9d4000)
	libc.so => not found

I am attempting to fix this on my own now and will report back, but it seems extremely plausible to me why Kata fails on our ARM64 nodes with:

spec.containers{nginx}: Error: failed to create containerd task: failed to create shim task: 500

[frezbo]

Talos has /lib/ld-musl-aarch64.so.1

talosctl -n 10.5.0.3 ls /lib/ld-musl-aarch64.so.1
ld-musl-aarch64.so.1

I wonder if it's compiled against some different musl version

Also seems ldd was run was a glibc container? anyways that would only resolve to interpreter from the container not talos

[KarstenB]

Yeah, so I was using kubectl debug node/arm64 --rm --image=debian --profile=sysadmin -n node-debug, which mounts the host path on /host.

But it seems I was looking at it wrong, because this seems to work just fine:

chroot /host/ /usr/local/bin/containerd-shim-kata-v2 --help
Usage of /usr/local/bin/containerd-shim-kata-v2:
  -address string
    	grpc address back to main containerd

Which unfortunately does leave me with no idea what the 500 error might actually be.