Using the synology-csi I am unable to mount the persistent volume

[cblument]

Bug Report

Using the synology-csi I am unable to mount the persistent volume. I can confirm that the LUN and target is created on the synology device.

Description

Installed extensions:

talosctl -n 192.168.3.30 get extensions
NODE           NAMESPACE   TYPE              ID            VERSION   NAME          VERSION
192.168.3.30   runtime     ExtensionStatus   0             1         btrfs         v1.13.6
192.168.3.30   runtime     ExtensionStatus   1             1         iscsi-tools   v0.2.0
192.168.3.30   runtime     ExtensionStatus   2             1         schematic     446d7bb4b23caecb9134bcab115e52d55af742da8e04760817f6a31997dc32d9
192.168.3.30   runtime     ExtensionStatus   modules.dep   1         modules.dep   6.18.38-talos

Listing iscsi through talosctl:

talosctl -n 192.168.3.30 list -r /usr/local/ | grep iscsi
192.168.3.30   etc/containers/iscsid.yaml
192.168.3.30   lib/containers/iscsid
192.168.3.30   lib/containers/iscsid/dev
192.168.3.30   lib/containers/iscsid/etc
192.168.3.30   lib/containers/iscsid/etc/hosts
192.168.3.30   lib/containers/iscsid/etc/iscsi
192.168.3.30   lib/containers/iscsid/etc/resolv.conf
192.168.3.30   lib/containers/iscsid/lib
192.168.3.30   lib/containers/iscsid/proc
192.168.3.30   lib/containers/iscsid/run
192.168.3.30   lib/containers/iscsid/sys
192.168.3.30   lib/containers/iscsid/usr
192.168.3.30   lib/containers/iscsid/usr/lib
192.168.3.30   lib/containers/iscsid/usr/local
192.168.3.30   lib/containers/iscsid/usr/local/lib
192.168.3.30   lib/containers/iscsid/usr/local/sbin
192.168.3.30   lib/containers/iscsid/var
192.168.3.30   lib/containers/iscsid/var/lib
192.168.3.30   lib/containers/iscsid/var/lib/iscsi
192.168.3.30   lib/libopeniscsiusr.so
192.168.3.30   lib/libopeniscsiusr.so.0
192.168.3.30   lib/libopeniscsiusr.so.0.2.0
192.168.3.30   lib/pkgconfig/libopeniscsiusr.pc
192.168.3.30   sbin/brcm_iscsiuio
192.168.3.30   sbin/iscsi-gen-initiatorname
192.168.3.30   sbin/iscsi-iname
192.168.3.30   sbin/iscsi_discovery
192.168.3.30   sbin/iscsi_fw_login
192.168.3.30   sbin/iscsi_offload
192.168.3.30   sbin/iscsiadm
192.168.3.30   sbin/iscsid
192.168.3.30   sbin/iscsistart
192.168.3.30   sbin/iscsiuio
192.168.3.30   share/spdx/open-iscsi.spdx.json

While trying to figure it out I found a reference to needing kubelet fat image and applied that to the config:

talosctl -n 192.168.3.30 get kubeletspec -o yaml| grep image
    image: ghcr.io/siderolabs/kubelet:v1.36.2-fat
        imageMaximumGCAge: 0s
        imageMinimumGCAge: 0s

When I don't set the iscsiadm path in the helm chart

  Normal   Scheduled               10m                 default-scheduler        Successfully assigned default/read-9xzrj to talos32.hamota.com
  Normal   SuccessfulAttachVolume  10m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161"
  Warning  FailedMount             32s (x13 over 10m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasbox.pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161], err: env: can't execute 'iscsiadm': No such file or directory
 (exit status 127)

When I set the iscsiadm path in the helm chart:

   Normal   Scheduled               10s                default-scheduler        Successfully assigned default/read-cc7lg to talos31.ahmota.com
  Normal   SuccessfulAttachVolume  10s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b1bfbb55-5a73-40eb-b68d-a93a5ef0d30c"
  Warning  FailedMount             0s                 kubelet                  MountVolume.MountDevice failed for volume "pvc-b1bfbb55-5a73-40eb-b68d-a93a5ef0d30c" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasbox.pvc-b1bfbb55-5a73-40eb-b68d-a93a5ef0d30c], err:  (executable file not found in $PATH)

Logs

Trying to use synology-csi and when trying the speedtest jobs I get the error when creating the jobs:

kubectl describe pod read-9xzrj
Name:             read-9xzrj
Namespace:        default
Priority:         0
Service Account:  default
Node:             talos32.myinfra.com/192.168.3.32
Start Time:       Thu, 09 Jul 2026 12:12:09 -0500
Labels:           app=speedtest
                  batch.kubernetes.io/controller-uid=fa1d6af6-0b28-4343-aedf-85dbd5c03178
                  batch.kubernetes.io/job-name=read
                  controller-uid=fa1d6af6-0b28-4343-aedf-85dbd5c03178
                  job=read
                  job-name=read
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    Job/read
Containers:
  read:
    Container ID:
    Image:         ubuntu:xenial
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      dd
      if=/mnt/pv/test.img
      of=/dev/null
      bs=8k
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/pv from test-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hq4fs (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  test-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  test-claim
    ReadOnly:   false
  kube-api-access-hq4fs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Warning  FailedScheduling        10m                 default-scheduler        0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
  Warning  FailedScheduling        10m (x2 over 10m)   default-scheduler        0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
  Normal   Scheduled               10m                 default-scheduler        Successfully assigned default/read-9xzrj to talos32.hamota.com
  Normal   SuccessfulAttachVolume  10m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161"
  Warning  FailedMount             32s (x13 over 10m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasbox.pvc-86bc15d9-292c-4c45-bd2a-1487dcd7e161], err: env: can't execute 'iscsiadm': No such file or directory
 (exit status 127)

Environment

• Talos version:

talosctl -n 192.168.3.30 version
Client:
        Tag:         v1.13.5
        SHA:         51b0d8ed
        Built:
        Go version:  go1.26.4
        OS/Arch:     darwin/arm64
Server:
        NODE:        192.168.3.30
        Tag:         v1.13.6
        SHA:         04318854
        Built:
        Go version:  go1.26.5
        OS/Arch:     linux/amd64
        Enabled:     RBAC

• Kubernetes version:

❯ kubectl version
Client Version: v1.36.1
Kustomize Version: v5.8.1
Server Version: v1.36.2

• Platform: metal

❯ helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
synology-csi    synology-csi    3               2026-07-09 12:30:59.645836 -0500 CDT    deployed        synology-csi-0.9.3-SNAPSHOT     v1.3.0

[frezbo]

Is the helm chart using chroot otherwise it might not work

[cblument]

Is the helm chart using chroot otherwise it might not work

I have chroot disabled in configuration

> helm -n synology-csi get values synology-csi| grep -i chroot
    supportChrootDir: false

[frezbo]

It should probably use chroot, also look at talosctl support bundle for kubelet logs and the sysnology-csi pod logs

[cblument]

I deleted a previous comment because of a red herring.

It should probably use chroot, also look at talosctl support bundle for kubelet logs and the sysnology-csi pod logs

I had it configured to use chroot initially (before using the fat image) but ran into the error:

err: chroot: can't execute '/usr/bin/env': No such file or directory

I found this closed issue siderolabs/talos#11469 which explains that chroot should not be used.

❯ helm -n synology-csi get values synology-csi| grep -i chroot
    supportChrootDir: true

 Normal   Scheduled               14s                default-scheduler        Successfully assigned default/read-kp4f6 to talos32.myinfra.com
  Normal   SuccessfulAttachVolume  14s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-73c3be5e-428f-4f96-9569-280556c73131"
  Warning  FailedMount             3s (x5 over 11s)   kubelet                  MountVolume.MountDevice failed for volume "pvc-73c3be5e-428f-4f96-9569-280556c73131" : rpc error: code = Int
ernal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasbox.pvc-73c3be5e-428f-4f96-9569-280556c73131], err: chroot: can't execute '/usr
/bin/env': No such file or directory
 (exit status 127)

The logs for the synology-csi-node:

csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeStageVolume
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/0d1cc2cc488855a507e76c08
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/initiator.go:103] Failed to run iscsiadm session: exit status 127
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/initiator.go:206] Failed to discover portal [192.168.1.126:3260]: chroot: can't execute '/usr/bin/env': No such file or directory
csi-plugin  (exit status 127)
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/utils.go:126] GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.
csi-plugin  (exit status 127)

[frezbo]

I deleted a previous comment because of a red herring.

It should probably use chroot, also look at talosctl support bundle for kubelet logs and the sysnology-csi pod logs

I had it configured to use chroot initially (before using the fat image) but ran into the error:

err: chroot: can't execute '/usr/bin/env': No such file or directory

I found this closed issue siderolabs/talos#11469 which explains that chroot should not be used.

❯ helm -n synology-csi get values synology-csi| grep -i chroot
    supportChrootDir: true

 Normal   Scheduled               14s                default-scheduler        Successfully assigned default/read-kp4f6 to talos32.myinfra.com
  Normal   SuccessfulAttachVolume  14s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-73c3be5e-428f-4f96-9569-280556c73131"
  Warning  FailedMount             3s (x5 over 11s)   kubelet                  MountVolume.MountDevice failed for volume "pvc-73c3be5e-428f-4f96-9569-280556c73131" : rpc error: code = Int
ernal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:nasbox.pvc-73c3be5e-428f-4f96-9569-280556c73131], err: chroot: can't execute '/usr
/bin/env': No such file or directory
 (exit status 127)

The logs for the synology-csi-node:

csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeGetCapabilities
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:128] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}}]}
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:122] GRPC call: /csi.v1.Node/NodeStageVolume
csi-plugin 2026-07-10T14:39:45Z [INFO] [driver/utils.go:123] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/0d1cc2cc488855a507e76c08
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/initiator.go:103] Failed to run iscsiadm session: exit status 127
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/initiator.go:206] Failed to discover portal [192.168.1.126:3260]: chroot: can't execute '/usr/bin/env': No such file or directory
csi-plugin  (exit status 127)
csi-plugin 2026-07-10T14:39:45Z [ERROR] [driver/utils.go:126] GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.
csi-plugin  (exit status 127)

We've a test with iscsi CSI, which does only use chroot it's becuase the script tries to use /usr/bin/env , it should just use chroot