zfs: shutdown `zpool export -a` wedge on v1.13.0 — detach timeline from a hyperconverged DRBD/LINSTOR node (data point for #1085)

[dafnrs]

Summary

Reboot out of Talos v1.13.0 wedges permanently in the shutdown sequence on a hyperconverged ZFS + DRBD/LINSTOR node. Signature matches #1085 (zpool export -a at shutdown, no timeout, SIGKILLed mid-export past the 10s grace). Reporting because our captured timeline has a detail the existing reports lack: the export stall is not slow — it is permanently stuck; 3 of 15 DRBD-held zvols never release, with zero kernel activity for 25+ minutes, so no grace period would have been long enough while DRBD held the volumes.

Environment

• Talos v1.13.0 (Cozystack build ghcr.io/cozystack/cozystack/talos), kernel 6.18.24-talos, upgrading to v1.13.6
• Extensions: zfs 2.4.3, drbd 9.3.2 (LINSTOR/Piraeus on zvols), bare-metal OVH, 3× control-plane hyperconverged
• Node under reboot was fully drained (kubectl drain complete; only DaemonSets remained)

Sequence observed (talm/talosctl reboot, default mode)

1. Reboot sequence starts; stopAllPods, dbus, stopServicesForUpgrade proceed; kubelet, etcd, trustd, cri all stop cleanly ("Service finished successfully").
2. ext-zfs-service enters Stopping; machined escalates to SIGKILL: Sending SIGKILL to task ext-zfs-service (PID 6249) — the task survives SIGKILL indefinitely (uninterruptible D-state, consistent with being blocked in a zfs ioctl mid-export).
3. Concurrently, DRBD secondaries on the node hit local-IO errors as their backing zvols go away and detach one at a time (disk( UpToDate -> Failed ) [local-io-error] → disk( Failed -> Diskless ) [go-diskless], peers take over UUIDs cleanly).
4. Detach cadence: 6 volumes in the first 6 seconds, 2 at +1min, 2 at +2min, 1 at +4min, 1 at +19min — then nothing. 3 of 15 DRBD resources never detach. After the last detach: zero kernel log lines of any kind for 25+ minutes. MachineStatus stays stage: rebooting forever; no hung-task warnings (detector presumably off).
5. Only exit: talosctl reboot -m force (real reset; on our AMI boards a dirty reset then risks a POST hang, which is how one attempt ended in a manual power-cycle).

This was our second encounter: an earlier upgrade attempt on this cluster hit the same wedge on an undrained node (that time first visible as "gave up ... stopping cri", ending in the same terminal state).

Confirmations / cross-refs

• Matches ZFS extension v1.13.0+ wedges shutdown on clusters with non-trivial pools (regression from v1.12.x) #1085 (v1.13.0 zpool export -a, no timeout); understood fixed by fix: bound zpool export at shutdown with a configurable timeout #1104 (zfs ext ≥ v1.13.4). Our wedge fires on the reboot out of v1.13.0, which still runs the old code — i.e. every v1.13.0 → newer upgrade on such nodes hits it.
• Workaround validated for us by Cozystack: after draining, drbdadm down all in the node's LINSTOR satellite pod, then reboot immediately — with DRBD not holding the zvols the export completes within grace.
• Related open issues we'll be watching post-upgrade: ZFS extension: shutdown still wedges indefinitely on v1.13.5 (post-#1104) when a zvol is mounted and in use — zombie zpool child blocks PID-ns teardown #1174 (shutdown wedge with mounted in-use zvol), ZFS extension: boot-time zpool import -fal can hang on hyperconverged DRBD nodes (no timeout, unlike shutdown export in #1104) #1167 (boot-time zpool import -fal without timeout on hyperconverged DRBD nodes).

Full service-state dumps, kernel log excerpts with timestamps, and the per-volume detach timeline are captured and available on request.