ZFS extension: shutdown still wedges indefinitely on v1.13.5 (post-#1104) when a zvol is mounted and in use — zombie zpool child blocks PID-ns teardown

[lukepostulka]

Summary

#1085 was closed as fixed by #1104 (bounded zpool export + clean exit, shipped in
1.13.4). On Talos v1.13.5 with zfs extension 2.4.3-v1.13.5 we still hit a
shutdown wedge in the same family, with a different and worse signature: it does not
self-clear (>40 min observed; #1085's wedge cleared in ~20 min via kernel timeout).
Recovery required an out-of-band power cut.

Distinguishing condition: the pool had a mounted, in-use zvol at shutdown
(ext4-on-zvol mounted via an ExtensionServiceConfig volume, hosting Longhorn replicas
with active workloads — jbd2 for the zvol was live).

Observed sequence (graceful talosctl reboot)

1. Talos stops k8s workloads, then stops ext-zfs-service: SIGTERM at T, SIGKILL at
   T+10 s (the fixed 10 s grace) — i.e. the service did not exit 0 within grace,
   despite fix: bound zpool export at shutdown with a configurable timeout #1104's 8 s export bound.
2. zfs-service (PID-ns init of its container) is left a zombie with two
   remarkable properties, stable for 40+ minutes:
   • an unreaped child: zpool (PPid = the zfs-service, State: Z, TracerPid: 0) —
     PID range dates it to the shutdown window, i.e. plausibly the fix: bound zpool export at shutdown with a configurable timeout #1104 export itself;
   • its last live thread stuck in kernel space:

     [<0>] do_wait+0x81/0x120
     [<0>] kernel_wait4+0x144/0x1a0
     [<0>] zap_pid_ns_processes+0x11d/0x1c0
     [<0>] do_exit+0x89f/0xa60
     [<0>] do_group_exit+0x77/0x90
     [<0>] get_signal+0x702/0x790
     

     — the dying namespace-init waits to collect its members and never succeeds,
     even though the only remaining member is the zombie child above. (Why wait4
     cannot collect that zombie is observed, not explained — kernel 6.18.36-talos.)
3. The containerd shim (containerd-shim-runc-v2, system namespace) stays alive,
   sleeping — it never sees the task exit.
4. machined's stopServicesForUpgrade "gave up on" stopping cri after ~30 s, but has
   no give-up for extension services — the whole shutdown sequence blocks behind
   ext-zfs-service (Stopping) indefinitely. Every subsequent talosctl reboot --mode powercycle / shutdown re-enters the same sequence and blocks the same way.
   Nothing in userspace recovers the node; only a power cut does.

Environment

• Talos v1.13.5 (metal), kernel 6.18.36-talos, zfs extension 2.4.3-v1.13.5
• Single pool; one zvol with ext4, mounted via ExtensionServiceConfig at
  /var/mnt/longhorn, actively used by Longhorn v1.x replicas at shutdown
• Second cluster node without a mounted zvol reboots clean every time (matches
  ZFS extension v1.13.0+ wedges shutdown on clusters with non-trivial pools (regression from v1.12.x) #1085's "non-trivial pools" scoping)

Suggested angles

• fix: bound zpool export at shutdown with a configurable timeout #1104 bounds the export but the timeout path appears not to survive an export child
  that dies awkwardly (or a D-state interval) under an in-use mounted zvol: the service
  neither exits 0 in grace nor leaves a reapable namespace.
• Separately (possibly a talos-repo issue): machined applies a give-up to cri but not
  to extension services; a stop-timeout there would convert this class from
  "power-cut required" to "delayed reboot".

Happy to provide full talosctl reads (proc status/stack captures, service events,
dmesg) — all retained.