You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#1104 (fixing #1085) bounded the shutdown zpool export -a with a configurable
timeout (default 8s) and, on timeout, deliberately "leaves pools for boot-time
import". However the boot-time zpool import -fal in zfs-service
(storage/zfs/zfs-service/main.go) has no timeout / no bound.
On hyperconverged Piraeus/DRBD nodes (a ZFS pool backing many DRBD zvols), zpool import -fal on boot can block for a very long time / indefinitely on
the DRBD-backed zvols. Because ext-zfs-service never finishes its start
phase, services that depend on it never come up: the node is reachable on the
network but the Talos API (:50000) is down and the node stays NotReady.
So the failure mode from #1085 is not eliminated for this topology — it is moved from shutdown to the next boot.
Node NotReady; services depending on ext-zfs-service never start.
Note: on our affected node the hang is currently inferred from these symptoms
(network up, apid refused, services not started). We are working to attach
the exact boot-time console log / service ext-zfs-service state and will
add it here as soon as we regain console access to the node.
Suggested fix
Bound zpool import -fal with a configurable timeout, symmetric to the
export fix (e.g. ZFS_IMPORT_TIMEOUT, sane default), so a hung/slow import
cannot wedge the whole boot; log and continue so machined can proceed and
the import can be retried/completed asynchronously.
Optionally, import the pool without forcing all zvols online at boot and
bring zvols up lazily.
Summary
#1104 (fixing #1085) bounded the shutdown
zpool export -awith a configurabletimeout (default 8s) and, on timeout, deliberately "leaves pools for boot-time
import". However the boot-time
zpool import -falinzfs-service(
storage/zfs/zfs-service/main.go) has no timeout / no bound.On hyperconverged Piraeus/DRBD nodes (a ZFS pool backing many DRBD zvols),
zpool import -falon boot can block for a very long time / indefinitely onthe DRBD-backed zvols. Because
ext-zfs-servicenever finishes its startphase, services that depend on it never come up: the node is reachable on the
network but the Talos API (
:50000) is down and the node staysNotReady.So the failure mode from #1085 is not eliminated for this topology — it is
moved from shutdown to the next boot.
Context
zpool import -fal) is not.of a ZFS pool with per-volume zvols).
Reproduction
(Piraeus/LINSTOR), replicas shared with peers.
zpool import -falblocks on the DRBD-backed zvols.Diagnostic signature
talosctl -n <node> ... :50000->connection refused(apid never starts).NotReady; services depending onext-zfs-servicenever start.Suggested fix
zpool import -falwith a configurable timeout, symmetric to theexport fix (e.g.
ZFS_IMPORT_TIMEOUT, sane default), so a hung/slow importcannot wedge the whole boot; log and continue so machined can proceed and
the import can be retried/completed asynchronously.
bring zvols up lazily.
Versions
2.4.3-v1.13.69.3.2-v1.13.6