Skip to content

ZFS extension: boot-time zpool import -fal can hang on hyperconverged DRBD nodes (no timeout, unlike shutdown export in #1104) #1167

Description

@IvanHunters

Summary

#1104 (fixing #1085) bounded the shutdown zpool export -a with a configurable
timeout (default 8s) and, on timeout, deliberately "leaves pools for boot-time
import". However the boot-time zpool import -fal in zfs-service
(storage/zfs/zfs-service/main.go) has no timeout / no bound.

On hyperconverged Piraeus/DRBD nodes (a ZFS pool backing many DRBD zvols),
zpool import -fal on boot can block for a very long time / indefinitely on
the DRBD-backed zvols. Because ext-zfs-service never finishes its start
phase, services that depend on it never come up: the node is reachable on the
network but the Talos API (:50000) is down and the node stays NotReady.

So the failure mode from #1085 is not eliminated for this topology — it is
moved from shutdown to the next boot.

Context

Reproduction

  1. Talos node with the ZFS extension and a pool containing DRBD zvols
    (Piraeus/LINSTOR), replicas shared with peers.
  2. Reboot / upgrade the node (or hit the shutdown export timeout from fix: bound zpool export at shutdown with a configurable timeout #1104).
  3. On the following boot, zpool import -fal blocks on the DRBD-backed zvols.

Diagnostic signature

  • Node answers ICMP (network is up).
  • talosctl -n <node> ... :50000 -> connection refused (apid never starts).
  • Node NotReady; services depending on ext-zfs-service never start.

Note: on our affected node the hang is currently inferred from these symptoms
(network up, apid refused, services not started). We are working to attach
the exact boot-time console log / service ext-zfs-service state and will
add it here as soon as we regain console access to the node.

Suggested fix

  • Bound zpool import -fal with a configurable timeout, symmetric to the
    export fix (e.g. ZFS_IMPORT_TIMEOUT, sane default), so a hung/slow import
    cannot wedge the whole boot; log and continue so machined can proceed and
    the import can be retried/completed asynchronously.
  • Optionally, import the pool without forcing all zvols online at boot and
    bring zvols up lazily.

Versions

  • Talos v1.13.6
  • zfs extension 2.4.3-v1.13.6
  • drbd extension 9.3.2-v1.13.6

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions