Skip to content

feat: add mergerfs extension - #1264

Open
micke wants to merge 1 commit into
siderolabs:mainfrom
micke:feat/mergerfs
Open

micke wants to merge 1 commit into
siderolabs:mainfrom
micke:feat/mergerfs

Conversation

@micke

@micke micke commented Sep 25, 2026

Copy link
Copy Markdown

mergerfs is a FUSE union filesystem that pools several filesystems into one mount. This extension builds it statically from the release tarball and runs it as a host-mode service, so the mount lives in the host mount namespace where the kubelet and other extensions such as nfs-server can see it. FUSE is built into the Talos kernel (CONFIG_FUSE_FS=y) and mergerfs links its own libfuse, so nothing else is needed.

A small supervisor, starts one mergerfs -f -o config=<file> per /etc/mergerfs/*.ini supplied with EtcFileConfig. A mergerfs that exits leaves its FUSE mount behind answering ENOTCONN, so the supervisor lazily detaches leftovers before mounting, and it exits whenever a mergerfs exits or the configuration changes so that Talos restarts the service with the current set of pools.

A pre-shutdown hook unmounts every pool before Talos stops the services. Talos stops extension services concurrently, and zfs-service fails hard if zfs unmount -au finds a dataset busy; without the hook a mergerfs still holding a branch file open at that moment would race it. The hook leaves a drain request in /run/mergerfs that the supervisor honours until the request expires, so a shutdown that Talos aborts recovers on its own.

Requires Talos v1.14 for host-mode services and pre-shutdown hooks.

A design choice I'd like your opinion on: the supervisor treats the pools as one unit.
When one mergerfs process dies, or any file in /etc/mergerfs changes, every pool is unmounted and the service restarts, so the other pools are gone for the 5 seconds until Talos restarts it, and anything with files open on them sees errors.
That is the same model as the nfs-server supervisor and it keeps the code small.
An earlier version handled this per pool instead: a change to one file only remounted that pool, and a crashed mergerfs was restarted on its own with backoff while the others kept serving.
It worked, but it needed a reconcile loop, per-pool restart state and a rule for two files claiming the same mountpoint.
If you'd rather have that behaviour, I can bring it back.

Tested:

  • amd64 and arm64 builds through bldr, including the extensions validator and a check that the binary is fully static.
  • Go unit tests, plus a FUSE test in a privileged container with a restart loop standing in for Talos: stacked stale mounts detached at start, hardlinks across branches, kill -9 of a mergerfs process (service restarts with the leftover detached), config add/change/remove, duplicate mountpoints refused, the pre-shutdown drain, its expiry, and the hook with no supervisor running.
  • On a Talos v1.14.1 node: two ZFS-backed pools mounted at boot, visible to pods via hostPath with HostToContainer propagation, exported with nfs-server (NFSv4) and mounted from another host; a reboot shows the hook draining the pools before zfs-service exports them, and zpool history records a clean export every time.

One thing worth knowing for reviewers: with branches-mount-timeout set, mergerfs tries to run mount for each branch that is not mounted yet and aborts when there is no mount binary, which is the case on Talos. The service restart covers it (the pool comes up once the branches are mounted) and the README documents the log line users will see.

mergerfs is a FUSE union filesystem that pools several filesystems into
one mount. The extension builds it statically from the release tarball
and runs it as a host-mode service so the mount lives in the host mount
namespace, where the kubelet and other extensions such as nfs-server can
see it.

A small supervisor starts one `mergerfs -f -o config=<file>` per
/etc/mergerfs/*.ini, supplied with EtcFileConfig. A mergerfs that exits
leaves its FUSE mount behind answering ENOTCONN; the supervisor lazily
detaches it before mounting again, and exits whenever a mergerfs exits
or the configuration changes so that Talos restarts the service. A
pre-shutdown hook unmounts every pool before Talos stops the services
owning the branches, such as zfs-service, so they never race mergerfs
at shutdown.

Requires Talos v1.14 for host-mode services and pre-shutdown hooks.

Signed-off-by: Micke Lisinge <hi@micke.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

1 participant