Conversation
mergerfs is a FUSE union filesystem that pools several filesystems into one mount. The extension builds it statically from the release tarball and runs it as a host-mode service so the mount lives in the host mount namespace, where the kubelet and other extensions such as nfs-server can see it. A small supervisor starts one `mergerfs -f -o config=<file>` per /etc/mergerfs/*.ini, supplied with EtcFileConfig. A mergerfs that exits leaves its FUSE mount behind answering ENOTCONN; the supervisor lazily detaches it before mounting again, and exits whenever a mergerfs exits or the configuration changes so that Talos restarts the service. A pre-shutdown hook unmounts every pool before Talos stops the services owning the branches, such as zfs-service, so they never race mergerfs at shutdown. Requires Talos v1.14 for host-mode services and pre-shutdown hooks. Signed-off-by: Micke Lisinge <hi@micke.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mergerfs is a FUSE union filesystem that pools several filesystems into one mount. This extension builds it statically from the release tarball and runs it as a host-mode service, so the mount lives in the host mount namespace where the kubelet and other extensions such as
nfs-servercan see it. FUSE is built into the Talos kernel (CONFIG_FUSE_FS=y) and mergerfs links its own libfuse, so nothing else is needed.A small supervisor, starts one
mergerfs -f -o config=<file>per/etc/mergerfs/*.inisupplied withEtcFileConfig. A mergerfs that exits leaves its FUSE mount behind answeringENOTCONN, so the supervisor lazily detaches leftovers before mounting, and it exits whenever a mergerfs exits or the configuration changes so that Talos restarts the service with the current set of pools.A pre-shutdown hook unmounts every pool before Talos stops the services. Talos stops extension services concurrently, and
zfs-servicefails hard ifzfs unmount -aufinds a dataset busy; without the hook a mergerfs still holding a branch file open at that moment would race it. The hook leaves a drain request in/run/mergerfsthat the supervisor honours until the request expires, so a shutdown that Talos aborts recovers on its own.Requires Talos v1.14 for host-mode services and pre-shutdown hooks.
A design choice I'd like your opinion on: the supervisor treats the pools as one unit.
When one mergerfs process dies, or any file in
/etc/mergerfschanges, every pool is unmounted and the service restarts, so the other pools are gone for the 5 seconds until Talos restarts it, and anything with files open on them sees errors.That is the same model as the
nfs-serversupervisor and it keeps the code small.An earlier version handled this per pool instead: a change to one file only remounted that pool, and a crashed mergerfs was restarted on its own with backoff while the others kept serving.
It worked, but it needed a reconcile loop, per-pool restart state and a rule for two files claiming the same mountpoint.
If you'd rather have that behaviour, I can bring it back.
Tested:
kill -9of a mergerfs process (service restarts with the leftover detached), config add/change/remove, duplicate mountpoints refused, the pre-shutdown drain, its expiry, and the hook with no supervisor running.hostPathwithHostToContainerpropagation, exported withnfs-server(NFSv4) and mounted from another host; a reboot shows the hook draining the pools beforezfs-serviceexports them, andzpool historyrecords a clean export every time.One thing worth knowing for reviewers: with
branches-mount-timeoutset, mergerfs tries to runmountfor each branch that is not mounted yet and aborts when there is nomountbinary, which is the case on Talos. The service restart covers it (the pool comes up once the branches are mounted) and the README documents the log line users will see.