feat: add mergerfs extension

.kres.yaml

@@ -45,6 +45,7 @@ spec:
     - mdadm
     - mei
     - mellanox-mstflint
+    - mergerfs
     - metal-agent
     - multipath-tools
     - nebula

MAINTAINERS.md

@@ -42,6 +42,7 @@ If the field is marked as `Needs Maintainer`, it means that the package is curre
 | lldpd                                     | Nokia (Niklas Wik) | [salkin](https://github.com/salkin)                                                      |
 | mdadm                                     | Serge Logvinov     | [sergelogvinov](https://github.com/sergelogvinov)                                        |
 | mei                                       | Nick Meyer         | [e3b0c442](https://github.com/e3b0c442)                                                  |
+| mergerfs                                  | Micke Lisinge      | [micke](https://github.com/micke)                                                        |
 | metal-agent                               | Sidero Labs        | NA                                                                                       |
 | multipath-tools                           | INS                | [Untersander](https://github.com/Untersander), [SimLi1333](https://github.com/SimLi1333) |
 | nebula                                    | s e                | [iamwacko](https://github.com/iamwacko)                                                  |

README.md

@@ -165,6 +165,7 @@ tiers based on support level:
 | [fuse3](storage/fuse3) | :green_square: core | [ghcr.io/siderolabs/fuse3](https://github.com/siderolabs/extensions/pkgs/container/fuse3) | `3.18.3` |  This system extension provides fuse3 functionality. |
 | [iscsi-tools](storage/iscsi-tools) | :green_square: core | [ghcr.io/siderolabs/iscsi-tools](https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools) | `v0.2.0` |  This system extension provides iscsi-tools. |
 | [mdadm](storage/mdadm) | :warning: deprecated | [ghcr.io/siderolabs/mdadm](https://github.com/siderolabs/extensions/pkgs/container/mdadm) | `v4.6` |  This system extension is no-op, for backward compatibility. |
+| [mergerfs](storage/mergerfs) | :white_large_square: contrib | [ghcr.io/siderolabs/mergerfs](https://github.com/siderolabs/extensions/pkgs/container/mergerfs) | `2.42.0` |  This system extension provides mergerfs, a FUSE union filesystem, as a host service that mounts one pool per /etc/mergerfs/*.ini file. |
 | [multipath-tools](storage/multipath-tools) | :yellow_square: extra | [ghcr.io/siderolabs/multipath-tools](https://github.com/siderolabs/extensions/pkgs/container/multipath-tools) | `v0.1.0` |  This system extension provides multipath-tools. |
 | [nfs-server](storage/nfs-server) | :white_large_square: contrib | [ghcr.io/siderolabs/nfs-server](https://github.com/siderolabs/extensions/pkgs/container/nfs-server) | `v0.1.0` |  This system extension provides the nfs-utils server-side daemons, turning the node into an NFS server serving NFSv3, v4.0, v4.1 and v4.2.  exportfs syncs the kernel export table from /etc/exports. rpc.mountd services NFSv3 mount requests and the kernel's export upcalls. nfsdcld tracks NFSv4 clients so they can reclaim state after a server restart. rpc.nfsd starts the kernel NFS server threads.  Requires the nfsd kernel module from the `nfsd` extension, and the `nfs-utils` extension for rpcbind and rpc.statd. |
 | [nfs-utils](storage/nfs-utils) | :white_large_square: contrib | [ghcr.io/siderolabs/nfs-utils](https://github.com/siderolabs/extensions/pkgs/container/nfs-utils) | `v0.1.1` |  This system extension provides rpcbind and rpc.statd for NFSv3 file locking support.  rpcbind is a server that converts RPC program numbers into universal addresses. rpc.statd is the NSM (Network Status Monitor) service daemon that notifies NFS peers of restarts.  These services are required for NFSv3 mounts with file locking support. |

storage/mergerfs/README.md

@@ -0,0 +1,104 @@
+# mergerfs
+
+This extension provides [mergerfs](https://github.com/trapexit/mergerfs), a FUSE union
+filesystem that pools several filesystems ("branches") into one mount. The `ext-mergerfs`
+host service runs one mergerfs process per config file in `/etc/mergerfs`.
+
+## Installation
+
+See [Installing Extensions](https://github.com/siderolabs/extensions#installing-extensions).
+
+Requires Talos v1.14.0 or newer for host-mode extension services. FUSE is built into the
+Talos kernel, so no kernel module or other extension is needed.
+
+## Usage
+
+Describe each pool in an `EtcFileConfig` document. mergerfs reads all of its options from
+the file as `key=value` lines:
+
+```yaml
+apiVersion: v1alpha1
+kind: EtcFileConfig
+name: mergerfs/media.ini
+mode: 0o644
+contents: |
+  branches=/var/mnt/disk1:/var/mnt/disk2
+  mountpoint=/var/mnt/media
+  fsname=media
+  category.create=pfrd
+  cache.files=partial
+  # wait for the branches to be mounted before pooling them
+  branches-mount-timeout=300
+  branches-mount-timeout-fail=true
+```
+
+Branches and mountpoints have to live under `/var`, the writable part of the Talos
+filesystem: user volumes are mounted at `/var/mnt/<name>`, and ZFS datasets need a
+`mountpoint` under `/var`. The mountpoint directory must exist (create it once from a
+privileged debug pod) and must not be a symlink. `allow_other` is set automatically since
+mergerfs runs as root. Everything else is covered by the
+[mergerfs options](https://trapexit.github.io/mergerfs/latest/config/options/).
+
+### Exporting over NFS
+
+A pool can be exported with the `nfs-server` extension like any other directory. mergerfs
+recommends `inodecalc=path-hash` and `never-forget-nodes=true` in the pool config, leaving
+`lazy-umount-mountpoint` at its default, and an explicit `fsid=` on the export because
+FUSE filesystems share a device number. The `mp` export option keeps the export inactive
+until the pool is mounted:
+
+```text
+/var/mnt/media 10.0.0.0/24(rw,sync,no_subtree_check,mp,fsid=0b6a5c1e-3c5b-4bd8-9e0d-2e8b6f1c7a42)
+```
+
+## How It Works
+
+The service reads `/etc/mergerfs` when it starts. When a file is added, changed or removed,
+or a mergerfs process exits, it stops every pool and exits, and Talos restarts it five
+seconds later with the current configuration; the mount a crashed mergerfs leaves behind is
+detached first. A file without `mountpoint=`, or two files with the same mountpoint, keep
+the service from starting until fixed, and the log says which.
+
+### Waiting for branches
+
+`ext-mergerfs` starts as soon as `/var` is mounted, which can be before the branches are,
+for example before the `zfs` extension has imported its pools. `branches-mount-timeout`
+makes mergerfs wait until every branch is a mount of its own, and
+`branches-mount-timeout-fail=true` makes it exit rather than mount a partial pool. Branches
+that are plain directories on the same filesystem as the mountpoint need a
+`.mergerfs.branch` marker file instead.
+
+While waiting, mergerfs tries to run `mount` for each missing branch and aborts when there
+is no such command, as on Talos. The service then restarts every five seconds, so the pool
+still comes up once the branches are mounted; until then the log shows one
+`mergerfs exited: signal: aborted` line per attempt.
+
+### Shutdown
+
+On reboot, shutdown, reset and upgrade, the extension's pre-shutdown hook unmounts every
+pool while the services that own the branches (such as `ext-zfs-service`) and the ones
+serving the pools (such as `ext-nfs-server`) are still running, so nothing races mergerfs
+for the branches. A pool that has not unmounted after 20 seconds is lazily detached. The
+hook never fails, since a failed hook aborts the whole sequence.
+
+## Testing
+
+Once the node is up, `talosctl service ext-mergerfs` is `Running` and the pools show up in
+the service log and in the mount table:
+
+```bash
+$ talosctl logs ext-mergerfs
+10.5.0.3: mergerfs-supervisor: media.ini: started mergerfs on /var/mnt/media
+$ talosctl mounts | grep /var/mnt/media
+10.5.0.3   media   42.95   0.00   42.95   0.00%   /var/mnt/media
+```
+
+A pod with a `hostPath` volume on the mountpoint and `mountPropagation: HostToContainer`
+sees the files of every branch. After `talosctl reboot` the log shows
+`draining: stopping all mounts` before the other services stop, and the pools are mounted
+again after boot.
+
+## References
+
+- [mergerfs documentation](https://trapexit.github.io/mergerfs/)
+- [mergerfs and NFS](https://trapexit.github.io/mergerfs/latest/remote_filesystems/)

storage/mergerfs/manifest.yaml.tmpl

@@ -0,0 +1,11 @@
+version: v1alpha1
+metadata:
+  name: mergerfs
+  version: "{{ .VERSION }}"
+  author: Micke Lisinge
+  description: |
+    [{{ .TIER }}] This system extension provides mergerfs, a FUSE union filesystem, as a host
+    service that mounts one pool per /etc/mergerfs/*.ini file.
+  compatibility:
+    talos:
+      version: ">= v1.14.0"

storage/mergerfs/mergerfs.yaml

@@ -0,0 +1,15 @@
+name: mergerfs
+runnerMode: host
+depends:
+  # /var (and the user volumes and ZFS datasets under /var/mnt) is mounted.
+  - service: cri
+container:
+  entrypoint: /usr/local/libexec/mergerfs-supervisor
+restart: always
+# Unmount every pool while the services that own the branches (ZFS, NFS)
+# still run; without it they race mergerfs for the branches at shutdown.
+preShutdown:
+  entrypoint: /usr/local/libexec/mergerfs-supervisor
+  args:
+    - pre-shutdown
+  timeout: 30s

storage/mergerfs/mounts/cmd/mergerfs-supervisor/main.go

@@ -0,0 +1,51 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+// Command mergerfs-supervisor runs one mergerfs per /etc/mergerfs/*.ini.
+//
+// Without arguments it supervises the mounts. `mergerfs-supervisor pre-shutdown`
+// is the Talos pre-shutdown hook: it makes the running supervisor unmount
+// everything while the services owning the branches (ZFS, NFS) still run.
+package main
+
+import (
+	"context"
+	"log"
+	"os"
+	"os/signal"
+	"path/filepath"
+	"syscall"
+
+	"github.com/siderolabs/extensions/storage/mergerfs/mounts/server"
+)
+
+func main() {
+	log.SetFlags(0)
+	log.SetPrefix("mergerfs-supervisor: ")
+
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	runtime := server.NewOSRuntime()
+	config := server.DefaultConfig()
+
+	switch {
+	case len(os.Args) == 1:
+		if err := server.Run(ctx, runtime, config); err != nil {
+			log.Print(err)
+
+			os.Exit(1)
+		}
+	case len(os.Args) == 2 && os.Args[1] == "pre-shutdown":
+		// A failed hook makes Talos abort the reboot, shutdown or upgrade;
+		// a mount that is still there is the lesser problem, so only log.
+		if err := server.PreShutdown(ctx, runtime, config); err != nil {
+			log.Printf("pre-shutdown: %v", err)
+		}
+	default:
+		log.Printf("usage: %s [pre-shutdown]", filepath.Base(os.Args[0]))
+
+		os.Exit(1)
+	}
+}

storage/mergerfs/mounts/go.mod

@@ -0,0 +1,3 @@
+module github.com/siderolabs/extensions/storage/mergerfs/mounts
+
+go 1.26.8

storage/mergerfs/mounts/mounts.go

@@ -0,0 +1,151 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+// Package mounts inspects the mount table for mergerfs mounts and detaches the
+// ones left behind by a mergerfs process that died.
+package mounts
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"strings"
+	"syscall"
+)
+
+const (
+	mountinfoPath = "/proc/self/mountinfo"
+
+	// Filesystem type mergerfs mounts are recorded as.
+	fsType = "fuse.mergerfs"
+
+	// maxStaleLayers bounds DetachStale's loop; each restart of a crashed
+	// mergerfs adds at most one layer.
+	maxStaleLayers = 64
+)
+
+// Mounter provides the filesystem operations required by DetachStale.
+type Mounter interface {
+	ReadMountinfo() ([]byte, error)
+	Unmount(target string, flags int) error
+}
+
+// OSMounter performs mount operations using the host operating system.
+type OSMounter struct{}
+
+// ReadMountinfo returns the mount table of the calling process.
+func (OSMounter) ReadMountinfo() ([]byte, error) {
+	return os.ReadFile(mountinfoPath)
+}
+
+// Unmount unmounts a filesystem.
+func (OSMounter) Unmount(target string, flags int) error {
+	return syscall.Unmount(target, flags)
+}
+
+// IsMounted reports whether a mergerfs mount is what is visible at mountpoint.
+//
+// The mount table records the path a mount was made at, so mountpoint has to
+// be that path: cleaned and without symlinks. It is never looked up on disk,
+// which would hang on a wedged FUSE mount.
+func IsMounted(mounter Mounter, mountpoint string) (bool, error) {
+	mountinfo, err := mounter.ReadMountinfo()
+	if err != nil {
+		return false, fmt.Errorf("read mount table: %w", err)
+	}
+
+	return IsMergerfsMount(mountinfo, mountpoint), nil
+}
+
+// DetachStale lazily detaches the mergerfs mounts stacked at mountpoint (see
+// IsMounted for the form it takes) and returns how many there were.
+//
+// A mergerfs process that dies without unmounting leaves its mount behind,
+// answering ENOTCONN; mounting a new instance on top of it would stack a second
+// mount. Only the mount visible at mountpoint is considered, and only while it
+// is a `fuse.mergerfs` mount: a different filesystem mounted over a stale
+// mergerfs mount is neither detached nor looked under.
+func DetachStale(mounter Mounter, mountpoint string) (int, error) {
+	detached := 0
+
+	for range maxStaleLayers {
+		mountinfo, err := mounter.ReadMountinfo()
+		if err != nil {
+			return detached, fmt.Errorf("read mount table: %w", err)
+		}
+
+		if !IsMergerfsMount(mountinfo, mountpoint) {
+			return detached, nil
+		}
+
+		if err = mounter.Unmount(mountpoint, syscall.MNT_DETACH); err != nil {
+			return detached, fmt.Errorf("detach %s: %w", mountpoint, err)
+		}
+
+		detached++
+	}
+
+	return detached, fmt.Errorf("detach %s: still mounted after %d detaches", mountpoint, detached)
+}
+
+// IsMergerfsMount reports whether the mount visible at path is a fuse.mergerfs mount.
+//
+// mountinfo lists mounts in the order they were made, so of several mounts at
+// the same path the last entry is the one on top.
+func IsMergerfsMount(mountinfo []byte, path string) bool {
+	fstype, ok := TopMount(mountinfo, path)
+
+	return ok && fstype == fsType
+}
+
+// TopMount returns the filesystem type of the mount visible at path.
+func TopMount(mountinfo []byte, path string) (fstype string, ok bool) {
+	for line := range bytes.Lines(mountinfo) {
+		// 36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - fuse.mergerfs pool rw,...
+		pre, post, found := strings.Cut(strings.TrimRight(string(line), "\n"), " - ")
+		if !found {
+			continue
+		}
+
+		fields := strings.Fields(pre)
+		postFields := strings.Fields(post)
+
+		if len(fields) < 5 || len(postFields) < 1 {
+			continue
+		}
+
+		if unescape(fields[4]) == path {
+			fstype, ok = postFields[0], true
+		}
+	}
+
+	return fstype, ok
+}
+
+// unescape decodes the octal escapes (\040 for space, \011, \012, \134) the kernel
+// uses for whitespace and backslashes in mountinfo paths.
+func unescape(field string) string {
+	if !strings.Contains(field, `\`) {
+		return field
+	}
+
+	var builder strings.Builder
+
+	for i := 0; i < len(field); i++ {
+		if field[i] == '\\' && i+3 < len(field) && isOctal(field[i+1]) && isOctal(field[i+2]) && isOctal(field[i+3]) {
+			builder.WriteByte((field[i+1]-'0')<<6 | (field[i+2]-'0')<<3 | (field[i+3] - '0'))
+			i += 3
+
+			continue
+		}
+
+		builder.WriteByte(field[i])
+	}
+
+	return builder.String()
+}
+
+func isOctal(c byte) bool {
+	return c >= '0' && c <= '7'
+}