AerynOS: Musings on Inode Watchers and Atomic Live Upgrades

[khaga87]

Tooling and Packaging Team member Joey Riches has written up a blog post covering some of his recent development work. This is Joey's first blog post for AerynOS and we are excited to share what he has been working on! Please go check it out and comment below if you have any questions!

https://aerynos.com/blog/2025/11/04/musings-on-inode-watchers-and-atomic-live-upgrades/

Thank you again to everyone who has sponsored the project, your on-going support helps us to keep innovating in the Linux Distribution space. For anyone who hasn't already, please do consider donating to us to allow us to continue these development paths.

[Murbert]

FWIW, I do not consider it to be overly bothersome to reboot after an update. I guess the question I have is, why is it so important to not have to?

[ermo]

Some people (yours truly included) are tired of image-based immutable distros being not only slow to apply updates, but also requiring reboots for all updates.

Any improvements in this regard would add to the feeling of "fast" and "convenient" we would like people to associate with AerynOS and its tech stack.

[ams-123]

I'm with @Murbert on this one. While impressive, there's a risk of over-engineering this one. A simple reboot (once a week in this case) for a non-server OS is not a problem that needs solving.

[ermo]

The thing is: There is nothing stopping people who want to do a reboot to do a reboot. This is about not forcing reboots unless it is absolutely necessary, backed by evidence.

We are currently running an engineering workstream in the background that looks at classifying the cases that require either a reboot, a relog, restarting a service, or simply closing the updated application and opening it again within the current session. Over time, this will make moss able to better ensure that the right option is escalated to the user via either terminal output (when moss is run via the terminal), or potentially GUI and/or notifications that recommend the correct action.

If the outcome of that workstream is that we need to accept that we will need a reboot in most cases, ok, then we have a new situation. But I just want to point out that the reason we have hotplug in Linux is that people looked at the status quo and said:

"This can't be right; why do I need to drop to the terminal and edit device files just because I want to plug in my (insert suitable device)?!"

We're doing the exploratory work for innovation. None of this impacts people who think it's ok to reboot because that's what existing solutions have conditioned them to do. They can still do that just fine.

[ams-123]

Apologies @ermo, I realise that my previous comment comes across as flippant and a little disrespectful. I'm not opposed to any of the fine work being done by the AerynOS team. And point taken on the nature of innovation - it is all about moving the needle after all.

[pwtail]

Obviously patching GNOME can be a solution, later they will develop an API for this

[ermo]

later they will develop an API for this

Do you know for sure that GNOME will develop an API for this, or are you guessing...?

[pwtail]

guessing, of course. Not the end of the world if they won't :)

[daviewales]

There seem to be a few possible approaches here:

1. Give up and accept the need to reboot.
2. Patch every app which watches files in /usr to also watch the parent directories, to detect when the world is atomically swapped out from under them.
3. Patch the Linux kernel to allow you to reuse inodes, while maintaining the ability to atomically swap directory trees.
4. Change your approach to atomic updates, such that old inodes are preserved. (The only way I can imagine this working is to temporarily switch to a snapshot of the current state, make all the changes required for the update under the original inode tree, then switch back to the original inode tree. But this is not satisfactory, as I think the double-swap breaks atomicity, and could cause watcher applications mix their inodes up between trees.)
5. Create a new filesystem with database transaction semantics, so you can tell the filesystem: BEGIN TRANSACTION, then stage your updates to the existing directory tree in-place, but have this invisible to the rest of the system until you mark the transaction as complete. Then, if the transaction fails, automatically roll back. Otherwise, commit the transaction atomically.

Ideally, what I think you want is option 3 or 5?

Edit: Would it be possible to use Btrfs snapshots for this?

1. Remount /usr as read-only for the duration of the transaction. (Or maybe it's permanently read-only?)
2. Snapshot the current /usr directory.
3. Mount the snapshot somewhere as read-write, and update the files in the snapshot, rather than the current system.
4. If the update succeeds, roll forward to the new snapshot. This should be exactly like a Btrfs rollback, but roll forwards instead. I think this should leave your inodes intact.

Update: I don't think that Btrfs snapshots will work. I've tested, and inotifywait stops detecting changes after you unmount the directory and mount the snapshot. It also doesn't detect changes if you swap the original subvolume and the snapshot with mv --exchange -T snapshot original.

Looking at the ZFS docs, I don't think that ZFS snapshots and clones would work either.

On reflection, I feel that this would be best solved at the filesystem level, with a filesystem that supports multi-file transaction semantics. Then the filesystem would be responsible for triggering the inotify updates once the transaction is committed.

I found one filesystem which supports multi-file transactions, but so far as I can tell it hasn't progressed beyond the research phase: exF2FS (PDF) (The commit messages in the git repository do not inspire confidence.) And I have no idea if it batches the inotify updates until after the transaction is committed (if at all).

[joebonrichie]

Would dbus-broker respect the IN_IGNORED event, from a high-level overview it looks like it may not

https://github.com/bus1/dbus-broker/blob/404f092d3742c57b58fba5169e64c2999f00f794/src/util/dirwatch.c#L109

For that application, instead of touching the old tree's subdir we would have created a tempfile in the old tree so the existing dbus watcher would see the new events.

[daviewales]

I could be wrong, but my reading of the quote here is that IN_IGNORED is always generated when a watch is deleted or unmounted, even if not explicitly requested.

However, inotifywait doesn't seem to print IN_IGNORED in the list of watch notifications, so might need further testing.

Some further digging on the kernel side:
The kernel runs the various fsnotify_ functions at the end of filesystem functions to trigger the notifications.
For example, search 'notify` in fs/namei.c.

The fsnotify_ functions are defined in include/linux/fsnotify.h.

[joebonrichie]

Looks like glib explicitly ignores IN_IGNORE on event callback

https://gitlab.gnome.org/GNOME/glib/-/blob/main/gio/inotify/inotify-path.c#L532

Probably passing IN_UNMOUNT would be the next best thing

[ermo]

1. Does it make sense to ask the glib developers why glib ignores IN_IGNORE on event callback?

2. If IN_UNMOUNT is the ticket, then let's just do that. This is effectively what's happening anyway, in the sense that we're doing an atomic unmount/remount of /usr in practice.

[joebonrichie]

0001-fsnotify-revoke-subtree-watchers.patch

I asked an LLM to take a look at the problem and this is what it came up with. Not upstreamable in it's current state but it does work. If we issue that ioctl to the old /usr tree after an atomic swap all the inode watchers at that subtree rearm themselves. Fixes all known issues with gnome shell/dbus-broker/mime types not getting updated/etc.

[joebonrichie]

This is all super frustrating as the kernel isn't doing anything wrong per-se. It's just that various projects have started to use inode watchers as a performance hack to avoid expensive stat calls to do path based monitoring. This was traditionally not an issue as traditional package managers extracted and removed files directly to the path. However, as we're doing something novel, we are getting bitten in the arse by this assumption.

[daviewales]

Would it be an option for AerynOS to delete the old directory tree? That would notify and delete the watchers without needing to modify the kernel.

Also, something I want to test is whether subdirectories get notifications when a parent is unmounted if that parent is a separate volume (not bind-mounted).

Otherwise, could a case be made that software like Gnome Shell is making an invalid assumption that nothing will ever happen to parent directories, and they need to put a watcher on each parent back to the root directory? (These parent watchers would not need to watch their children, just the directory itself.)

[joebonrichie]

Honestly, deleting old trees would be the simplest solution. I do recall bringing that up as an option at some point but vaguely remember that people weren't happy with it. We can trivially rebuild usr trees, although, do note; we do provide a moss.fstx=n cmdline argument which provides a failsafe boot option in case of catastrophic failure. So as long as we can successfully build a working tree within the initrd constraint, deleting old trees would be the best option.