systems with SELinux fail to activate MD arrays with MD 4.5

[mwilck]

I have spent the last two days debugging an odd issue we found with mdadm 4.5. I'd mentioned it on #245. It has been observed on openSUSE Tumbleweed, but because openSUSE uses the same SELinux setup as Fedora, it will probably affect Fedora and possibly other distros, too.

This is not actually an mdadm "bug", but I felt that it deserves being reported here anyway.

We observed boot failures on systems with MD RAID. MD arrays were not activated. On closer inspection, it turned out the the md_mod module was never loaded. Strangely, simply running mdadm commands from the shell would work. Looking further, we found that mdadm's attempts to call modprobe md_mod were blocked by SELinux. Consequently, we created an SELinux PR. Initially I wasn't convinced that relaxing SELinux rules for this was a good idea, but in the meantime I think it's the best we can get.

However, the respective part of the SELinux rules had been unchanged for quite some time. So why didn't we observe the problem with SLE 16.0? The reason is subtle, and related to the mdadm version update.

On 16.0, too, I observed that mdadm's attempts to load md_mod from udev rules were prevented by SELinux. The reason that MD arrays could be set up nontheless is an old kernel feature called CONFIG_BLOCK_LEGACY_AUTOLOAD, which I hadn't heard of before:

config BLOCK_LEGACY_AUTOLOAD
        bool "Legacy autoloading support"
        default y
        help
          Enable loading modules and creating block device instances based on
          accesses through their device special file.  This is a historic Linux
          feature and makes no sense in a udev world where device files are
          created on demand, but scripts that manually create device nodes and
          then call losetup might rely on this behavior.

I realized this because on 16.0, I observed this message in the kernel logs:

kernel: block device autoloading is deprecated and will be removed.

This message has been introduced to the kernel in 15.7 (2022) already (fbdee71 ("block: deprecate autoloading based on dev_t")), but the feature is still there, and is still enabled in recent kernels. What it does is that if someone opens a block device for which the kernel has no driver, it will call request_module() to try to load the respective driver.

And this is what mdadm did until 4.4. I am not sure if that was intentional, but if it failed to create an array via normal means, in dev_open() it'd call mknod() to create a temporary device node with major:minor 9:0, and open() it. This open() would cause the kernel to load the md_mod module in blkdev_get_no_open(), at which point everything starts to work as if mod_mod had been loaded beforehand: md_mod is loaded, the MD device is created, and the array possibly started.

mdadm 4.5 changed this code in d354d31 ("mdadm: Create array with sync del gendisk mode"). If mdadm can't access certain sysfs attribute of the md_mod module early on, it prints an error message and quits, and never gets to the point where earlier mdadm versions would try the mknod()/open() trick.

To avoid misunderstanding: I am not arguing that the old behavior should be reinstated. It is indeed "legacy" behavior. I just wanted to note this as an explanation why the behavior has changed with mdadm 4.5.

CC: @XiaoNi87, @mtkaczyk, @ncroxon, @ca-hu

[mtkaczyk]

@mwilck when I was at Intel I wanted to remove creation by mknod and always use create_named_array. Could you please check if
create_named_array is free from regression? see create_mddev(), you have to set mdadm --create params somehow special for native to force mdadm to use named_array. Sorry, I don't remember it in details..

[mtkaczyk]

mdadm 4.5 changed this code in d354d31 ("mdadm: Create array with sync del gendisk mode"). If mdadm can't access certain sysfs attribute of the md_mod module early on, it prints an error message and quits, and never gets to the point where earlier mdadm versions would try the mknod()/open() trick.

but functionally, we have a clear regression for users- if we can avoid it, we probably should. @XiaoNi87 what do you think here?

[mwilck]

Could you please check if create_named_array is free from regression?

Could you be more explicit in what you want me to test? The problem I explained above occurs only when mdadm is called from udev rules, where it has a special SELinux context. When it is called from the command line by root, I don't expect an issue.

but functionally, we have a clear regression for users

Only on systems with SELinux enabled, where the policy forbids what mdadm is trying to do. I don't think that we want to re-introduce that weird mknod()/open() way to load the md kernel module.

(Btw, I found the following in the kernel code in drivers/md/Kconfig:

config BLK_DEV_MD
	tristate "RAID support"
	select BLOCK_HOLDER_DEPRECATED if SYSFS
	select BUFFER_HEAD
	# BLOCK_LEGACY_AUTOLOAD requirement should be removed
	# after relevant mdadm enhancements - to make "names=yes"
	# the default - are widely available.
	select BLOCK_LEGACY_AUTOLOAD

IOW the kernel md driver depends on BLOCK_LEGACY_AUTOLOAD because of an alleged deficiency of mdadm ... do you understand what this is about?)

Back to the concrete issue described here, there are basically two straightforward ways to fix this problem:

1. Allow mdadm to load kernel modules even if run from udev rules. This is what the above SELinux policy PR does. It is obviously a security concern, as rules need to be relaxed. It's best practice to keep the set of allowed system calls as small as possible. OTOH, I believe that this modification doesn't significantly increase the attack surface of a typical Linux system. I just can judge this, I'm not a security expert.
2. Load md_mod early on on systems where mdadm is installed, e.g. by adding it to modules-load.d. This works, but is also frowned upon by some, because it may increase kernel size in memory even on systems that don't use MD (why would these have mdadm installed, then?). This was my first idea to solve the issue on openSUSE, but it was turned down because of openSUSE-specific package dependency issues it caused. So we're now pursuing 1).

[mtkaczyk]

Could you be more explicit in what you want me to test? The problem I explained above occurs only when mdadm is called from udev rules, where it has a special SELinux context. When it is called from the command line by root, I don't expect an issue.

Sure, there is following function:
int create_named_array(char *devnm)

Which is used to create MD device by writing to the file when SOME criteria are met. It is ugly but by playing with mdadm.conf and name in metadata you can force mdadm to use this way (i.e create the md device by writing to "/sys/module/md_mod/parameters/new_array" instead of mknod and open).

See create_mddev but first take some pills to avoid headache :)

IOW the kernel md driver depends on BLOCK_LEGACY_AUTOLOAD because of an alleged deficiency of mdadm ... do you understand what this is about?)

Reading Here
Looks like this is exactly the functionality you are described in details.

If I'm right, the third way would be to finally create md-devices by this magic "/sys/module/md_mod/parameters/new_array" file but it would be painful to implement at this stage of project.