Mirror array fails to start after boot with one drive removed

[boekhold]

First of all, I'm not sure if this is an issue with mdadm itself, with systemd, udev or Debian, but I'll try here first. Feel free to tell me to raise this issue somewhere else if this is not related to this project.

I have a Debian Trixie 13 system, installed with debootstrap, with an madm RAID1 array /dev/md/esp mounted under /boot/efi. It's actually a KVM VM that I built to test this setup. If I shut down the VM, remove one of the drives and restart, the array does not get assembled/started and remains as inactive as per proc/mdstat.

As far as I have been able to determine, what is supposed to happen is that udev detects the remaining drive and then starts a systemd timer mdadm-last-resort@md127.timer through a udev rule from /lib/udev/rules.d/64-md-raid-assembly.rules:

ACTION!="remove", ENV{MD_STARTED}=="*unsafe*", ENV{MD_FOREIGN}=="no", ENV{SYSTEMD_WANTS}+="mdadm-last-resort@$env{MD_DEVICE}.timer"

This rule never seems to get fired however. I can test this by running in one terminal:

udevadm monitor --environment --udev

and then from another terminal:

sudo udevadm trigger --action=add /sys/class/block/md127

The first terminal will now show:

monitor will print the received events for:
UDEV - the event which udev sends out after rule processing

UDEV  [71.177525] add      /devices/virtual/block/md127 (block)
ACTION=add
DEVPATH=/devices/virtual/block/md127
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/md127
DEVTYPE=disk
DISKSEQ=2
SEQNUM=2697
USEC_INITIALIZED=1228668
ID_IGNORE_DISKSEQ=1
SYSTEMD_READY=0
UDISKS_MD_DEVICES=1
UDISKS_MD_METADATA=1.0
UDISKS_MD_UUID=348d222f:a97865fe:e49a1e7b:c3db1ab1
UDISKS_MD_NAME=any:esp
UDISKS_MD_DEVICE_dev_vda1_ROLE=spare
UDISKS_MD_DEVICE_dev_vda1_DEV=/dev/vda1
MAJOR=9
MINOR=127
TAGS=:systemd:
CURRENT_TAGS=:systemd:

I can just not figure out why this udev rule does not get fired and the systemd timer does not get created. I can start the systemd timer manually (sudo systemctl start mdadm-last-resort@md127.timer) and this does cause the array to get started (although I have to mount it manually then).

Array was created with:

yes | mdadm --create --verbose --level 1 --metadata 1.0 \
    --homehost any --raid-devices 2 /dev/md/esp \
    /dev/vda1 /dev/vdb1
mdadm --assemble --scan
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Note that I followed the instructions from the ZFSBootMenu project for this. Also, this really seems to create /dev/md127, with /dev/md/esp as a symlink to that.

mdadmin.conf:

# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

AUTO +all

# definitions of existing MD arrays
ARRAY /dev/md127  metadata=1.0 UUID=348d222f:a97865fe:e49a1e7b:c3db1ab1

# This configuration was auto-generated on Fri, 17 Apr 2026 09:01:14 +0000 by mkconf

/etc/fstab:

UUID=F2A1-708E /boot/efi vfat defaults,x-systemd.device-timeout=40s,nofail 0 0

Versions:

mdadm - v4.4 - 2024-11-07 - Debian 4.4-11
Linux hoster 6.12.74+deb13+1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.74-2 (2026-03-08) x86_64 GNU/Linux

[boekhold]

I've discovered how to enable debug logging for udev. Below is the full debug log of sudo udevadm trigger --action=add /sys/class/block/md127 obtained using sudo journalctl -f -u systemd-udevd. I see an error related to vdev_id failing on a missing file /etc/zfs/vdev_id.config. Could this be related?

Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: Successfully created symlink '/dev/block/9:127' to '/dev/md127'
Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: sd-device: Created database file '/run/udev/data/b9:127' for '/devices/virtual/block/md127'.
Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: Adding watch on '/dev/md127'
Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: sd-device: Created database file '/run/udev/data/b9:127' for '/devices/virtual/block/md127'.
Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: Device processed (SEQNUM=2717, ACTION=add)
Apr 19 19:11:47 hoster (udev-worker)[2680]: md127: sd-device-monitor(worker): Passed 920 byte to netlink monitor.
Apr 19 19:11:47 hoster systemd-udevd[615]: No events are queued, removed /run/udev/queue.
Apr 19 19:11:50 hoster systemd-udevd[615]: Cleaning up idle workers.
Apr 19 19:11:50 hoster systemd-udevd[615]: Worker [2680] exited.
Apr 19 19:11:54 hoster systemd-udevd[615]: Cleaning up idle workers.
Apr 19 19:12:25 hoster systemd-udevd[615]: md127: Device is queued (SEQNUM=2718, ACTION=add)
Apr 19 19:12:25 hoster systemd-udevd[615]: md127: Device ready for processing (SEQNUM=2718, ACTION=add)
Apr 19 19:12:25 hoster systemd-udevd[615]: Successfully forked off '(udev-worker)' as PID 2713.
Apr 19 19:12:25 hoster systemd-udevd[615]: md127: Worker [2713] is forked for processing SEQNUM=2718.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Processing device (SEQNUM=2718, ACTION=add)
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Removing watch handle 11.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: sd-device: Created database file '/run/udev/data/b9:127' for '/devices/virtual/block/md127'.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/50-udev-default.rules:88 GROUP 6
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/63-md-raid-arrays.rules:21 Importing properties from results of '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Starting '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: Successfully forked off '(spawn)' as PID 2714.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'(out) 'MD_LEVEL=raid1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'(out) 'MD_DEVICES=2'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'(out) 'MD_METADATA=1.0'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'(out) 'MD_UUID=348d222f:a97865fe:e49a1e7b:c3db1ab1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/sbin/mdadm --detail --no-devices --export /dev/md127'(out) 'MD_NAME=any:esp'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Process '/usr/sbin/mdadm --detail --no-devices --export /dev/md127' succeeded.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/63-md-raid-arrays.rules:22 Added SYMLINK 'disk/by-id/md-name-any:esp'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/63-md-raid-arrays.rules:23 Added SYMLINK 'disk/by-id/md-uuid-348d222f:a97865fe:e49a1e7b:c3db1ab1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/63-md-raid-arrays.rules:30 Importing properties from results of builtin command 'blkid'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Probe /dev/md127 with raid and offset=0
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/63-md-raid-arrays.rules:33 Added SYMLINK 'disk/by-uuid/F2A1-708E'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/69-vdev.rules:5 Importing properties from results of '/usr/lib/udev/vdev_id -d md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Starting '/usr/lib/udev/vdev_id -d md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: Successfully forked off '(spawn)' as PID 2715.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/usr/lib/udev/vdev_id -d md127'(out) 'Error: Config file "/etc/zfs/vdev_id.conf" not found'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Process '/usr/lib/udev/vdev_id -d md127' failed with exit code 1.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/69-vdev.rules:5 Command "/usr/lib/udev/vdev_id -d md127" returned 1 (error), ignoring
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/80-udisks2.rules:22 Importing properties from results of '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Starting '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''
Apr 19 19:12:25 hoster (udev-worker)[2713]: Successfully forked off '(spawn)' as PID 2716.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_LEVEL=raid1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_DEVICES=2'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_METADATA=1.0'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_UUID=348d222f:a97865fe:e49a1e7b:c3db1ab1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_NAME=any:esp'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_DEVICE_dev_vda1_ROLE=0'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g''(out) 'UDISKS_MD_DEVICE_dev_vda1_DEV=/dev/vda1'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Process '/bin/sh -c '/sbin/mdadm --detail --export /dev/md127 | /bin/sed s/^MD_/UDISKS_MD_/g'' succeeded.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Preserve permissions of /dev/md127, uid=0, gid=6, mode=0660
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Successfully created symlink '/dev/block/9:127' to '/dev/md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: sd-device: Created database file '/run/udev/data/b9:127' for '/devices/virtual/block/md127'.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Adding watch on '/dev/md127'
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: sd-device: Created database file '/run/udev/data/b9:127' for '/devices/virtual/block/md127'.
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: Device processed (SEQNUM=2718, ACTION=add)
Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: sd-device-monitor(worker): Passed 920 byte to netlink monitor.
Apr 19 19:12:25 hoster systemd-udevd[615]: No events are queued, removed /run/udev/queue.
Apr 19 19:12:29 hoster systemd-udevd[615]: Cleaning up idle workers.
Apr 19 19:12:29 hoster systemd-udevd[615]: Worker [2713] exited.
Apr 19 19:12:33 hoster systemd-udevd[615]: Cleaning up idle workers.

[mwilck]

Could this be related?

No. As you can see, this problem is ignored:

Apr 19 19:12:25 hoster (udev-worker)[2713]: md127: /usr/lib/udev/rules.d/69-vdev.rules:5 Command "/usr/lib/udev/vdev_id -d md127" returned 1 (error), ignoring

(You apparentöy have some zfs related packages installed without using them, which might indicate a problem in a different area).

You are looking at the add event for the md device, which is not what matters here. You should be looking at the events related to the member device(s) of the MD array (vda1 and vdb1). For a RAID1, the add event of the first member device (probably vda1) must trigger mdadm-last-resort@md127.timer, as you figured out correctly.

I suggest booting with udev.log_level=debug and examining the entire boot log, specifically what happens when vda and vda1 are started. Is the timer started as expected? If yes, what happens when it expires? Or perhaps, does another timeout expire earlier and prevent the md timeout from setting up the RAID device in degraded state?

When you are in emergency mode, can you start the array manually e.g. by running mdadm --run /dev/md127?

[mtkaczyk]

I have a Debian Trixie 13 system, installed with debootstrap, with an madm RAID1 array /dev/md/esp mounted under /boot/efi. It's actually a KVM VM that I built to test this setup. If I shut down the VM, remove one of the drives and restart, the array does not get assembled/started and remains as inactive as per proc/mdstat.

We have an issue with kernel module loading, perhaps it is related? what is your's mdadm version and kernel version?
see:
#228

Could you please test with the latest mdadm version 4.6?

I see that in Trixie, mdadm 4.4-11 is used, right?
https://packages.debian.org/trixie/mdadm

Just a blind guess of course, you can either proceed with debugging as @mwilck suggested. Choose what is best for you.