systemd-gpt-auto-generator settings (Dependency failed for Initrd Root Device)

[hv15]

Hi there,

Problem:

I keep experience a delay on system startup due to systemd not being able to load the root device (see journalctl output bellow). To be clear, the system does boot and eventually I reach a TTY to login to. Funnily though, my /boot partition does not get mounted, I suspect this is due to the dependency on the root device.

Jun 20 15:12:14 sys1 systemd[1]: dev-gpt\x2dauto\x2droot.device: Job dev-gpt\x2dauto\x2droot.device/start timed out.
Jun 20 15:12:14 sys1 systemd[1]: Timed out waiting for device /dev/gpt-auto-root.
Jun 20 15:12:14 sys1 systemd[1]: Dependency failed for Initrd Root Device.
Jun 20 15:12:14 sys1 systemd[1]: initrd-root-device.target: Job initrd-root-device.target/start failed with result 'dependency'.
Jun 20 15:12:14 sys1 systemd[1]: dev-gpt\x2dauto\x2droot.device: Job dev-gpt\x2dauto\x2droot.device/start failed with result 'timeout'.

System Information and mkinitcpio-systemd-tool Configuration

I use Arch Linux (linux-lts kernel). My boot drive has two partitions on it, one for boot (EFI vFat) and the other is the root partition, setup as LVM on LUKs. The LVM partition has two volume groups (VGs), one for the root filesystem (called vg-root) and one for the home directory (called vg-home). I use systemd-boot with the these kernel parameters:

ipv6.disable_ipv6=1 vsyscall=emulate intel_idle.max_cstate=1 quiet rw` 

(notice that I omit root=).

Versions

$ systemctl --version
systemd 245 (245.6-7-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
$ uname -a
Linux sys1 5.4.47-1-lts #1 SMP Wed, 17 Jun 2020 19:42:02 +0000 x86_64 GNU/Linux

systemd services

I have the following mkinitcpio-systemd-tool services enabled:

$ find /etc/systemd/system -type l -iname 'initrd*'
/etc/systemd/system/initrd-debug-shell.service.wants/initrd-debug-progs.service
/etc/systemd/system/initrd-root-fs.target.wants/initrd-sysroot-mount.service
/etc/systemd/system/sysinit.target.wants/initrd-cryptsetup.path
/etc/systemd/system/sysinit.target.wants/initrd-debug-progs.service
/etc/systemd/system/sysinit.target.wants/initrd-dropbear.service

Files

mkinitcpio-systemd-tool/config/crypttab has only this:

cryptlvm	UUID=3cdf6a91-1a36-4cdc-8eb4-fbd2d799d8d9

mkinitcpio-systemd-tool/config/fstab has only this (where vg-root is the name of the LVM VG containing the root filesystem):

/dev/mapper/vg-root	/sysroot         auto	x-systemd.device-timeout=9999h	0	1

crypttab has:

cryptlvm	UUID=3cdf6a91-1a36-4cdc-8eb4-fbd2d799d8d9

fstab has:

# /dev/mapper/vg-root
UUID=30803547-45e3-4905-b601-ecb92522c40c	/         	ext4      	rw,relatime,data=ordered	0 1

# /dev/mapper/vg-home
UUID=76d8381f-fc87-4f22-b32a-6c1873b12ad1	/home     	ext4      	rw,relatime,data=ordered	0 2

UUID=E361-10AD      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro	0 2

Is my setup of mkinitcpio-systemd-tool wrong? I followed the 'instructions' as outlined in #62.

Any suggestions are welcome 😸 !

[Andrei-Pozolotin]

1. please confirm lvm is present in /etc/mkinitcpio.conf

HOOKS=(base keyboard autodetect modconf block filesystems fsck systemd sd-lvm2 systemd-tool)

2. try to read through more lvm-related wiki cases
   https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-LVM
   https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Complete-BTRFS-Setup-(step-by-step)

[hv15]

Hi @Andrei-Pozolotin, oops forgot to include content of mkinitcpio.conf (hooks):

HOOKS=(base keyboard autodetect modconf block filesystems fsck systemd sd-vconsole sd-lvm2 systemd-tool)

[Andrei-Pozolotin]

assume resolved

[dvzrv]

I'm actually running into the same issue on several systems (with a system setup close to the one described above).

When booting without network access (e.g. on a laptop), after typing in the password, a start job for dev-gpt-auto-root.device blocks early userspace for 1:30m. After that boot commences.
When booting with network access the message about the service is never shown, but I can see it in the list of units (i.e. systemctl list-units -a) as "starting" when ssh'ing to the machine and using the shell.

Fun fact: My fstab for mkinitcpio-systemd-tool does not have a x-systemd.device-timeout=9999h set, so eventually the Starting Cryptography Setup for xxx... fails. As I have now also added the sd-encrypt hook to my mkinitcpio.conf, I can still ssh to the machine after the timeout is reached, type in my password and the boot goes through instantly (no .device unit blocking).
This is of course no improvement (as waiting for the timeout takes even longer), but an interesting observation about this issue.

[dvzrv]

Also, this entire issue does not appear, when I do not use crypttab and fstab in the initrd but rely on kernel commandline parameters.

[Andrei-Pozolotin]

@dvzrv great, since you can actually reproduce this :-)

[dvzrv]

I believe I have figured out what triggers this behavior (in my case), now that I had some time to read up on things.

The systemd-gpt-auto-generator acts according to the "discoverable partitions specification", which in turn makes a few assumptions about the layout of a partition.

My system facilitates btrfs on LUKS in a flat hierarchy. Here the default subvolume (id 5) is not mounted as /, but a specific subvolume below it is used for that. The opened device is named root0 in my case (as I have an automatic way of installing systems using ansible, that may use more than one disk to e.g. form a btrfs raid).
This does not seem to go too well with the aforementioned generator which probably tries to use the default subvolume (id 5) as root until timing out (also, its default is to open the device as root).

I have now added systemd.gpt_auto=no to my kernel parameters to disable the generator altogether (as fstab is in use anyways) and the systemd boots without delay.

@hv15 I hope this will also fix your issues (which seem to be of the same nature, but I'm not 100% sure).
@Andrei-Pozolotin I guess this is worth documenting, because I believe I am not the only one who is using a flat hierarchy for the btrfs subvolumes.

[ArchangeGabriel]

@dvzrv What is there inside your /etc/mkinitcpio-systemd-tool/config/fstab? In my case I specified the subvolume there (because I also use a flat hierarchy) and it works.

[dvzrv]

@ArchangeGabriel it is only:

/etc/mkinitcpio-systemd-tool/config/fstab:

/dev/mapper/root0      /sysroot         btrfs           noatime,ssd,subvolid=259,subvol=/@,subvol=@     0       1

On that specific machine I only have a subvolume for home otherwise, so that's not important in early userspace (which is why I only specify /sysroot.

That being said, this is my list of subvolumes:

D 256 gen 17832 top level 5 path @home
ID 257 gen 8 top level 5 path @swap
ID 259 gen 17830 top level 5 path @
ID 262 gen 25 top level 259 path @/var/lib/portables
ID 263 gen 26 top level 259 path @/var/lib/machines

[dvzrv]

@ArchangeGabriel did you set your btrfs subvolume used for / as the default (i.e. btrfs subvolume set-default)? I did not and maybe that is the culprit (doing it is also mentioned in the generator's man page).

[ArchangeGabriel]

No, I did not. Over I notice that you specified 3 time the subvol directive, not sure that’s a good thing (I know all three settings are correct, but ykk). My line is just:

/dev/mapper/root	/sysroot	auto	subvol=@,x-systemd.device-timeout=9999h	0	1

[Andrei-Pozolotin]

@dvzrv re: doc: added to wiki home: https://github.com/random-archer/mkinitcpio-systemd-tool/wiki

[dvzrv]

@ArchangeGabriel ah, I believe that it is because your device opens as /dev/mapper/root, mine as /dev/mapper/root0.
The generator defaults to using root as device according to their specification. I guess in my case the generator finds a btrfs device through device-mapper, but it is not able to mount it, because it is already mounted via root0 at that point in time.