Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1976621 - Boot stops at Reached target Initrd Root Device
Summary: Boot stops at Reached target Initrd Root Device
Keywords:
Status: CLOSED DUPLICATE of bug 1976653
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 34
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-27 15:34 UTC by Henrik Nordström
Modified: 2021-07-09 11:38 UTC (History)
31 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-30 00:40:10 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1976653 1 unspecified NEW dracut lock-up of 1m30s in initrd due to new dbus-broker ordering causing slow boot 2022-05-16 11:32:56 UTC
Red Hat Bugzilla 1976873 1 unspecified CLOSED PC doesn't start with kernel 5.12.12 with dracut 055-2.fc34 2022-05-16 11:32:56 UTC

Description Henrik Nordström 2021-06-27 15:34:44 UTC
1. Please describe the problem:

Boot stops very early at "Reached target Initrd Root Device" if configured for suspend support (suspend=... kernel argument).

System still responds to keyboard input and reboot is kind of possible, ctrl-alt-delete performs the reboot actions but do not actually reboot.

If the suspend= argument is removed from the boot command line the boot works as usual.

2. What is the Version-Release number of the kernel:

kernel-5.12.12-300.fc34.x86_64
kernel-5.12.13-300.fc34.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

Not sure on exact version, or even if this is a kernel issue or a dracut issue.

It does work with kernel-5.11.17-300.fc34.x86_64 on the same system, but that initramfs is a bit older.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Configure the system for suspend (suspend=/dev/mapper/fedora-swap). Try to boot.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Not tested beyond current updates-testing.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

Tested without any additional modules loaded.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Issue occurs before logs can be saved.

Not sure if it's a kernel or dracut issues. Was very many updates at the same time 5.12.12 was installed, and not familiar with how to diagnose this early stage of the boot.

Comment 1 Garry T. Williams 2021-06-27 16:03:17 UTC
Just to be clear, I have *never* configured suspend= on any of my machines.  I encountered the hang at "Reached target Initrd Root Device" on my XPS 13 with this (5.12.13-300.fc34.x86_64) kernel.  Booting the previous kernel (5.12.12-300.fc34.x86_64) fixes the problem for me.  This is clearly, for me on my laptop, a regression.

Incidentally, the 5.12.13-300.fc33.x86_64 kernel boots without incident on my workstation with F33.  I haven't upgraded my workstation to F34 yet.

Comment 2 Henrik Nordström 2021-06-27 16:19:00 UTC
Correction, the boot argument is resume= not suspend=

Tested 5.11.18-300.fc34.x86_64 and that works with a fresh initramfs, so it's most likely kernel dependent. Will test more versions to narrow it down further.

Comment 3 Henrik Nordström 2021-06-27 17:05:29 UTC
Hmm.. testing this is harder than expected. Just found that the 5.11.18 results is invalid as not same initramfs was generated, missing the hibernate-resume dracut module. Restarting.

Comment 4 Henrik Nordström 2021-06-27 17:32:12 UTC
With initramfs regenerated from the working 5.11.17 release resulting in an initramfs that includes the resume dracut module then 5.11.18-300 also fails. So my issue is very likely dracut related. But will verify by regenerating the 5.11.17 initramfs as well. The working 5.11.17 initramfs do include the resume dracut module, but much is older versions.

To make things even more confusing there is different screen resolution selected with or without resume= kernel argument. Without resume= kernel argument the screen gets mirrored on both displays. With resume= kernel argument the screen is only shown on the internal screen and at native resolution, but then hangs at the "Reached target Initrd Root Device".

I do not think there was any other difference than the resume dracut module when the initramfs was regenerated. But unfortunately I did not save the previous updated version which was generated without it.

Comment 5 Henrik Nordström 2021-06-27 17:45:59 UTC
Looking at the old initramfs its based on dracut-053-5.fc34 while the new is dracut-055-2.fc34

Comment 6 Stefano Corti 2021-06-27 18:47:12 UTC
I have the same the lock at "Reached target Initrd Root Device" on kernel-5.12.12-300.fc34.x86_64 + dracut-055-2.fc34.x86_64 creating initramfs.

Additional note, the boot is not completely locked. After a couple of minutes (on an Intel P4) the boot restarts, however there are odd behaviours in the system - I noticed systemd-resolved did not setup the system properly , but I had not time so far for a complete analysis.

As I had similar problems in the past with dracut, I downgraded dracut to dracut-053-4.fc34.x86_64 and recreated initramfs with "dracut -f -v".
This action solves the issue, I'd say it's not about the kernel but some dracut regression.

Comment 7 Garry T. Williams 2021-06-27 21:56:41 UTC
(In reply to Stefano Corti from comment #6)
> Additional note, the boot is not completely locked. After a couple of
> minutes (on an Intel P4) the boot restarts,

I tested this on the XPS 13 and confirm that the boot sequence eventually proceeds.

> however there are odd behaviours
> in the system - I noticed systemd-resolved did not setup the system properly

I see no "odd behaviors" here.  The boot looks to be resulting in a normal system after the long delay.

Comment 8 Henrik Nordström 2021-06-27 22:36:07 UTC
I agree. This looks very much like a dracut issue. And at least for me it seems to be isolated to the resume module as without resume= on the command line or alternatively without the dracut resume module (effectively the same as no resume= argument) in the initramfs the boot proceeds as normal.

For certain it does NOT look like a kernel issue, or at least not a kernel regression as the exact same result is seen with older kernels. Reassigning the bug to dracut.

Garry, you say you have not configured suspend. Please double check by running
  grep resume= /proc/cmdline

Or checking the kernel command line contents in the grub menu for the boot entry that exibits the problem.

It is quite likely the most recent dracut update hit your system after the 5.12.12 kernel update and only the 5.12.13 intramfs is built with the latest dracut in your setup, and this is why you don't see it with 5.12.12.

It's good the system eventually boots. I apparently do not have as much patience as you.

Comment 9 Henrik Nordström 2021-06-27 22:52:28 UTC
Probably the same issue as Bug #1976653

Comment 10 Henrik Nordström 2021-06-27 23:09:38 UTC
Downgrading dracut to dracut-053-5.fc34.x86_64 solves the issue.

dracut-054-12.git20210521.fc34.x86_64 fails.

Have not yet tested the other versions inbetween these.

Works means for me that it boots in the same manner as if I remove the resume= option from the kernel command line: no long delay and boot screen is mirrored on both displays at crappy resolution. When it fails boot screen is only shown on internal high res display in high resolution, and there is a very long delay (or I assume it is just a long delay, have not actually verified myself that it eventually boots but I take the word of the others in this thread)

Comment 11 Henrik Nordström 2021-06-27 23:14:35 UTC
Just to be crystal clear, dracut-055-2.fc34.x86_64 also fails.

dracut-055-2 was the version used then the bug was discovered. Tried downgrading first to dracut-054-12.git20210521.fc34.x86_64 which did not solve the problem and then all the way back to the previous known good version dracut-053-5.fc34.x86_64 and the problem is gone without downgrading any of the other updated packages or kernel. Yes the initramfs was regenereated at each step.

Comment 12 Garry T. Williams 2021-06-27 23:32:18 UTC
(In reply to Henrik Nordström from comment #8)
> Garry, you say you have not configured suspend. Please double check by
> running
>   grep resume= /proc/cmdline

Oops.  Yes, I have resume= still configured from a long time ago.  I just deleted it and rebooted.  Now no delay during the boot-up sequence.

Comment 13 Henrik Nordström 2021-06-27 23:51:36 UTC
dracut-054-6.git20210518.fc34.x86_64 fails.

dracut-054-5.git20210517.fc34 is deleted so can not narrow it down further in versions.

dracut-053-5.fc34.x86_64 works.

Comment 14 Henrik Nordström 2021-06-28 00:11:59 UTC
(In reply to Garry T. Williams from comment #12)

> Oops.  Yes, I have resume= still configured from a long time ago.  I just
> deleted it and rebooted.  Now no delay during the boot-up sequence.

Good, Same as me then, also have it configured from a long time ago and forgot about it. Good. So there is very very strong indication that is's the dracut resume module that is the culpit.

Comment 15 Henrik Nordström 2021-06-28 00:32:09 UTC
Can not see any changes in the resume module, but 054 brings in some additional modules compared to 053

--- ./dracut-053-5.fc34/initramfs/usr/lib/dracut/modules.txt	2021-04-23 10:57:15.000000000 +0200
+++ ./dracut-054-6.git20210518.fc34/initramfs/usr/lib/dracut/modules.txt	2021-05-18 10:36:38.000000000 +0200
@@ -1,8 +1,11 @@
 bash
 systemd
 systemd-initrd
+systemd-sysusers
 nss-softokn
+dbus-broker
 rngd
+dbus
 i18n
 network-manager
 network


Disabling dbus does get rid of the problem. But also disables network. Now I am not sure why network is enabled in the first place, there is no need for network in a local initramfs. But network also was in the 053 initramfs. Also not sure what the relation to resume is or why the framebuffer behaves differently.

$ sudo dracut -o "dbus" -f
dracut: dracut module 'network-manager' depends on 'dbus', which can't be installed
dracut: dracut module 'network' depends on 'network-manager', which can't be installed
dracut: dracut module 'ifcfg' depends on 'network', which can't be installed
dracut: dracut module 'ifcfg' depends on 'network', which can't be installed

Verified on 053-5 and 055-2

Note: There is warnings about dbus during the boot of 053 at the same time where later versions waits.

Comment 16 Satish Balay 2021-06-28 18:10:28 UTC
I have this issue [Boot stops very early at "Reached target Initrd Root Device"] on a ThinkPad T430s

This is with dracut-055-2.fc34.x86_64

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.12.13-300.fc34.x86_64 root=/dev/mapper/fedora-root ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet "acpi_osi=!Windows 2012"

Removing "acpi_osi=!Windows 2012" did not help

downgrading to dracut-053-4.fc34.x86_64 and reinstalling kernel works.

I have a couple other boxes - another ThinkPad T430s, and T530 - with F34 - but they don't have this issue.

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.12.13-300.fc34.x86_64 root=UUID=d30b0af7-cac0-4be8-bfee-1bc608b8edc9 ro rootflags=subvol=root rd.luks.uuid=luks-da35535c-851d-46a7-b5e6-fdd75a2636ea rhgb quiet

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.12.13-300.fc34.x86_64 root=UUID=a7c35b40-18d5-477e-8047-16b9b78b6111 ro resume=UUID=dc581e65-3727-4724-aae5-182856f6bed6 rhgb quiet

Comment 17 Silvio Schneider 2021-06-29 06:59:50 UTC
I have the same problem on a Compaq CQ2000 PC.
See https://bugzilla.redhat.com/show_bug.cgi?id=1976873

Comment 18 Stefano Corti 2021-06-29 12:14:59 UTC
(In reply to Garry T. Williams from comment #7)
> > however there are odd behaviours
> > in the system - I noticed systemd-resolved did not setup the system properly
> 
> I see no "odd behaviors" here.  The boot looks to be resulting in a normal
> system after the long delay.

It turns out that the systemd-resolved issue is unrelated to this one, likely the same of https://bugzilla.redhat.com/show_bug.cgi?id=1976161

Comment 19 ryan@testtoast.com 2021-06-29 21:52:15 UTC
Confirmed on a Lenovo ThinkPad X1Y4/FC34/custom 5.13+ kernel - boot fails/delayed after install using dracut 0.55, downgrade to 0.53 and reinstalling kernel works.

Comment 20 Henrik Nordström 2021-06-29 23:41:31 UTC
(In reply to Satish Balay from comment #16)

> downgrading to dracut-053-4.fc34.x86_64 and reinstalling kernel works.

The two workarounds mentioned above should also work.

Either

a) Rebuild the initramfs without dbus

   sudo dracut -o "dbus" -f

b) Or disable hibernate/resume support (resume= kernel command line option in grub)
 
> I have a couple other boxes - another ThinkPad T430s, and T530 - with F34 -
> but they don't have this issue.
> 
> # cat /proc/cmdline 
> BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.12.13-300.fc34.x86_64
> root=UUID=d30b0af7-cac0-4be8-bfee-1bc608b8edc9 ro rootflags=subvol=root
> rd.luks.uuid=luks-da35535c-851d-46a7-b5e6-fdd75a2636ea rhgb quiet

On this one you have not enabled hibernate/resume support (no resume= option).

> $ cat /proc/cmdline 
> BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.12.13-300.fc34.x86_64
> root=UUID=a7c35b40-18d5-477e-8047-16b9b78b6111 ro
> resume=UUID=dc581e65-3727-4724-aae5-182856f6bed6 rhgb quiet

Odd that this does one not show the same problem. Are you sure the initramfs was generated with dracut-055?

  journalctl -b | grep dracut- | head -1

Note: There is no need to reinstall the kernel. To rebuild the initramfs just run "sudo dracut -f".

Comment 21 Henrik Nordström 2021-06-30 00:15:42 UTC
Apparently caused by an dbus update. See https://bugzilla.redhat.com/show_bug.cgi?id=1976653#c4 for a discussion and alternate workaround.

Comment 22 Satish Balay 2021-06-30 00:19:56 UTC
> b) Or disable hibernate/resume support (resume= kernel command line option in grub)

These 3 boxes got OS installs at different times - i.e at different Fedora version release time (and upgraded to newer releases via dnf system-upgrade) - so the differences in kernel options are primary from these different versions of Fedora installers. The one without `resume=` was a clean F34 install. Its good to know that `resume=` is optional and can be removed.

> Odd that this does one not show the same problem. Are you sure the initramfs was generated with dracut-055?

$ rpm -qa --last kernel dracut
kernel-5.12.13-300.fc34.x86_64                Sat 26 Jun 2021 04:20:05 PM CDT
dracut-055-2.fc34.x86_64                      Tue 22 Jun 2021 09:43:21 AM CDT
kernel-5.12.12-300.fc34.x86_64                Sun 20 Jun 2021 10:34:21 PM CDT
kernel-5.12.11-300.fc34.x86_64                Thu 17 Jun 2021 02:55:59 PM CDT

Well my currently running kernel is installed after dracut-055-2 was installed - so I assumed that was used to generated its initramfs

# journalctl -b | grep dracut- | head -1
Jun 26 11:26:17 sb dracut-cmdline[288]: dracut-34 (Workstation Edition) dracut-055-2.fc34

Comment 23 Henrik Nordström 2021-06-30 00:27:56 UTC
(In reply to Satish Balay from comment #22)

> Well my currently running kernel is installed after dracut-055-2 was
> installed - so I assumed that was used to generated its initramfs

New findings indicate is that this isn't a dracut issue as such, but actually caused by an update to dbus-broker that broke dbus usage in the initramfs. So it's possible you have a working dracut-055 initramfs if it's build with older dbus-broker release. See Comment #21.

Comment 24 Henrik Nordström 2021-06-30 00:40:10 UTC
The workaround in bug #1976653 solves the issue. Marking this as duplicate of that bug.

*** This bug has been marked as a duplicate of bug 1976653 ***

Comment 25 Satish Balay 2021-06-30 00:43:30 UTC
> So it's possible you have a working dracut-055 initramfs if it's build with older dbus-broker release

$ rpm -q --last kernel-5.12.13-300.fc34.x86_64 dbus-broker dracut
kernel-5.12.13-300.fc34.x86_64                Sat 26 Jun 2021 04:20:05 PM CDT
dbus-broker-29-1.fc34.x86_64                  Sat 26 Jun 2021 04:16:13 PM CDT
dracut-055-2.fc34.x86_64                      Tue 22 Jun 2021 09:43:21 AM CDT

Comment 26 RedTed 2021-07-08 16:25:29 UTC
Am also able to reproduce this issue and can confirm that removing the "resume" argument resolves the issue.

For anyone encountering the issue who needs help with removing the kernel argument, here are some instructions:

run

sudo grubby --info=DEFAULT

look for the "resume=..." argument on the "args" line. My output looks like this:

                               VVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
args="ro rd.lvm.lv=fedora/root resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/swap quiet systemd.unified_cgroup_hierarchy=0 mpt3sas.max_queue_depth=10000"

then remove the argument from all kernels. In my case the command is

                           VVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
sudo grubby --remove-args="resume=/dev/mapper/fedora-swap" --update-kernel=ALL

To check your work, run sudo grubby --info=DEFAULT again and check the "args" line.

Reboot the system to apply the change.

Comment 27 Leonard Ehrenfried 2021-07-09 07:09:31 UTC
What are the consequences of removing the resume boot option?

I don't remember putting it there.

Comment 28 Henrik Nordström 2021-07-09 11:38:52 UTC
Note: Discussion continues in bug #1976653

The drawback from removing the resume= kernel argument is that hibernate to disk no longer works, only suspend to ram. I can not remember last time I used hibernate to disk however, only suspend to ram and full poweroff.


Note You need to log in before you can comment on or make changes to this bug.