Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1955416

Summary: Lenovo ThinkPad T490, unable to boot following clean install, stuck at splash screen
Product: [Fedora] Fedora Reporter: Chris Murphy <bugzilla>
Component: shimAssignee: Peter Jones <pjones>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: fmartine, germano.massullo, martin, mjg59, opendreas, pjones, redhat-bugzilla, sethgoldin, woat
Target Milestone: ---Flags: woat: needinfo? (sethgoldin)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
efivars
none
dmesg
none
Verbose output before hanging
none
Screenshot of text appearing on screen none

Description Chris Murphy 2021-04-30 04:58:51 UTC
Description of problem:

Following a clean default/automatic installation of Fedora 34, the system hangs at the Lenovo splash screen.

Booting a USB stick again, efibootmgr shows a bat guano bootorder that is instigated by shim 15.4-4, because the problem doesn't happen upon downgrading to shim 15-8, but immediately reoccurs when upgrading back to 15.4-4.

The bootorder is apparently not completely honored by the firmware, FEdora is in the 8th position and yet fallback isn't working well enough to get to that point. So there's certainly a firmware bug here, but it seems shim 15.4-4 is instigating part of this in a way that shim 15-8 wasn't.


Version-Release number of selected component (if applicable):
shim 15.4-4

How reproducible:
Always


Steps to Reproduce:
1. Install https://download.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-34-1.2.iso
2. Reboot
3.

Actual results:

Hang at Lenovo splash screen


Expected results:

Should boot


Additional info:

efibootmgr -v following boot from USB stick after failed first boot (of the installed system)

BootCurrent: 001F
Timeout: 0 seconds
BootOrder: 001F,0010,0011,0012,0013,0014,0015,0000,0019,001A,001B,001C,001D,001E,0020,0021,0022,0023
Boot0000* Fedora	HD(1,GPT,fb2c442e-2249-4bf8-a6c4-391e52174312,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
Boot0010  Setup	FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu	FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen	FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics	FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Regulatory Information	FvFile(478c92a0-2622-42b7-a65d-5894169e4d24)
Boot0015  ThinkShield secure wipe	FvFile(3593a0d5-bd52-43a0-808e-cbff5ece2477)
Boot0016  Startup Interrupt Menu	FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0017  Rescue and Recovery	FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0018  MEBx Hot Key	FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0019* USB CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot001A* USB FDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot001B* NVMe0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001C* NVMe1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a401)
Boot001D* ATA HDD0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot001E* ATA HDD1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f601)
Boot001F* USB HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot0020* PXE BOOT	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot0021* LENOVO CLOUD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,ad38ccbbf7edf04d959cf42aa74d3650)/Uri(https://download.lenovo.com/pccbbs/cdeploy/efi/boot.efi)
Boot0022  Other CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)
Boot0023  Other HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)
Boot0024* IDER BOOT CDROM	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,1)
Boot0025* IDER BOOT Floppy	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,0)
Boot0026* ATA HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0027* ATAPI CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)

anaconda storage.log shows



        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -c -w -L Fedora -d /dev/nvme0n1 -p 1 -l \EFI\fedora\shimx64.efi
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -b 0000 -B
        INFO:program:Running in chroot '/mnt/sysroot'... efibootmgr -c -w -L Fedora -d /dev/nvme0n1 -p 1 -l \EFI\fedora\shimx64.efi

Comment 1 Chris Murphy 2021-04-30 05:04:02 UTC
It was possible to assemble the installed system in chroot, downgrade to shim 15-8, and 'efibootmgr --bootorder 0000' and reboot the installed system successfully. Upon updating to shim 15.4-4 though, we're back to a failed boot even though Fedora is first in the bootorder.

efivars tar will be attached matching this nvram state:

BootCurrent: 001F
Timeout: 0 seconds
BootOrder: 0000,0019,001A,001B,001C,001D,001E,001F,0020,0021,0022,0023
Boot0000* Fedora	HD(1,GPT,fb2c442e-2249-4bf8-a6c4-391e52174312,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
Boot0010  Setup	FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu	FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen	FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics	FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Regulatory Information	FvFile(478c92a0-2622-42b7-a65d-5894169e4d24)
Boot0015  ThinkShield secure wipe	FvFile(3593a0d5-bd52-43a0-808e-cbff5ece2477)
Boot0016  Startup Interrupt Menu	FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0017  Rescue and Recovery	FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0018  MEBx Hot Key	FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0019* USB CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot001A* USB FDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot001B* NVMe0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001C* NVMe1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a401)
Boot001D* ATA HDD0	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot001E* ATA HDD1	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f601)
Boot001F* USB HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot0020* PXE BOOT	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot0021* LENOVO CLOUD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,ad38ccbbf7edf04d959cf42aa74d3650)/Uri(https://download.lenovo.com/pccbbs/cdeploy/efi/boot.efi)
Boot0022  Other CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)
Boot0023  Other HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)
Boot0024* IDER BOOT CDROM	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,1)
Boot0025* IDER BOOT Floppy	PciRoot(0x0)/Pci(0x14,0x0)/USB(11,0)
Boot0026* ATA HDD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0027* ATAPI CD	VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)

Comment 2 Seth Goldin 2021-04-30 05:05:22 UTC
Created attachment 1777572 [details]
efivars

Comment 3 Seth Goldin 2021-04-30 05:06:33 UTC
Created attachment 1777575 [details]
dmesg

Comment 4 Chris Murphy 2021-04-30 05:09:46 UTC
*** Bug 1955390 has been marked as a duplicate of this bug. ***

Comment 5 Chris Murphy 2021-04-30 05:11:12 UTC
[    0.000000] DMI: LENOVO 20N2CTO1WW/20N2CTO1WW, BIOS N2IET94W (1.72 ) 02/18/2021

Comment 6 Peter Hazenberg 2021-04-30 18:43:34 UTC
Hi, I'm having this same issue. My laptop is slightly different but behaves the same as above.

It's a Thinkpad Yoga 370, so 2 generations older as the T490.

`dmesg | grep DMI:`
[    0.000000] DMI: LENOVO 20JJS0VK1F/20JJS0VK1F, BIOS R0HET56W (1.36 ) 08/06/2020

Downgrading the shim works here too.

Setting the boot order or creating a new boot item with efibootmgr seems to be reset after rebooting all the time, which by itself doesn't cause any problem. It does indicate that the firmware is being weird.

When I set `mokutil --set-verbosity true` and booted the new-broken shim, it showed me the attached output at the point of hanging.

I also created a video of the entire boot process here: https://www.youtube.com/watch?v=FBtazoABHYY

Comment 7 Peter Hazenberg 2021-04-30 18:44:23 UTC
Created attachment 1777882 [details]
Verbose output before hanging

Comment 8 Peter Hazenberg 2021-04-30 19:40:52 UTC
I played around with this issue some more, and I found out that with the new-broken shim in place, the problem only occurs when secure boot is disabled.

In other words. I enabled secure boot. Then I powered off the machine (seems to be required, but rebooting twice also works). Problem gone, I can now successfully boot!

To verify, re-disabled secure boot. First boot was fine, the second warm-boot did hang again. Hard poweroff, enabled secure boot again. Warm reboot hanged again, second attempt was fine again.

Meanwhile I also recorded a video of a bunch of text appearing after booting via the "ssd" option instead of the "fedora" option: https://www.youtube.com/watch?v=vqKFPMFt25Q

Relevant screenshot also attached. This text can sometimes also appear in the hanging situation if you wait for long enough.

Comment 9 Peter Hazenberg 2021-04-30 19:41:47 UTC
Created attachment 1777915 [details]
Screenshot of text appearing on screen

Comment 10 Peter Jones 2021-04-30 20:07:19 UTC
(In reply to Peter Hazenberg from comment #9)
> Created attachment 1777915 [details]
> Screenshot of text appearing on screen

This looks very much like the firmware call to HandleProtocol() (shim.c:1104) returned success but gave us back a handle that's not completely populated.  Unfortunately that print would be the best clue as to what it's even trying to do when booting the "SSD" option, so I really have no idea what's going on there.  That said, it's *probably* unrelated to the original issue.

Comment 11 Seth Goldin 2021-04-30 20:41:47 UTC
I just reported this over to the Lenovo folks to give them a heads up.

Comment 12 Seth Goldin 2021-05-01 16:07:10 UTC
Firmware 0.1.72 for this ThinkPad, from LVFS, is the latest: https://fwupd.org/lvfs/devices/com.lenovo.ThinkPadN2IETXXP.firmware

Might need to wait for an update from Lenovo.

Comment 13 Seth Goldin 2021-05-05 04:36:34 UTC
Just a thought: would it matter whether the Secure Boot mode is Standard or Custom?

Comment 14 Seth Goldin 2021-05-25 03:35:31 UTC
The workaround that I have found works best was from Chris Murphy's instructions, to download the older 15-8 shim and replace the defective components of the 15.4-4 with the older, working 15-8 components: https://www.reddit.com/r/Fedora/comments/n27212/fedora_wont_boot_after_attempting_update_to_34/gwic9d4/?utm_source=reddit&utm_medium=web2x&context=3

Would be really nice to have some sort of acknowledgement that this is even being worked on. Seems like a huge issue for what's supposed to be a flagship line of laptops for Fedora.

Comment 15 Seth Goldin 2021-06-03 03:00:38 UTC
Firmware N2IET95P, version 1.73 was released today via LVFS: https://fwupd.org/lvfs/devices/com.lenovo.ThinkPadN2IETXXP.firmware

Can anyone see if this one works out of the box with shim-x64-15.4-4?

Comment 16 woat 2021-06-08 17:53:19 UTC
I can't find it now but someone on reddit said you can get Fedora 34 to boot by entering the BIOS menu then exit discarding changes.  That worked and allowed me to boot into Fedora 34 long enough to install the old version of shim*.rpm.

It took me the whole day but I now have a working computer again :)

@sethgoldin My firmwares are fully up to date and Fedora 34 still wouldn't boot with shim-x64-15.4.4

Comment 17 Javier Martinez Canillas 2021-06-23 09:31:59 UTC
*** Bug 1954245 has been marked as a duplicate of this bug. ***