Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1558793

Summary: UEFI fallback fails to boot, system resets
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: shimAssignee: Peter Jones <pjones>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: awilliam, mjg59, pbrobinson, pjones
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-11 10:16:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Paul Whalen 2018-03-21 02:59:40 UTC
Description of problem:
UEFI fallback fails to boot, system resets in a loop. 

Version-Release number of selected component (if applicable):
shim-13-5

Additional info:

From OpenQA on aarch64:

FSOpen: Open '\EFI\BOOT\BOOTAA64.EFI' Success
...
[0m[37m[40mSystem BootOrder not found.  Initializing defaults.
FSOpen: Open 'EFI' Success
FSOpen: Open 'fedora' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\fedora\BOOTAA64.CSV' Success

Reset System

BOOTAA64.CSV is a zero length file, also happens on x86_64.

Full logs:
https://openqa.stg.fedoraproject.org/tests/259978#step/_console_wait_login/22

Comment 1 Adam Williamson 2018-04-04 00:53:52 UTC
So, hum. We're basically poking about here:

https://github.com/rhboot/shim/blob/1ff4a36a23ac5c17144275ccb3e1e1061750a137/fallback.c#L1003

One thing I notice is that we hit this part:

console_print(L"Reset System\n");

and that does this:

gRT->ResetSystem(EfiResetCold, EFI_SUCCESS, 0, NULL);

note that's a *cold* reset. Exactly what that means for edk2-aarch64 I'm having fun trying to figure out, but it seems at least plausible that it does something similar to power off / power on. That's awkward for the scenario we're dealing with here: a VM with no persistent storage for EFI variables, which is expecting to boot successfully from the fallback path. I can see how it *could* lead to a loop like this - the fallback path boots, updates BootOrder, does a cold reset, BootOrder is gone again so we hit the fallback path, repeat ad infinitum...

If ever try_start_first_option() is hit and works, it'd break the loop, but clearly that's *not* happening in this case. I'm not sure if we're detecting the presence of a TPM and thus not even trying it, or trying it but it's failing (we'd need verbose messages to figure that out).

Here's a more detailed log extract:

[Bds]Booting UEFI Misc Device 3
 BlockSize : 512 
 LastBlock : 13FFFFF 
 BlockSize : 512 
 LastBlock : 1FFFFF 
 BlockSize : 512 
 LastBlock : 119B7FF 
FSOpen: Open '\EFI\BOOT\BOOTAA64.EFI' Success
[Bds] DevicePath expand: PciRoot(0x0)/Pci(0x5,0x0) -> PciRoot(0x0)/Pci(0x5,0x0)/HD(1,MBR,0x088DD077,0x800,0x64000)/\EFI\BOOT\BOOTAA64.EFI
[Security] 3rd party image[0] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x5,0x0)/HD(1,MBR,0x088DD077,0x800,0x64000)/\EFI\BOOT\BOOTAA64.EFI.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B FABC9040
Loading driver at 0x000F8501000 EntryPoint=0x000F8501120
Loading driver at 0x000F8501000 EntryPoint=0x000F8501120 
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF FABC9E98
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
[0m[37m[40mSystem BootOrder not found.  Initializing defaults.
FSOpen: Open 'EFI' Success
FSOpen: Open 'fedora' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\fedora\BOOTAA64.CSV' Success

Reset System

(then the whole thing happens again, over and over, till the test gives up and dies).

Comment 2 Adam Williamson 2018-04-04 00:54:55 UTC
For now I'm gonna see if I can possibly tweak the openQA bits such that we run these tests *with* persistent storage for EFI vars, at least across the lifetime of each test.

Comment 3 Adam Williamson 2018-04-04 18:09:15 UTC
Something else I forgot last night: this works on F28. It only fails on Rawhide.

Here's the logs from an F28 test:

[Bds]Booting UEFI Misc Device 3
 BlockSize : 512 
 LastBlock : 13FFFFF 
 BlockSize : 512 
 LastBlock : 1FFFFF 
 BlockSize : 512 
 LastBlock : 119B7FF 
FSOpen: Open '\EFI\BOOT\BOOTAA64.EFI' Success
[Bds] DevicePath expand: PciRoot(0x0)/Pci(0x5,0x0) -> PciRoot(0x0)/Pci(0x5,0x0)/HD(1,MBR,0x69E22731,0x800,0x64000)/\EFI\BOOT\BOOTAA64.EFI
[Security] 3rd party image[0] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x5,0x0)/HD(1,MBR,0x69E22731,0x800,0x64000)/\EFI\BOOT\BOOTAA64.EFI.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B FABC9040
Loading driver at 0x000F84FB000 EntryPoint=0x000F84FB120
Loading driver at 0x000F84FB000 EntryPoint=0x000F84FB120 
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF FABC9E98
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
[0m[37m[40mSystem BootOrder not found.  Initializing defaults.
FSOpen: Open 'EFI' Success
FSOpen: Open 'fedora' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\fedora\BOOTAA64.CSV' Success
Creating boot entry "Boot0001" with label "Fedora" for file "\EFI\fedora\shimaa64.efi"

FSOpen: Open '\EFI\fedora\shimaa64.efi' Success
[Security] 3rd party image[0] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x5,0x0)/HD(1,MBR,0x69E22731,0x800,0x64000)/\EFI\fedora\shimaa64.efi.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B F9260040
Loading driver at 0x000F8422000 EntryPoint=0x000F8422120
Loading driver at 0x000F8422000 EntryPoint=0x000F8422120 
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF F9260E98
FSOpen: Open '\EFI\fedora\grubaa64.efi' Success
[0m[30m[40m[2J[01;01H[0m[37m[40m[21;07HUse the ^ and v keys to change the selection.  

the difference is obvious: after we get to "System BootOrder not found.  Initializing defaults.", we actually log creating a new entry, and then immediately boot it (without the "Reset System" happening, so presumably `try_start_first_option()` ran and worked). For some reason, that's working in F28 but failing in Rawhide.

F28 has shim-aa64-13-4.aarch64 , Rawhide has shim-aa64-13-5.aarch64 . The former I think comes from https://koji.fedoraproject.org/koji/buildinfo?buildID=1054631 - shim-signed-13-4 - the latter from https://koji.fedoraproject.org/koji/buildinfo?buildID=1054372 - shim-13-5 . Both those seem to use the same shimaa64.efi source file (the SHA512 is the same in the `sources` file in both builds), but the BOOTAA64.EFI file winds up being a slightly different size between the two, not sure why.

Comment 4 Adam Williamson 2018-04-04 19:01:44 UTC
pwhalen actually noted what looks like the cause of this, which I missed: in shim-13-5 , BOOTAA64.CSV and BOOTIA32.CSV and BOOTX64.CSV are all 0-length. In shim-signed-13-4 they are not. This can also be seen in the package git repos, of course ('master' branch of shim has zero-length files, 'f28' branch of shim-signed does not). Looks like an easy fix to just put the right files back in the shim package build, but of course only a few people can actually build it, so we need one of them to do it.

Comment 5 Paul Whalen 2018-05-03 16:58:06 UTC
System BootOrder not found.  Initializing defaults.
FSOpen: Open 'EFI' Success
FSOpen: Open 'fedora' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\fedora\BOOTAA64.CSV' Success
Creating boot entry "Boot0001" with label "Fedora" for file "\EFI\fedora\shimaa64.efi"

FSOpen: Open '\EFI\fedora\shimaa64.efi' Success


shim-signed-15-2 looks to fix the issue, could we get this tagged into f29?

Comment 6 Paul Whalen 2018-05-04 18:00:48 UTC
Wrong package- shim-15-3 also fixes this and is indeed tagged into f29. Thanks pjones!