Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1615969

Summary: AArch64 - Error when booting "error: out of memory."
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: grub2Assignee: Peter Jones <pjones>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: awilliam, fzatlouk, gmarr, kevin, lkundrak, marc.c.dionne, pbrobinson, pjones, robatino
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: grub2-2.02-52.fc29 grub2-2.02-57.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 02:54:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1517011    
Attachments:
Description Flags
boot msgs none

Description Paul Whalen 2018-08-14 15:52:02 UTC
Description of problem:
When attempting to boot with grub2-2.02-47.fc29 on aarch64, the boot displays an error "Error: Out of memory.", then a kernel panic. 

Version-Release number of selected component (if applicable):
grub2-2.02-47.fc29

How reproducible:
Every time.

Additional info:

error: out of memory.

Press any key to continue...
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.0-1.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Mon Aug 13 15:54:13 UTC 2018
[    0.000000] Machine model: AMD Seattle (Rev.B0) Development Board (Overdrive)
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi:  ACPI 2.0=0x83ff1c3000  SMBIOS 3.0=0x83ff347798 
[    0.000000] cma: Reserved 64 MiB at 0x00000080fc000000

..

[    2.234658] md: autorun ...
[    2.237448] md: ... autorun DONE.
[    2.240804] VFS: Cannot open root device "mapper/fedora_seattle-root" or unknown-block(0,0): error -6
[    2.250016] Please append a correct "root=" boot option; here are the available partitions:
[    2.258360] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.266612] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G        W         4.18.0-1.fc29.aarch64 #1
[    2.275210] Hardware name: AMD Overdrive/Supercharger/Default string, BIOS ROD1002C 04/08/2016
[    2.283807] Call trace:
[    2.286245]  dump_backtrace+0x0/0x150
[    2.289895]  show_stack+0x24/0x30
[    2.293200]  dump_stack+0x90/0xb4
[    2.296503]  panic+0x114/0x278
[    2.299548]  mount_block_root+0x228/0x2c8
[    2.303545]  mount_root+0x84/0x90
[    2.306847]  prepare_namespace+0x134/0x178
[    2.310930]  kernel_init_freeable+0x318/0x33c
[    2.315275]  kernel_init+0x18/0x114
[    2.318752]  ret_from_fork+0x10/0x18
[    2.322318] SMP: stopping secondary CPUs
[    2.326233] Kernel Offset: disabled
[    2.329709] CPU features: 0x21806082
[    2.333271] Memory Limit: none
[    2.336316] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Comment 1 Paul Whalen 2018-08-14 15:55:24 UTC
Created attachment 1475897 [details]
boot msgs

Comment 2 Paul Whalen 2018-08-20 20:07:02 UTC
Testing with grub2-2.02-50.fc29:

loader/arm64/linux.c:62: UEFI stub kernel:
loader/arm64/linux.c:63: PE/COFF header @ 00000040
loader/arm64/linux.c:351: kernel file size: 24844800
loader/arm64/linux.c:354: kernel numpages: 6066
loader/arm64/linux.c:370: kernel @ 0x83fb020000
loader/efi/linux.c:44: shim_lock: 0x83fce316c8
loader/efi/linux.c:51: Asking shim to verify kernel signature
loader/efi/linux.c:53: shim_lock->verify(): 0
loader/efi/linux.c:56: Kernel signature verification passed
loader/arm64/linux.c:293: Loading initrd
error: out of memory.

Press any key to continue...
kern/efi/fdt.c:38: found registered FDT @ 0x801fffc000
loader/efi/fdt.c:64: allocating 13575 bytes for fdt
loader/arm64/linux.c:144: Installed/updated FDT configuration table @
0x83fb01c000
loader/arm64/linux.c:200: linux command line:
'BOOT_IMAGE=(tftp)/fedora/Fedora-29-20180817.n.0/Everything/vmlinuz
inst.repo=https://kojipkgs.fedoraproject.org/compose/branched//Fedora-29-201808
17.n.0/compose/Everything/aarch64/os/'
loader/efi/linux.c:83: kernel_addr: 0x83fb020000 handover_offset: 0x1036004
params: 0x83fb020000
loader/efi/linux.c:85: handover_func() = 0x83fc056004
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.1-300.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Thu Aug 16 16:53:57 UTC 2018

Comment 3 Marc Dionne 2018-08-28 18:09:11 UTC
Any more insights on this problem?

I'm seeing roughly the same thing when trying to boot an OverDrive 1000:

error: out of memory.

Press any key to continue...
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.5-300.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Fri Aug 24 17:18:34 UTC 2018
[    0.000000] Machine model: SoftIron Overdrive 1000 (AMD Seattle (Rev.B1))
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by SoftIron Overdrive 1000

.....

[    1.924078] VFS: Cannot open root device "UUID=de35480d-3e54-4296-b171-77876ce9c976" or unknown-block(0,0): error -6
[    1.934599] Please append a correct "root=" boot option; here are the available partitions:
[    1.942949] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Comment 4 Paul Whalen 2018-08-28 20:16:22 UTC
Proposing as a blocker for F29 Beta, this breaks aarch64 installs.

Comment 5 František Zatloukal 2018-08-29 14:26:51 UTC
(In reply to Paul Whalen from comment #4)
> Proposing as a blocker for F29 Beta, this breaks aarch64 installs.

Paul, if I didn't miss anything, aarch64 is not a primary architecture and I guess this won't be accepted as a blocker because of that. 

https://fedoraproject.org/wiki/Architectures

Comment 6 Peter Robinson 2018-08-29 14:29:17 UTC
> Paul, if I didn't miss anything, aarch64 is not a primary architecture and I
> guess this won't be accepted as a blocker because of that. 

ARM64 for Server is primary architecture and hence blocking as of F-28

Comment 7 Kevin Fenzi 2018-08-29 23:09:12 UTC
+1 blocker.

Comment 8 Geoffrey Marr 2018-08-30 14:44:18 UTC
+1 blocker

Comment 9 Geoffrey Marr 2018-08-30 14:45:10 UTC
Voted as a blocker in-bug:

The decision to classify this bug as an "AcceptedBlocker" (Beta) was made as it violates the following criteria:

"The installer must run when launched normally from the release-blocking images."

Comment 10 Peter Robinson 2018-08-30 16:18:56 UTC
For reference grub2-2.02-52.fc29 fixes this for me. Will be on it's way to f29 updates-testing shortly

Comment 11 Paul Whalen 2018-08-30 17:05:19 UTC
Verified fixed in grub2-2.02-52.fc29

Comment 12 Fedora Update System 2018-08-30 17:06:38 UTC
grub2-2.02-52.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 13 Marc Dionne 2018-08-30 19:09:17 UTC
FYI grub2-2.02-52.fc29 also fixes the boot problem for my aarch64 box.

Comment 14 Fedora Update System 2018-08-31 16:22:27 UTC
grub2-2.02-52.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 15 Fedora Update System 2018-09-01 01:49:40 UTC
grub2-2.02-52.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 16 Fedora Update System 2018-09-02 02:56:45 UTC
grub2-2.02-52.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 17 František Zatloukal 2018-09-06 09:47:21 UTC
Switching to VERIFIED as per Comment 11 .

Comment 18 Peter Robinson 2018-09-06 09:56:56 UTC
(In reply to František Zatloukal from comment #17)
> Switching to VERIFIED as per Comment 11 .

The update has been dropped due to issues with x86

Comment 19 Fedora Update System 2018-09-08 04:07:28 UTC
grub2-2.02-54.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 20 František Zatloukal 2018-09-08 11:16:35 UTC
grub2-2.02-54.fc29 worked fine on x86_64 UEFI (VM).

Can't test aarch64 though...

Comment 21 Fedora Update System 2018-09-08 16:11:21 UTC
grub2-2.02-54.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 22 Adam Williamson 2018-09-10 19:37:08 UTC
I did a couple of scratch builds of different possibilities to fix this while not regressing x86_64 UEFI. If folks with affected aarch64 systems could try them, it'd be great. Here they are:

* https://koji.fedoraproject.org/koji/taskinfo?taskID=29603292
* https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437

The first contains all the changes from -52, plus amartinez's fix for the x86_64 UEFI boot issue instead of Hans' that we shipped in -54.

The second contains only the bits from -51, plus the change that was intended to fix *this* bug (and one other that is a precursor for it). Other changes were dropped.

It'd be useful to know how both of these work compared to -51 and to -54. Thanks!

Comment 23 Marc Dionne 2018-09-10 20:14:08 UTC
(In reply to Adam Williamson from comment #22)
> I did a couple of scratch builds of different possibilities to fix this
> while not regressing x86_64 UEFI. If folks with affected aarch64 systems
> could try them, it'd be great. Here they are:
> 
> * https://koji.fedoraproject.org/koji/taskinfo?taskID=29603292
> * https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437
> 
> The first contains all the changes from -52, plus amartinez's fix for the
> x86_64 UEFI boot issue instead of Hans' that we shipped in -54.
> 
> The second contains only the bits from -51, plus the change that was
> intended to fix *this* bug (and one other that is a precursor for it). Other
> changes were dropped.
> 
> It'd be useful to know how both of these work compared to -51 and to -54.
> Thanks!

Both of those boot fine for me on my aarch64 box.  For reference, -51 did not boot, -54 was ok.

Comment 24 Adam Williamson 2018-09-11 01:45:49 UTC
Thanks a lot. Can you also test pjones' -55 build?

https://koji.fedoraproject.org/koji/buildinfo?buildID=1143536

Thanks again!

Comment 25 Paul Whalen 2018-09-11 02:58:37 UTC
grub2-2.02-55.fc29 working on mustang, seattle

Comment 26 Marc Dionne 2018-09-11 11:35:06 UTC
2.02-55.fc29.aarch64 looks fine here

Comment 27 Adam Williamson 2018-09-11 15:04:17 UTC
pwhalen: does the 'minimal' build - https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437 - work OK on your aarch64 systems? Thanks!

Comment 28 Marc Dionne 2018-09-11 16:18:27 UTC
(In reply to Adam Williamson from comment #27)
> pwhalen: does the 'minimal' build -
> https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437 - work OK on
> your aarch64 systems? Thanks!

It does here, as I reported in comment #23

Comment 29 Adam Williamson 2018-09-11 17:46:28 UTC
Marc: I know *you* reported success, I was looking for results from pwhalen as well :) Multiple tests is always good for safety.

Comment 30 Marc Dionne 2018-09-11 17:53:38 UTC
Adam, sorry, I interpreted the label incorrectly, thought you meant pwhalen was asking the question...  :)

Comment 31 Adam Williamson 2018-09-11 17:57:18 UTC
No problem :)

Sorry to be a pain, can you both also please test -56:

https://koji.fedoraproject.org/koji/taskinfo?taskID=29618610

and check that still works? Thanks!

Comment 32 Marc Dionne 2018-09-11 18:12:32 UTC
-56 works fine for me

Comment 33 Paul Whalen 2018-09-11 19:59:44 UTC
adamw : 2.02-54.2.minimal working on the seattle 

2.02-56.fc29 working on seattle and rpi3

Comment 34 Adam Williamson 2018-09-11 20:09:33 UTC
Thanks a lot guys!

Comment 35 Fedora Update System 2018-09-11 22:57:18 UTC
grub2-2.02-54.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 36 Fedora Update System 2018-09-12 02:54:03 UTC
grub2-2.02-57.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.