Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1732612 - Bootloader install fails on ppc64le since Fedora-Rawhide-20190719.n.1
Summary: Bootloader install fails on ppc64le since Fedora-Rawhide-20190719.n.1
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: rawhide
Hardware: ppc64le
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa
Depends On:
Blocks: PPCTracker
TreeView+ depends on / blocked
 
Reported: 2019-07-23 21:36 UTC by Adam Williamson
Modified: 2019-08-05 16:53 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-05 16:53:22 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Adam Williamson 2019-07-23 21:36:18 UTC 
Since Fedora-Rawhide-20190719.n.1 compose, all openQA ppc64le install tests fail during bootloader installation. The install logs show several errors:

19:57:28,686 INF program: Running... nvram --print-config=boot-device
19:57:28,735 INF program: nvram: ERROR: cannot open "/dev/nvram": No such file or directory
19:57:28,736 INF program: nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory
19:57:28,737 DBG program: Return code: 255
19:57:28,739 INF program: Running... ofpathname /dev/vda1
19:57:29,401 INF program: /pci@800000020000000/scsi@7
19:57:29,402 DBG program: Return code: 0
19:57:29,404 INF program: Running... nvram --update-config boot-device=/pci@800000020000000/scsi@7 nvram: ERROR: cannot open /dev/nvram: No such file or directory nvram: ERROR: cannot open /dev/misc/nvram: No such file or directory
19:57:29,428 INF program: nvram: ERROR: cannot open "/dev/nvram": No such file or directory
19:57:29,430 INF program: nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory
19:57:29,430 DBG program: Return code: 255
19:57:29,433 INF program: Running in chroot '/mnt/sysroot'... grub2-install --no-nvram --no-floppy /dev/vda1
19:57:29,738 INF program: Installing for powerpc-ieee1275 platform.
19:57:29,739 INF program: grub2-install: error: cannot find a GRUB drive for /dev/vda2.  Check your device.map.
19:57:29,740 DBG program: Return code: 1
19:57:29,745 INF program: Running in chroot '/mnt/sysroot'... grub2-set-default 991710e087d24d7c8c54ce81d5f0e159-5.3.0-0.rc0.git4.1.fc31.ppc64le
19:57:29,799 DBG program: Return code: 0
19:57:29,801 INF program: Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
19:57:30,827 INF program: Generating grub configuration file ...
19:57:30,830 INF program: /usr/sbin/grub2-probe: error: cannot find a GRUB drive for /dev/vda2.  Check your device.map.
19:57:30,830 DBG program: Return code: 1

It seems the issue is to do with nvram emulation or support somehow, but I'm not sure how. The bug may well not actually be in grub2, but I'm just not sure what else is a likely suspect; the kernel did not change in the compose, though glibc did, perhaps that could be involved. I'm going to CC folks who may be able to investigate this more.
Comment 1 Adam Williamson 2019-07-23 21:40:51 UTC 
Note, here's how the same period of logs looks in a successful install from a previous compose:

22:24:09,546 INF program: Running... nvram --print-config=boot-device
22:24:09,572 INF program: nvram: ERROR: cannot open "/dev/nvram": No such file or directory
22:24:09,573 INF program: nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory
22:24:09,574 DBG program: Return code: 255
22:24:09,575 INF program: Running... ofpathname /dev/vda1
22:24:10,060 INF program: /pci@800000020000000/scsi@7
22:24:10,061 DBG program: Return code: 0
22:24:10,062 INF program: Running... nvram --update-config boot-device=/pci@800000020000000/scsi@7 nvram: ERROR: cannot open /dev/nvram: No such file or directory nvram: ERROR: cannot open /dev/misc/nvram: No such file or directory
22:24:10,076 INF program: nvram: ERROR: cannot open "/dev/nvram": No such file or directory
22:24:10,078 INF program: nvram: ERROR: cannot open "/dev/misc/nvram": No such file or directory
22:24:10,081 DBG program: Return code: 255
22:24:10,083 INF program: Running in chroot '/mnt/sysroot'... grub2-install --no-nvram --no-floppy /dev/vda1
22:24:10,620 INF program: Installing for powerpc-ieee1275 platform.
22:24:10,622 INF program: Installation finished. No error reported.
22:24:10,624 DBG program: Return code: 0
22:24:10,652 INF program: Running... xfs_freeze -f /boot
22:24:10,696 DBG program: Return code: 0
22:24:10,699 INF program: Running... xfs_freeze -u /boot
22:24:10,722 DBG program: Return code: 0
22:24:10,727 INF program: Running in chroot '/mnt/sysroot'... grub2-set-default de505b69040c4b45bc9d5bc9b0f271ec-5.3.0-0.rc0.git4.1.fc31.ppc64le
22:24:10,768 DBG program: Return code: 0
22:24:10,770 INF program: Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
22:24:15,908 INF program: Generating grub configuration file ...
22:24:15,909 INF program: Generating boot entries from BLS files...
22:24:15,910 INF program: done
22:24:15,910 DBG program: Return code: 0

so the nvram errors are 'normal' and aren't the problem, probably. It's the grub2-install and grub2-mkconfig failures  that are new.
Comment 2 Michel Normand 2019-07-25 09:53:49 UTC 
FYIO, trying to investigate this problem with last compose 20190724 I raised another failure tracked by
Comment 3 Dan Horák 2019-07-25 10:11:22 UTC 
It could be a change coming with 5.3 kernel and the configs, seems we have
# CONFIG_NVRAM is not set
now (https://src.fedoraproject.org/rpms/kernel/blob/master/f/kernel-ppc64le.config#_3611)
Comment 4 Dan Horák 2019-07-25 10:18:45 UTC 
because NVRAM is disabled by default in https://src.fedoraproject.org/rpms/kernel/blob/master/f/configs/fedora/generic/CONFIG_NVRAM
And because PowerNV doesn't need nvram, I haven't noticed it even when running kernel-5.3.0-0.rc1.git0.1.fc31.op.1.ppc64le :-)
Comment 5 Dan Horák 2019-07-25 10:26:35 UTC 
or maybe it's not nvram's fault ...
Comment 6 Dan Horák 2019-07-25 11:22:30 UTC 
Better set back to grub, it might be in the rebase/cleanup in grub2-2.02-92.fc31
Comment 7 Menanteau Guy 2019-07-25 12:16:37 UTC 
Note that on an VM installed with Fedora-Rawhide-20190717.n.0 compose, changing /etc/default/grub and running "grub2-mkconfig -o /boot/grub2/grub.cfg" works fine with grub2 tools 2.02-92.fc31.ppc64le.rpm but it fails with on "/usr/sbin/grub2-probe: error: cannot find a GRUB drive for /dev/sda2.  Check your device.map." with grub2 tools updated to 2.02-94.fc31.src.rpm on same VM.
Comment 8 Adam Williamson 2019-07-25 14:56:37 UTC 
Yeah, sorry for mentioning the nvram stuff in #c0, but as #c1 says, it's almost certainly not the problem as we get those errors in the logs from a working install too. It's the grub2-install and grub2-mkconfig failures that are new.
Comment 9 Adam Williamson 2019-07-25 14:58:57 UTC 
BTW, one thing I considered is whether the rearrangement in a1dedc8a10fcb7bda743e4dc8470fefb6f6801df meant only grub2-tools-minimal was now included in the installer images but ppc64le needs grub2-tools, or something...but according to the logs, that's not it, both working and broken composes have both grub2-tools and grub2-tools-minimal in the image.
Comment 10 Dan Horák 2019-07-26 09:10:25 UTC 
(In reply to Adam Williamson from comment #8)
> Yeah, sorry for mentioning the nvram stuff in #c0, but as #c1 says, it's
> almost certainly not the problem as we get those errors in the logs from a
> working install too. It's the grub2-install and grub2-mkconfig failures that
> are new.

yes, I have focused too much on the nvram issue :-) Anyway, the NVRAM driver should be enabled, patch in https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org/message/HMZPWGSW5LU6ITH647IGTOELN2XGTIL5/
Comment 11 Dan Horák 2019-07-26 15:17:21 UTC 
So I've replicated the failure locally (KVM guest), device.map looks correct

[anaconda root@fedora-ppc grub2]# cat device.map 
# this device map was generated by anaconda
(hd0)      /dev/vda

and a verbose run of grub2-install from the new system's chroot is

[anaconda root@fedora-ppc /]# grub2-install -v --no-nvram --no-floppy /dev/vda1
Installing for powerpc-ieee1275 platform.
grub2-install: info: adding `hd0' -> `/dev/vda' from device.map.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/minix2_be.mod' -> `/boot/grub2/powerpc-ieee1275/minix2_be.mod'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/sfs.mod' -> `/boot/grub2/powerpc-ieee1275/sfs.mod'.
...
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/div.mod' -> `/boot/grub2/powerpc-ieee1275/div.mod'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/ofnet.mod' -> `/boot/grub2/powerpc-ieee1275/ofnet.mod'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/gcry_seed.mod' -> `/boot/grub2/powerpc-ieee1275/gcry_seed.mod'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/cmp_test.mod' -> `/boot/grub2/powerpc-ieee1275/cmp_test.mod'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/efiemu32.o' -> `/boot/grub2/powerpc-ieee1275/efiemu32.o'.
grub2-install: info: cannot open `/usr/lib/grub/powerpc-ieee1275/efiemu32.o': No such file or directory.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/efiemu64.o' -> `/boot/grub2/powerpc-ieee1275/efiemu64.o'.
grub2-install: info: cannot open `/usr/lib/grub/powerpc-ieee1275/efiemu64.o': No such file or directory.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/moddep.lst' -> `/boot/grub2/powerpc-ieee1275/moddep.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/command.lst' -> `/boot/grub2/powerpc-ieee1275/command.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/fs.lst' -> `/boot/grub2/powerpc-ieee1275/fs.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/partmap.lst' -> `/boot/grub2/powerpc-ieee1275/partmap.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/parttool.lst' -> `/boot/grub2/powerpc-ieee1275/parttool.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/video.lst' -> `/boot/grub2/powerpc-ieee1275/video.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/crypto.lst' -> `/boot/grub2/powerpc-ieee1275/crypto.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/terminal.lst' -> `/boot/grub2/powerpc-ieee1275/terminal.lst'.
grub2-install: info: copying `/usr/lib/grub/powerpc-ieee1275/modinfo.sh' -> `/boot/grub2/powerpc-ieee1275/modinfo.sh'.
grub2-install: info: copying `/usr/share/grub/unicode.pf2' -> `/boot/grub2/fonts/unicode.pf2'.
grub2-install: info: /dev/vda2 is present.
grub2-install: info: Looking for /dev/vda2.
grub2-install: info: /dev/vda is a parent of /dev/vda2.
grub2-install: info: /dev/vda2 starts from 10240.
grub2-install: info: opening the device hd0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
grub2-install: info: Scanning for DISKFILTER devices on disk hd0.
grub2-install: info: Scanning for mdraid1x devices on disk hd0.
grub2-install: info: Scanning for mdraid09_be devices on disk hd0.
grub2-install: info: Scanning for mdraid09 devices on disk hd0.
grub2-install: info: Scanning for dmraid_nv devices on disk hd0.
grub2-install: info: Scanning for ldm devices on disk hd0.
grub2-install: info: scanning hd0 for LDM.
grub2-install: info: no LDM signature found.
grub2-install: info: Scanning for lvm devices on disk hd0.
grub2-install: info: cannot find the partition of `/dev/vda2'.
grub2-install: info: /dev/vda2 is present.
grub2-install: info: Looking for /dev/vda2.
grub2-install: info: /dev/vda is a parent of /dev/vda2.
grub2-install: info: /dev/vda2 is present.
grub2-install: info: Looking for /dev/vda2.
grub2-install: info: /dev/vda is a parent of /dev/vda2.
grub2-install: error: cannot find a GRUB drive for /dev/vda2.  Check your device.map.
Comment 12 Dan Horák 2019-07-26 15:39:22 UTC 
and it's a change between grub2-2.02-92.fc31 (works) and grub2-2.02-93.fc31 (fails) to whose I've downgraded grub2 from Fedora-Rawhide-20190726.n.0 installation
Comment 13 Dan Horák 2019-07-26 16:44:31 UTC 
and grub2-2.02-92.fc31 rebuilt in current buildroot works too

The difference between good and bad is in

...
grub2-install: info: /dev/vda2 is present.
grub2-install: info: Looking for /dev/vda2.
grub2-install: info: /dev/vda is a parent of /dev/vda2.
grub2-install: info: /dev/vda2 starts from 10240.
grub2-install: info: opening the device hd0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
...

while the good run has 

...
grub2-install: info: /dev/vda2 is present.
grub2-install: info: Looking for /dev/vda2.
grub2-install: info: /dev/vda is a parent of /dev/vda2.
grub2-install: info: /dev/vda2 starts from 10240.
grub2-install: info: opening the device hd0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 20971520.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 20971520.
...

See the hd0 site info.
Comment 14 Javier Martinez Canillas 2019-08-02 16:13:12 UTC 
Hello Dan,

(In reply to Dan Horák from comment #13)
> and grub2-2.02-92.fc31 rebuilt in current buildroot works too
> 
> The difference between good and bad is in
> 
> ...
> grub2-install: info: /dev/vda2 is present.
> grub2-install: info: Looking for /dev/vda2.
> grub2-install: info: /dev/vda is a parent of /dev/vda2.
> grub2-install: info: /dev/vda2 starts from 10240.
> grub2-install: info: opening the device hd0.
> grub2-install: info: drive = 0.
> grub2-install: info: the size of hd0 is 0.
> grub2-install: info: drive = 0.
> grub2-install: info: the size of hd0 is 0.
> ...
> 
> while the good run has 
> 
> ...
> grub2-install: info: /dev/vda2 is present.
> grub2-install: info: Looking for /dev/vda2.
> grub2-install: info: /dev/vda is a parent of /dev/vda2.
> grub2-install: info: /dev/vda2 starts from 10240.
> grub2-install: info: opening the device hd0.
> grub2-install: info: drive = 0.
> grub2-install: info: the size of hd0 is 20971520.
> grub2-install: info: drive = 0.
> grub2-install: info: the size of hd0 is 20971520.
> ...
> 
> See the hd0 site info.

Indeed. That's strange since the changes in the package between the working (grub2-2.02-92.fc31) and failing (grub2-2.02-93.fc31) versions are not related to grub2-install at all. Maybe this is an incompatibility between the grub2-install binary and one of its linked libraries?

I noticed that there was a mass rebuild for Fedora 31 and the grub2 package build failed due releng not having proper credentials:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1326365

I did a manual build to make sure that the package is rebuilt as a part of the mass rebuild and this version (grub2-2.02-96.fc31) works correctly to me:

$ rpm -qa | grep grub2
grub2-tools-extra-2.02-94.fc31.ppc64le
grub2-ppc64le-modules-2.02-94.fc31.noarch
grub2-tools-2.02-94.fc31.ppc64le
grub2-common-2.02-94.fc31.noarch
grub2-ppc64le-2.02-94.fc31.ppc64le

$ grub2-install -v --no-nvram --no-floppy /dev/sda1
...
grub2-install: info: /dev/sda2 is present.
grub2-install: info: Looking for /dev/sda2.
grub2-install: info: /dev/sda is a parent of /dev/sda2.
grub2-install: info: /dev/sda2 starts from 10240.
grub2-install: info: opening the device hd0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
grub2-install: info: drive = 0.
grub2-install: info: the size of hd0 is 0.
...
grub2-install: error: cannot find a GRUB drive for /dev/sda2.  Check your device.map.

$ rpm -qa | grep grub2
grub2-tools-extra-2.02-96.fc31.ppc64le
grub2-tools-minimal-2.02-96.fc31.ppc64le
grub2-ppc64le-modules-2.02-96.fc31.noarch
grub2-ppc64le-2.02-96.fc31.ppc64le
grub2-common-2.02-96.fc31.noarch
grub2-tools-2.02-96.fc31.ppc64le

$ grub2-install -v --no-nvram --no-floppy /dev/sda1
...
grub2-install: info: /dev/sda1 is present.                                                                                                                                                                                                    
grub2-install: info: Looking for /dev/sda1.                                                                            
grub2-install: info: /dev/sda is a parent of /dev/sda1.                                                                
grub2-install: info: /dev/sda1 starts from 2048.          
grub2-install: info: opening the device hd0.
grub2-install: info: drive = 0.             
grub2-install: info: the size of hd0 is 26214400.         
grub2-install: info: drive = 0.             
grub2-install: info: the size of hd0 is 26214400.
...
Installation finished. No error reported.
Comment 15 Adam Williamson 2019-08-02 17:00:42 UTC 
We'll keep an eye on openQA and see if it goes back to working in the next successful compose, then. Thanks!
Comment 16 Dan Horák 2019-08-05 09:15:24 UTC 
I did a manual test from Fedora-Rawhide-20190804.n.0 and things seems to work again

from /tmp/program.log
...
11:10:56,677 INF program: Running... nvram --print-config=boot-device
11:10:56,810 INF program: 
11:10:56,811 DBG program: Return code: 0
11:10:56,812 INF program: Running... ofpathname /dev/vda1
11:10:57,331 INF program: /pci@800000020000000/scsi@5
11:10:57,332 DBG program: Return code: 0
11:10:57,333 INF program: Running... nvram --update-config boot-device=/pci@800000020000000/scsi@5
11:10:57,443 DBG program: Return code: 0
11:10:57,444 INF program: Running in chroot '/mnt/sysroot'... grub2-install --no-nvram --no-floppy /dev/vda1
11:11:06,341 INF program: Installing for powerpc-ieee1275 platform.
11:11:06,342 INF program: Installation finished. No error reported.
11:11:06,342 DBG program: Return code: 0
11:11:06,431 INF program: Running in chroot '/mnt/sysroot'... grub2-set-default 948cfd0ae0ee4bd998f7be2f8a2dc22d-5.3.0-0.rc2.git4.1.fc31.ppc64le
11:11:06,619 DBG program: Return code: 0
11:11:06,622 INF program: Running in chroot '/mnt/sysroot'... grub2-mkconfig -o /boot/grub2/grub.cfg
11:11:13,871 INF program: Generating grub configuration file ...
11:11:13,873 INF program: Generating boot entries from BLS files...
11:11:13,874 INF program: done
11:11:13,874 DBG program: Return code: 0
11:11:21,892 INF program: Running in chroot '/mnt/sysroot'... depmod -a 5.3.0-0.rc2.git4.1.fc31.ppc64le
11:11:29,160 DBG program: Return code: 0
11:11:29,161 INF program: Running in chroot '/mnt/sysroot'... dracut -f /boot/initramfs-5.3.0-0.rc2.git4.1.fc31.ppc64le.img 5.3.0-0.rc2.git4.1.fc31.ppc64le
11:11:52,703 DBG program: Return code: 0
11:11:52,715 INF program: Running in chroot '/mnt/sysroot'... /bin/sh /tmp/ks-script-xfghxjx3
11:11:53,448 DBG program: Return code: 0
11:11:53,450 INF program: Running... /bin/sh /tmp/ks-script-axijc95t
11:11:53,858 DBG program: Return code: 0
11:11:53,861 INF program: Running... /bin/sh /tmp/ks-script-didv2f1_
11:11:53,883 DBG program: Return code: 0
Comment 17 Michel Normand 2019-08-05 09:27:22 UTC 
Also confirmed by openQA results with compose 20190802/04
https://openqa.stg.fedoraproject.org/group_overview/3
Comment 18 Adam Williamson 2019-08-05 16:53:22 UTC 
Then we can probably close this, thanks.
Note You need to log in before you can comment on or make changes to this bug.