*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.14.4-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

I don't have ISO with 3.14.4-200.fc20 kernel, but I tried ISO with 3.14.3-200.fc20 kernel, the output is below. As you can see, newer kernel is unable to pass even on earlier boot stage, compared to old the boot/kernel.

  Booting `Install Fedora 20 (64-bit kernel)'

OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.14.3-200.fc20.ppc64 (mockbuild.fedoraproject.org) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7) (GCC) ) #1 SMP Tue May 6 15:51:15 MST 2014
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 512 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/ppc/ppc64/vmlinuz ro
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000006f80000
  alloc_top    : 0000000008000000
  alloc_top_hi : 0000000008000000
  rmo_top      : 0000000008000000
  ram_top      : 0000000008000000
found display   : /pci@800000020000202/display@1, opening... done
Could not allocate memory for RTAS
EXIT called ok
0 >

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs.

Fedora 20 has now been rebased to 3.17.2-200.fc20.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21.

If you experience different issues, please open a new bug report for those.

please provide URL to download installation ISO with newer mentioned kernel 3.17.2-200.fc200. Thanks.

Unfortunately there is no way to spin new ISOs, but the Fedora 21 beta installer should have a new kernel for you to try. Please test and let us know the results. If it is working, we can close this bug as resolved. If it is not working, we can reassign this bug to F21 where it might get fixed before release.

Welcome to the Fedora-Server 21 installer!


OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.17.4-301.fc21.ppc64 (mockbuild.fedoraproject.org) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #1 SMP Mon Dec 1 07:50:54 UTC 2014
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 512 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/ppc/ppc64/vmlinuz ro
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000007420000
  alloc_top    : 0000000008000000
  alloc_top_hi : 0000000008000000
  rmo_top      : 0000000008000000
  ram_top      : 0000000008000000
found display   : /pci@800000020000202/display@1, opening... done
Could not allocate memory for RTAS
EXIT called ok
0 >

if somebody else noticed, "alloc_bottom" is changing with every kernel version and increasing:

alloc_bottom : 0000000004330000 (3.9.5-301.fc19.ppc64)
alloc_bottom : 0000000006e80000 (3.11.10-301.fc20.ppc64)
alloc_bottom : 0000000006f80000 (3.14.3-200.fc20.ppc64)
alloc_bottom : 0000000007420000 (3.17.4-301.fc21.ppc64)

could it be the reason of ENOMEM for RTAS (first) and flatten_device_tree (second)

Installed debian 7 for ppc successfully (kernel 3.2.0-4-powerpc64) - rebooted, works. Updated to debian-testing (kernel 3.16.0-4-powerpc64) - rebooted, works as well. Cut of boot log:

boot: Linux
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 03080000, size: 15564 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.16.0-4-powerpc64 (debian-kernel.org) (gcc version 4.8.3 (Debian 4.8.3-16) ) #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 16 (NR_CPUS = 32)
Calling ibm,client-architecture-support... done
command line: root=UUID=7c27358c-21f3-49ae-ae2f-bad559ecbbb2 ro
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000003fc0000
  alloc_top    : 0000000008000000
  alloc_top_hi : 0000000008000000
  rmo_top      : 0000000008000000
  ram_top      : 0000000008000000
found display   : /pci@800000020000202/display@1, opening... done
instantiating rtas at 0x0000000006eb0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x00000000040d0000 -> 0x00000000040d1845
Device tree struct  0x00000000040e0000 -> 0x0000000004100000
Calling quiesce...
returning from prom_init
[    0.000000] Using pSeries machine description
[    0.000000] Page sizes from device-tree:



Not sure what to do next... Compare debian /boot/config-3.16.0-4-powerpc64 with fedora kernel config ? Or just attach it here ? Thanks.

> Not sure what to do next... Compare debian /boot/config-3.16.0-4-powerpc64
> with fedora kernel config ? Or just attach it here ? Thanks.

Yes, config comparison between Fedora and Debian kernels might help. Also does Debian kernel carry any powerpc specific patches?

well, according to https://www.debian.org/doc/manuals/apt-howto/ch-sourcehandling.en.html , "apt-get source linux-image-3.16.0-4-powerpc64" fetched me 3 files:

mator@debianppc:~/deb-source$ ll
-rw-r--r-- 1 mator mator  2990280 Dec  8 17:46 linux_3.16.7-ckt2-1.debian.tar.xz
-rw-r--r-- 1 mator mator   135254 Dec  8 17:46 linux_3.16.7-ckt2-1.dsc
-rw-r--r-- 1 mator mator 81676360 Dec  8 17:46 linux_3.16.7-ckt2.orig.tar.xz

If you don't have debian installed, you can try to download those files from here : http://mirror.yandex.ru/debian/pool/main/l/linux/

linux_3.16.7-ckt2-1.debian.tar.xz contains debian patches, but there's a lot of them (as like in any other packaged kernel). Can someone attach to this bug /boot/config or /proc/config file from fedora ISO file? Since I've no idea how to extract it from ISO file, nor I have installed fedora21 PPC available. 

Thanks.

PS: I'm going to attach /boot/config from kernel 3.16.0-4-powerpc64 debian 7 testing.

Created attachment 979146 [details]
debian 7 testing kernel config file

I see this on a Power6 blade as well (with FC21 GA).

Can we mirror this over the LTC BZ?

Also, does FC21 actually support Power6? That is, just verifying this wasn't intentional.

got config file for ppc64 kernel from http://ppc.koji.fedoraproject.org/koji/packageinfo?packageID=3142

Anyone can point me on how to rebuild installation ISO file with a newer (my own testing) kernel ? 

Thanks.

Anatoly,

I took a more direct approach and unzipped the initrd and started tearing stuff out of it (as it's quite large and unnecessarily slow on a P6, at least here).

mkdir /mnt/scratch
cd /mnt/scratch
zcat initrd.img | cpio -id [note that for some reason the ISO I have for FC21 ppc64 (BE) does not indicate the initrd.img is a xz compressed file, I need to check on that]
[start pulling out stuff, like possibly firmware or unneeded kernel modules]
find . | cpio --quiet -o -H newc | xz > initrd-netboot.img

I might have ripped out too much here, as it fails to mount root, but it gets much further in boot.

-Nish

(In reply to Nish Aravamudan from comment #12)
> I see this on a Power6 blade as well (with FC21 GA).
> 
> Can we mirror this over the LTC BZ?
> 
> Also, does FC21 actually support Power6? That is, just verifying this wasn't
> intentional.

yes, Power6 is supported, even Power5 works fine, so it is likely a subtle compatibility issue

Can someone try to boot the vmlinuz-* file extracted from the latest Fedora kernel rpms? From installed Debian or whatever, just to see whether it gets over the early init phase?

I get this when booting the last F-21 kernel from a RHEL-6.6 system (LPAR on JS22), I have used just the vmlinuz, that's why it fails later

...
Welcome to Red Hat Enterprise Linux!
Hit <TAB> for boot options
Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-43.el6)
Enter "help" to get some basic usage information
boot: 
  linux                      fedora                   
boot: fedora
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 03b00000, size: 18889 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.18.3-201.fc21.ppc64p7 (mockbuild.fedoraproject.org) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #1 SMP Wed Jan 21 01:58:17 UTC 2015
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 512 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: root=/dev/mapper/vg_ibmjs22vios01lp1-lv_root ro rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us rd_LVM_LV=vg_ibmjs22vios01lp1/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_ibmjs22vios01lp1/lv_root rd_NO_DM rhgb quiet 
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000004d80000
  alloc_top    : 0000000008000000
  alloc_top_hi : 0000000008000000
  rmo_top      : 0000000008000000
  ram_top      : 0000000008000000
instantiating rtas at 0x00000000074e0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000004d90000 -> 0x0000000004d91507
Device tree struct  0x0000000004da0000 -> 0x0000000004dc0000
Calling quiesce...
returning from prom_init
 -> smp_release_cpus()
spinning_secondaries = 7
 <- smp_release_cpus()
 <- setup_system()
/init: 140: cannot create /sys/class/firmware/timeout: Directory nonexistent
udevd[132]: error setting /sys/class/firmware/timeout: No such file or directory

FATAL: Could not load /lib/modules/3.18.3-201.fc21.ppc64p7/modules.dep: No such file or directory

even kernel-3.17.4-301.fc21.ppc64 boots this way, so IMO either the problem is related to using bare metal or to the size of initrd in the installer

also the installer vmlinuz and initrd.img can be booted when supplied as a second boot option to yaboot, maybe grub2 plays a role in the failure too

What does the Debian use - yaboot or grub2?

(In reply to Dan Horák from comment #18)
> even kernel-3.17.4-301.fc21.ppc64 boots this way, so IMO either the problem
> is related to using bare metal or to the size of initrd in the installer

Yeah, I'm fairly sure it's the size of the initrd that's a problem. It's been growing each release and we have limited space in the RMA on Power.

> What does the Debian use - yaboot or grub2?

yaboot


I've taken latest ppc64 kernel from koji, which is kernel-core-3.18.3-201.fc21.ppc64.rpm and kernel-modules-3.18.3-201.fc21.ppc64.rpm , extracted, moved kernel to /boot and modules to /lib/modules , generated new kernel initrd with :

# mkinitramfs -o /boot/initrd.img-3.18.3-201.fc21.ppc64 3.18.3-201.fc21.ppc64

added kernel to /etc/yaboot.conf :

root@debianppc:/home/mator# cat /etc/yaboot.conf 
## yaboot.conf generated by debian-installer
##
## run: "man yaboot.conf" for details. Do not make changes until you have!!
## see also: /usr/share/doc/yaboot/examples for example configurations.
##
## For a dual-boot menu, add one or more of:
## bsd=/dev/hdaX, macos=/dev/hdaY, macosx=/dev/hdaZ

boot="/dev/disk/by-id/scsi-35000c5000f6510cf-part1"
device=/pci@800000020000200/pci1014,02BD@1/sas/disk@30000
partition=2
root="UUID=7c27358c-21f3-49ae-ae2f-bad559ecbbb2"
timeout=50
install=/usr/lib/yaboot/yaboot
enablecdboot

image=/boot/vmlinux
        label=Linux
        read-only
        initrd=/boot/initrd.img

image=/boot/vmlinux.old
        label=old
        read-only
        initrd=/boot/initrd.img.old

image=/boot/vmlinuz-3.18.3-201.fc21.ppc64
        label=fedora
        read-only
        initrd=/boot/initrd.img-3.18.3-201.fc21.ppc64


and tried to boot. It boots successfully up to a login prompt (skipped in the followed output):

Welcome to yaboot version 1.3.16
Enter "help" to get some basic usage information
boot: fedora     
Please wait, loading kernel...
   Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 03b80000, size: 14312 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.18.3-201.fc21.ppc64 (mockbuild.fedoraproject.org) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #1 SMP Wed Jan 21 02:00:40 UTC 2015
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 512 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: root=UUID=7c27358c-21f3-49ae-ae2f-bad559ecbbb2 ro
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000004980000
  alloc_top    : 0000000008000000
  alloc_top_hi : 0000000008000000
  rmo_top      : 0000000008000000
  ram_top      : 0000000008000000
found display   : /pci@800000020000202/display@1, opening... done
instantiating rtas at 0x0000000006eb0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000004990000 -> 0x0000000004991845
Device tree struct  0x00000000049a0000 -> 0x00000000049c0000
Calling quiesce...
returning from prom_init
[    0.000000] Using pSeries machine description

So, can we say now that it has been related to the size of initrd image? Can someone teach me on how to rebuild ISO with a new kernel (initrd) and make it bootable and/or just create fedora 21 ISO with reduced initrd image for test and pass it to me? Thanks.

Looks to me that the failure is in prom_instantiate_rtas() (arch/powerpc/kernel/prom_init.c) and there is simply not enough space for the RTAS between alloc_bottom and alloc_top. What is interesting for me is the difference between the number Anatoly sees when using grub and my attempts with yaboot. Maybe it is grub2 who is allocating the kernel too high in the memory. But someone with better knowledge of the internals (the platform and grub2) should look on it.

I recall there is a workaround for failed memory allocations during boot, also mentioned in
"Q: I’m installing Linux on my system and the boot failed. ..." at http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaae/liaaefaq.htm?lang=en

I'm sorry, but I've lost access to hardware (it was switched off and dismantled)... :(

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 21 kernel bugs.

Fedora 21 has now been rebased to 3.19.5-200.fc21.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 22, and are still experiencing this issue, please change the version to Fedora 22.

If you experience different issues, please open a new bug report for those.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.