Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1790436

Summary: Supermin fails to mount root due to virtio modprobe failure on power9
Product: [Fedora] Fedora Reporter: Jakub Čajka <jcajka>
Component: SLOFAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: crobinso, dan, gduarte, menantea, pbonzini, ptoscano, rjones, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64le   
OS: Linux   
Whiteboard:
Fixed In Version: SLOF-0.1.git20191022-1.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-06 21:44:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1071880    

Description Jakub Čajka 2020-01-13 11:04:26 UTC
Description of problem:
Fedora CoreOS assembler fails to run prepared supermin appliance during build of Fedora CoreOS on power9 systems(works just fine on Power8 bare metal hosts).

Version-Release number of selected component (if applicable):
supermin-5.1.20-11.fc31.ppc64le

How reproducible:
Always on power9 bare metal system

Steps to Reproduce:
1. Build Fedora CoreOS on Power9 system(I should be able to provide access to system in which the issue can be reproduced).

Unfortunately I don't have yet more reduced reproducer.

Actual results:
qemu-system-ppc64 -accel kvm -smp 16 -vga none -nodefaults -nographic -m 2048 -no-reboot -cpu host -kernel ./tmp/supermin.build/kernel -initrd ./tmp/supermin.build/initrd -netdev user,id=eth0,hostname=supermin -device virtio-net-pci,netdev=eth0 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -drive if=none,id=drive-scsi0-0-0-0,snapshot=on,file=./tmp/supermin.build/root -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -virtfs local,id=workdir,path=./,security_model=none,mount_tag=workdir -serial stdio -append 'root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1' -drive if=virtio,id=target,format=qcow2,file=./tmp/build/fedora-coreos-31.20200110.dev.0-qemu.ppc64le.qcow2.tmp,cache=unsafe


SLOF **********************************************************************
QEMU Starting
 Build Date = Nov 12 2019 00:00:00
 FW Version = mockbuild@ release 20191022
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /pci@800000020000000
                     00 0000 (D) : 1af4 1000    virtio [ net ]
                     00 0800 (D) : 1af4 1005    legacy-device*
                     00 1000 (D) : 1af4 1009    virtio [ network ]
Populating /pci@800000020000000/virtio-9p@2
                     00 1800 (D) : 1af4 1004    virtio [ scsi ]
Populating /pci@800000020000000/scsi@3
       SCSI: Looking for devices
          100000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
                     00 2000 (D) : 1af4 1001    virtio [ block ]
No NVRAM common partition, re-initializing...
Scanning USB 
Using default console: /vdevice/vty@71000000
Detected RAM kernel at 400000 (1e54e58 bytes) 
     
  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Booting from memory...
OF stdout device is: /vdevice/vty@71000000
Preparing to boot Linux version 5.4.8-200.fc31.ppc64le (mockbuild.fedoraproject.org) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Mon Jan 6 16:29:22 UTC 2020
Detected machine type: 0000000000000101
command line: root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...qemu-system-ppc64: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM
Falling back to kernel-irqchip=off


SLOF **********************************************************************
QEMU Starting
 Build Date = Nov 12 2019 00:00:00
 FW Version = mockbuild@ release 20191022
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /pci@800000020000000
                     00 0000 (D) : 1af4 1000    virtio [ net ]
                     00 0800 (D) : 1af4 1005    legacy-device*
                     00 1000 (D) : 1af4 1009    virtio [ network ]
Populating /pci@800000020000000/virtio-9p@2
                     00 1800 (D) : 1af4 1004    virtio [ scsi ]
Populating /pci@800000020000000/scsi@3
       SCSI: Looking for devices
          100000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
                     00 2000 (D) : 1af4 1001    virtio [ block ]
Scanning USB 
Using default console: /vdevice/vty@71000000
Detected RAM kernel at 400000 (1e54e58 bytes) 
     
  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Booting from memory...
OF stdout device is: /vdevice/vty@71000000
Preparing to boot Linux version 5.4.8-200.fc31.ppc64le (mockbuild.fedoraproject.org) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Mon Jan 6 16:29:22 UTC 2020
Detected machine type: 0000000000000101
command line: root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...Node not supported 
Node not supported 
Node not supported 
 not implemented
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000002270000
  alloc_top    : 0000000010000000
  alloc_top_hi : 0000000080000000
  rmo_top      : 0000000010000000
  ram_top      : 0000000080000000
instantiating rtas at 0x000000000fff0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000002580000 -> 0x0000000002580b42
Device tree struct  0x0000000002590000 -> 0x00000000025a0000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000000400000 ...
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Activating Kernel Userspace Execution Prevention
[    0.000000] radix-mmu: Activating Kernel Userspace Access Prevention
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000000080000000 with 1.00 GiB pages
[    0.000000] lpar: Using radix MMU under hypervisor
[    0.000000] Linux version 5.4.8-200.fc31.ppc64le (mockbuild.fedoraproject.org) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Mon Jan 6 16:29:22 UTC 2020
[    0.000000] Found initrd at 0xc000000002270000:0xc0000000024eba00
[    0.000000] Using pSeries machine description
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Partition configured for 16 cpus.
[    0.000000] CPU maps initialized for 1 thread per core
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x80000000
[    0.000000] dcache_bsize      = 0x80
[    0.000000] icache_bsize      = 0x80
[    0.000000] cpu_features      = 0x0001c06f8f4f91a7
[    0.000000]   possible        = 0x0001fbffcf5fb1a7
[    0.000000]   always          = 0x00000003800081a1
[    0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[    0.000000] mmu_features      = 0xbc006041
[    0.000000] firmware_features = 0x00000005455a445f
[    0.000000] vmalloc start     = 0xc008000000000000
[    0.000000] IO start          = 0xc00a000000000000
[    0.000000] vmemmap start     = 0xc00c000000000000
[    0.000000] -----------------------------------------------------
[    0.000000] numa:   NODE_DATA [mem 0x7ff5ef80-0x7ff63fff]
[    0.000000] rfi-flush: fallback displacement flush available
[    0.000000] rfi-flush: ori type flush available
[    0.000000] rfi-flush: mttrig type flush available
[    0.000000] link-stack-flush: software flush enabled.
[    0.000000] count-cache-flush: full software flush sequence enabled.
[    0.000000] stf-barrier: eieio barrier available
[    0.000000] PCI host bridge /pci@800000020000000  ranges:
[    0.000000]   IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[    0.000000]  MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000 
[    0.000000]  MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000 
[    0.000000] PCI: OF: PROBE_ONLY disabled
[    0.000000] PPC64 nvram contains 65536 bytes
[    0.000000] barrier-nospec: using ORI speculation barrier
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] percpu: Embedded 11 pages/cpu s622872 r0 d98024 u720896
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32736
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1
[    0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[    0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 1970560K/2097152K available (16000K kernel code, 3776K rwdata, 3968K rodata, 5504K init, 1747K bss, 126592K reserved, 0K cma-reserved)
[    0.000000] random: get_random_u64 called from __kmem_cache_create+0x58/0x9e0 with crng_init=0
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000] ftrace: allocating 32187 entries in 12 pages
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=16.
[    0.000000] 	Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16
[    0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[    0.000000] xive: Using IRQ range [0-f]
[    0.000000] xive: Interrupt handling initialized with spapr backend
[    0.000000] xive: Using priority 6 for all interrupts
[    0.000000] xive: Using 64kB queues
[    0.000001] time_init: 56 bit decrementer (max: 7fffffffffffff)
[    0.000487] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[    0.001336] clocksource: timebase mult[1f40000] shift[24] registered
[    0.001884] Console: colour dummy device 80x25
[    0.002251] printk: console [hvc0] enabled
[    0.002251] printk: console [hvc0] enabled
[    0.002609] printk: bootconsole [udbg0] disabled
[    0.002609] printk: bootconsole [udbg0] disabled
[    0.003038] pid_max: default: 32768 minimum: 301
[    0.003128] LSM: Security Framework initializing
[    0.003195] Yama: becoming mindful.
[    0.003228] SELinux:  Initializing.
[    0.003313] *** VALIDATE SELinux ***
[    0.003354] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.003403] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.003470] *** VALIDATE tmpfs ***
[    0.003622] *** VALIDATE proc ***
[    0.003684] *** VALIDATE cgroup1 ***
[    0.003706] *** VALIDATE cgroup2 ***
[    0.003928] EEH: pSeries platform initialized
[    0.003962] POWER9 performance monitor hardware support registered
[    0.004034] rcu: Hierarchical SRCU implementation.
[    0.004478] smp: Bringing up secondary CPUs ...
[    0.019450] smp: Brought up 1 node, 16 CPUs
[    0.019571] numa: Node 0 CPUs: 0-15
[    0.019596] Using standard scheduler topology
[    0.020214] devtmpfs: initialized
[    0.021718] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.021789] futex hash table entries: 4096 (order: 3, 524288 bytes, linear)
[    0.022477] NET: Registered protocol family 16
[    0.022789] audit: initializing netlink subsys (disabled)
[    0.023078] audit: type=2000 audit(1578676904.020:1): state=initialized audit_enabled=0 res=1
[    0.023301] cpuidle: using governor menu
[    0.023452] pstore: Registered nvram as persistent store backend
Linux ppc64le
#1 SMP Mon Jan 6[    0.028257] PCI: Probing PCI hardware
[    0.028308] PCI host bridge to bus 0000:00
[    0.028335] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    0.028387] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[    0.028448] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[    0.028494] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.028549] pci 0000:00:00.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[    0.028694] pci 0000:00:01.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[    0.028836] pci 0000:00:02.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[    0.028978] pci 0000:00:03.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[    0.029118] pci 0000:00:04.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[    0.031152] IOMMU table initialized, virtual merging enabled
[    0.031202] pci 0000:00:00.0: Adding to iommu group 0
[    0.031248] pci 0000:00:00.0: of_irq_parse_pci: failed with rc=134
[    0.031296] pci 0000:00:01.0: Adding to iommu group 0
[    0.031338] pci 0000:00:01.0: of_irq_parse_pci: failed with rc=134
[    0.031388] pci 0000:00:02.0: Adding to iommu group 0
[    0.031429] pci 0000:00:02.0: of_irq_parse_pci: failed with rc=134
[    0.031478] pci 0000:00:03.0: Adding to iommu group 0
[    0.031519] pci 0000:00:03.0: of_irq_parse_pci: failed with rc=134
[    0.031568] pci 0000:00:04.0: Adding to iommu group 0
[    0.031609] pci 0000:00:04.0: of_irq_parse_pci: failed with rc=134
[    0.031731] pci_bus 0000:00: resource 4 [io  0x10000-0x1ffff]
[    0.031773] pci_bus 0000:00: resource 5 [mem 0x200080000000-0x2000ffffffff]
[    0.031814] pci_bus 0000:00: resource 6 [mem 0x210000000000-0x21ffffffffff]
[    0.032412] EEH: No capable adapters found: recovery disabled.
[    0.034921] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.034979] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.202440] alg: No test for 842 (842-generic)
[    0.202847] alg: No test for 842 (842-scomp)
[    0.212460] random: fast init done
[    0.217633] iommu: Default domain type: Translated 
[    0.217855] vgaarb: loaded
[    0.217955] SCSI subsystem initialized
[    0.218463] usbcore: registered new interface driver usbfs
[    0.218511] usbcore: registered new interface driver hub
[    0.218881] usbcore: registered new device driver usb
[    0.218954] pps_core: LinuxPPS API ver. 1 registered
[    0.218995] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti>
[    0.219063] PTP clock support registered
[    0.219746] EDAC MC: Ver: 3.0.0
[    0.220323] NetLabel: Initializing
[    0.220357] NetLabel:  domain hash size = 128
[    0.220403] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.220464] NetLabel:  unlabeled traffic allowed by default
[    0.221050] clocksource: Switched to clocksource timebase
[    0.236610] *** VALIDATE bpf ***
[    0.236970] VFS: Disk quotas dquot_6.6.0
[    0.237024] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    0.237082] *** VALIDATE ramfs ***
[    0.237109] *** VALIDATE hugetlbfs ***
[    0.248797] thermal_sys: Registered thermal governor 'fair_share'
[    0.248798] thermal_sys: Registered thermal governor 'step_wise'
[    0.248901] NET: Registered protocol family 2
[    0.249071] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    0.249128] TCP established hash table entries: 16384 (order: 1, 131072 bytes, linear)
[    0.249193] TCP bind hash table entries: 16384 (order: 2, 262144 bytes, linear)
[    0.249255] TCP: Hash tables configured (established 16384 bind 16384)
[    0.249318] UDP hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.249365] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.249456] NET: Registered protocol family 1
[    0.249492] NET: Registered protocol family 44
[    0.249560] PCI: CLS 0 bytes, default 128
[    0.249617] Trying to unpack rootfs image as initramfs...
[    0.250340] Freeing initrd memory: 2496K
[    0.250454] rtas_flash: no firmware flash support
[    0.250958] Initialise system trusted keyrings
[    0.250999] Key type blacklist registered
[    0.251814] workingset: timestamp_bits=38 max_order=15 bucket_order=0
[    0.253214] zbud: loaded
[    0.263121] NET: Registered protocol family 38
[    0.263159] Key type asymmetric registered
[    0.263186] Asymmetric key parser 'x509' registered
[    0.263225] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.264074] io scheduler mq-deadline registered
[    0.264114] io scheduler kyber registered
[    0.264169] io scheduler bfq registered
[    0.267145] atomic64_test: passed
[    0.267209] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.267296] virtio-pci 0000:00:00.0: enabling device (0000 -> 0003)
[    0.267359] virtio-pci 0000:00:00.0: virtio_pci: leaving for legacy driver
[    0.267410] virtio-pci 0000:00:00.0: ibm,query-pe-dma-windows(2026) 0 1000 0 returned -3
[    0.267531] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)
[    0.267591] virtio-pci 0000:00:01.0: virtio_pci: leaving for legacy driver
[    0.267696] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)
[    0.267758] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[    0.267862] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)
[    0.267924] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    0.268032] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)
[    0.268094] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[    0.270092] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.275391] Non-volatile memory driver v1.3
[    0.275601] virtio_rng: probe of virtio1 failed with error -22
[    0.275986] libphy: Fixed MDIO Bus: probed
[    0.276196] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.276247] ehci-pci: EHCI PCI platform driver
[    0.276288] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.276335] ohci-pci: OHCI PCI platform driver
[    0.276376] uhci_hcd: USB Universal Host Controller Interface driver
[    0.276449] usbcore: registered new interface driver usbserial_generic
[    0.276496] usbserial: USB Serial support registered for generic
[    0.276696] mousedev: PS/2 mouse device common for all mice
[    0.276920] rtc-generic rtc-generic: registered as rtc0
[    0.276997] device-mapper: uevent: version 1.0.3
[    0.277199] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel
[    0.278250] hidraw: raw HID events driver (C) Jiri Kosina
[    0.278315] usbcore: registered new interface driver usbhid
[    0.278349] usbhid: USB HID core driver
[    0.278600] drop_monitor: Initializing network drop monitor service
[    0.278697] Initializing XFRM netlink socket
[    0.278843] NET: Registered protocol family 10
[    0.280171] Segment Routing with IPv6
[    0.280215] mip6: Mobile IPv6
[    0.280243] NET: Registered protocol family 17
[    0.280947] drmem: No dynamic reconfiguration memory found
[    0.281331] registered taskstats version 1
[    0.281412] Loading compiled-in X.509 certificates
[    0.321540] Loaded X.509 cert 'Fedora kernel signing key: 17d6619228eb72d3e5a27c990bdeef850440c07b'
[    0.321665] zswap: loaded using pool lzo/zbud
[    0.322519] Key type ._fscrypt registered
[    0.322561] Key type .fscrypt registered
[    0.322692] pstore: Using crash dump compression: deflate
[    0.325938] Key type big_key registered
[    0.328271] Key type encrypted registered
[    0.328315] ima: No TPM chip found, activating TPM-bypass!
[    0.328365] ima: Allocated hash algorithm: sha256
[    0.328419] ima: No architecture policies found
[    0.329040] rtc-generic rtc-generic: setting system clock to 2020-01-10T17:21:44 UTC (1578676904)
[    0.331406] Freeing unused kernel memory: 5504K
[    0.331454] This architecture does not have kernel memory protection.
[    0.331509] Run /init as init process
supermin: mounting /proc
supermin: ext2 mini initrd starting up: 5.1.20 glibc
supermin: cmdline: root=/dev/sda console=hvc0 selinux=1 enforcing=0 autorelabel=1
supermin: uptime: 0.32 151.65
supermin: mounting /sys
supermin: internal insmod crc32c-vpmsum.ko
supermin: internal insmod crct10dif-vpmsum.ko
supermin: internal insmod crc32_generic.ko
supermin: internal insmod virtio_blk.ko
[    0.446112] virtio_blk: probe of virtio4 failed with error -22
supermin: internal insmod virtio_console.ko
supermin: internal insmod crypto_engine.ko
supermin: internal insmod virtio_crypto.ko
supermin: internal insmod failover.ko
supermin: internal insmod net_failover.ko
supermin: internal insmod virtio_net.ko
[    0.501110] virtio_net: probe of virtio0 failed with error -22
supermin: internal insmod libnvdimm.ko
supermin: internal insmod nd_pmem.ko
supermin: internal insmod rpmsg_core.ko
supermin: internal insmod virtio_rpmsg_bus.ko
supermin: internal insmod virtio_scsi.ko
[    0.561795] virtio_scsi: probe of virtio3 failed with error -22
supermin: internal insmod virtio_balloon.ko
supermin: internal insmod virtio_input.ko
supermin: internal insmod virtio_mmio.ko
supermin: internal insmod fuse.ko
[    0.573292] fuse: init (API version 7.31)
[    0.573362] *** VALIDATE fuse ***
[    0.573401] *** VALIDATE fuse ***
supermin: internal insmod virtiofs.ko
supermin: internal insmod crc-itu-t.ko
supermin: internal insmod crc4.ko
supermin: internal insmod crc64.ko
supermin: internal insmod crc7.ko
supermin: internal insmod crc8.ko
supermin: internal insmod libcrc32c.ko
supermin: waiting another 1024000000 ns for /sys/block/sda/dev to appear
This usually means your kernel doesn't support virtio, or supermin was unable
to load some kernel modules (see module loading messages above).
supermin: waiting another 2048000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 4096000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 8192000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 16384000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 32768000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 65536000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 131072000000 ns for /sys/block/sda/dev to appear
supermin: waiting another 262144000000 ns for /sys/block/sda/dev to appear
supermin: creating /dev/root as block special 18:0
supermin: mounting new root on /root
mount: /root: No such device or address
[  524.888668] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[  524.888742] CPU: 0 PID: 1 Comm: init Not tainted 5.4.8-200.fc31.ppc64le #1
[  524.888800] Call Trace:
[  524.888829] [c00000007a5cfc40] [c000000000f670a4] dump_stack+0xb8/0x104 (unreliable)
[  524.888901] [c00000007a5cfc80] [c00000000016a5dc] panic+0x168/0x40c
[  524.888961] [c00000007a5cfd10] [c000000000173abc] do_exit+0xcac/0xd80
[  524.889020] [c00000007a5cfdd0] [c000000000173c5c] sys_exit_group+0x5c/0x110
[  524.889080] [c00000007a5cfe20] [c00000000000b9d0] system_call+0x5c/0x68


Expected results:
Applience able to mount the rootfs and coreos assembler finishing.

Additional info:
lsinitrd ./tmp/supermin.build/initrd
Image: ./tmp/supermin.build/initrd: 2.5M
========================================================================
Version: 

Arguments: 
dracut modules:
========================================================================
drwxr-xr-x   2 1000  1000         0 Jan 10 18:14 .
-rw-r--r--   1 1000  1000     17486 Jan 10 18:14 crc32c-vpmsum.ko
-rw-r--r--   1 1000  1000      9294 Jan 10 18:14 crc32_generic.ko
-rw-r--r--   1 1000  1000      5446 Jan 10 18:14 crc4.ko
-rw-r--r--   1 1000  1000      7502 Jan 10 18:14 crc64.ko
-rw-r--r--   1 1000  1000      5806 Jan 10 18:14 crc7.ko
-rw-r--r--   1 1000  1000      6246 Jan 10 18:14 crc8.ko
-rw-r--r--   1 1000  1000      6070 Jan 10 18:14 crc-itu-t.ko
-rw-r--r--   1 1000  1000     16142 Jan 10 18:14 crct10dif-vpmsum.ko
-rw-r--r--   1 1000  1000     21702 Jan 10 18:14 crypto_engine.ko
-rw-r--r--   1 1000  1000     16702 Jan 10 18:14 failover.ko
-rw-r--r--   1 1000  1000    280446 Jan 10 18:14 fuse.ko
-rwxr-xr-x   1 1000  1000   1280976 Jan 10 18:14 init
-rw-r--r--   1 1000  1000      7766 Jan 10 18:14 libcrc32c.ko
-rw-r--r--   1 1000  1000    390646 Jan 10 18:14 libnvdimm.ko
-rw-r--r--   1 1000  1000       363 Jan 10 18:14 modules
-rw-r--r--   1 1000  1000      9622 Jan 10 18:14 nd_pmem.ko
-rw-r--r--   1 1000  1000     27646 Jan 10 18:14 net_failover.ko
-rw-r--r--   1 1000  1000     26806 Jan 10 18:14 rpmsg_core.ko
-rw-r--r--   1 1000  1000     36134 Jan 10 18:14 virtio_balloon.ko
-rw-r--r--   1 1000  1000     36334 Jan 10 18:14 virtio_blk.ko
-rw-r--r--   1 1000  1000     71574 Jan 10 18:14 virtio_console.ko
-rw-r--r--   1 1000  1000     45662 Jan 10 18:14 virtio_crypto.ko
-rw-r--r--   1 1000  1000     46342 Jan 10 18:14 virtiofs.ko
-rw-r--r--   1 1000  1000     19726 Jan 10 18:14 virtio_input.ko
-rw-r--r--   1 1000  1000     24638 Jan 10 18:14 virtio_mmio.ko
-rw-r--r--   1 1000  1000    109430 Jan 10 18:14 virtio_net.ko
-rw-r--r--   1 1000  1000     35774 Jan 10 18:14 virtio_rpmsg_bus.ko
-rw-r--r--   1 1000  1000     37246 Jan 10 18:14 virtio_scsi.ko
========================================================================

Comment 1 Richard W.M. Jones 2020-01-13 12:03:22 UTC
That error is weird:

[    0.561795] virtio_scsi: probe of virtio3 failed with error -22

errno 22 = EINVAL

Comment 2 Richard W.M. Jones 2020-01-13 12:05:09 UTC
Similar bug here:

https://bugs.launchpad.net/qemu/+bug/1847440

The suggested solution there was to update SLOF (the virtual firmware).

Comment 3 Jakub Čajka 2020-01-13 13:45:00 UTC
H(In reply to Richard W.M. Jones from comment #2)
> Similar bug here:
> 
> https://bugs.launchpad.net/qemu/+bug/1847440
> 
> The suggested solution there was to update SLOF (the virtual firmware).

I have also run build witch used virt-preview repo(https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/) and I hit same issue. AFAIK it should contain the patched SLOF version(https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/build/1092953/).

Comment 4 Jakub Čajka 2020-01-13 14:23:50 UTC
For the record coreos assembler tracking issue https://github.com/coreos/coreos-assembler/issues/1041 .

Comment 5 Jakub Čajka 2020-01-15 11:58:04 UTC
I have done more testing with different SLOF versions and qemu(with and without virt-preview repo) and it seems that using qemu-slof-20191209 SLOF version with the qemu from F31 works. I'm still looking on what is minimal version of SLOF that will work in this case.

Comment 6 Jakub Čajka 2020-01-16 13:19:42 UTC
I have done more testing and it seems to be resolved with the most recent SLOF. I have tested it both on p9 and p8 hosts with success. 
So switching this to SLOF. Could we get rebase to qemu-slof-20191217 in F31?

Comment 7 Jakub Čajka 2020-01-16 14:57:39 UTC
For the record I have not been able to run coreos assembler(qemu in container) with virt-preview repo enabled on f31(latest kernel) host, with the up mentioned issue/error, both with virt-preview version of SLOF or "mine" latest(qemu-slof-20191217). On power9 bare metal host.

Comment 8 Cole Robinson 2020-01-20 22:16:56 UTC
(In reply to Jakub Čajka from comment #7)
> For the record I have not been able to run coreos assembler(qemu in
> container) with virt-preview repo enabled on f31(latest kernel) host, with
> the up mentioned issue/error, both with virt-preview version of SLOF or
> "mine" latest(qemu-slof-20191217). On power9 bare metal host.

I'm a bit confused, is this an additional problem, or does this invalidate your suggestion to backport rawhide SLOF to f31?

I'm happy to do the f31 build if that improves things for you

Comment 9 Jakub Čajka 2020-01-21 13:33:04 UTC
(In reply to Cole Robinson from comment #8)
> (In reply to Jakub Čajka from comment #7)
> > For the record I have not been able to run coreos assembler(qemu in
> > container) with virt-preview repo enabled on f31(latest kernel) host, with
> > the up mentioned issue/error, both with virt-preview version of SLOF or
> > "mine" latest(qemu-slof-20191217). On power9 bare metal host.
> 
> I'm a bit confused, is this an additional problem, or does this invalidate
> your suggestion to backport rawhide SLOF to f31?
>

That has been note that I have not been able to run build of coreos/run supermin with the virt-preview repo enabled and using packages from it(regardless on version of SLOF used). Sorry for confusion.
 
> I'm happy to do the f31 build if that improves things for you

Thank you. I would very much appreciate that, if that is possible as it resolves issue that I have hit.

Comment 10 Cole Robinson 2020-01-21 19:54:17 UTC
I submitted https://bodhi.fedoraproject.org/updates/FEDORA-2020-cf8a0689a8

Comment 11 Fedora Update System 2020-01-24 05:49:37 UTC
SLOF-0.1.git20191022-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-cf8a0689a8

Comment 12 Fedora Update System 2020-03-06 21:44:52 UTC
SLOF-0.1.git20191022-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.