Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1741935
Summary: | emulation deficiency breaks rpmdb on s390x | ||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Stephan Mueller <smueller> | ||||||||||||||||||||||||||||||||||
Component: | qemu | Assignee: | Fedora Virtualization Maintainers <virt-maint> | ||||||||||||||||||||||||||||||||||
Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||||||||||||||||||||
Severity: | unspecified | Docs Contact: | |||||||||||||||||||||||||||||||||||
Priority: | unspecified | ||||||||||||||||||||||||||||||||||||
Version: | rawhide | CC: | amit, anaconda-maint-list, berrange, cfergeau, cohuck, crobinso, dan, david, dhildenb, dwmw2, itamar, jkonecny, jonathan, kellin, pbonzini, quantum.analyst, rjones, thuth, vanmeeuwen+fedora, virt-maint, vponcova, wwoods | ||||||||||||||||||||||||||||||||||
Target Milestone: | --- | ||||||||||||||||||||||||||||||||||||
Target Release: | --- | ||||||||||||||||||||||||||||||||||||
Hardware: | All | ||||||||||||||||||||||||||||||||||||
OS: | Unspecified | ||||||||||||||||||||||||||||||||||||
Whiteboard: | abrt_hash:9b9691c40db5d0bcea22de6d8b4075a03e747fdc31ad8dfeee3f72fb7ac9923e; | ||||||||||||||||||||||||||||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||||||||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||||||||||||||||||||||
Clone Of: | Environment: | ||||||||||||||||||||||||||||||||||||
Last Closed: | 2020-01-13 17:06:11 UTC | Type: | --- | ||||||||||||||||||||||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||||||||||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||||||||||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||||||||||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||||||||||||||||
Embargoed: | |||||||||||||||||||||||||||||||||||||
Bug Depends On: | |||||||||||||||||||||||||||||||||||||
Bug Blocks: | 467765 | ||||||||||||||||||||||||||||||||||||
Attachments: |
|
Description
Stephan Mueller
2019-08-16 13:22:31 UTC
Created attachment 1604461 [details]
File: anaconda-tb
Created attachment 1604462 [details]
File: anaconda.log
Created attachment 1604463 [details]
File: dbus.log
Created attachment 1604464 [details]
File: dnf.librepo.log
Created attachment 1604465 [details]
File: environ
Created attachment 1604466 [details]
File: hawkey.log
Created attachment 1604467 [details]
File: lorax-packages.log
Created attachment 1604468 [details]
File: lsblk_output
Created attachment 1604469 [details]
File: lvm.log
Created attachment 1604470 [details]
File: nmcli_dev_list
Created attachment 1604471 [details]
File: os_info
Created attachment 1604472 [details]
File: program.log
Created attachment 1604473 [details]
File: storage.log
Created attachment 1604474 [details]
File: syslog
Created attachment 1604475 [details]
File: packaging.log
If I understand the problem correctly we are missing requirement to add zipl package to the installation. Hello, could you please verify if this will happen when you manually add zipl to the installation? I booted from the S390 ISO installation image to install the system. As one of the last steps, Anaconda tries to install the boot loader which fails since the zipl command is not found by Anaconda. So, up to this point I yet do not have a system that I have under my control. I.e. I cannot install anything myself so far. Yes, I understand that you can't boot to the system. What I was asking is to make a kickstart installation with zipl in the %packages section, my assumption is that it should work correctly. Could you please verify this assumption? If you need, I can provide you a kickstart file which can be used to start the installation. From packaging.log: 13:12:36,627 INF packaging: Verifying: s390utils-base-2:2.8.0-2.fc30.s390x The package s390utils-base is installed on the new system and zipl should be part of this package. Reassigning to s390utils. 13:12:50,109 WRN bootloader.installation: No kernel was installed. The boot loader configuration unchanged. from the anaconda.log looks weird I see it's an installation in KVM guest, what method or command did you use? Yes, it is a KVM guest installation. Re kickstart script: I am not versed in the kickstart script configuration. But it should be easy to recreate: create a z390 KVM on x86 and try to install the s390 ISO from Fedora 30. Not sure I understand, is your installation run on a real s390x machine, or do you use the full system emulation in qemu from a x86 machine? I use an emulated machine provided by QEMU Stephan, hasn't you see a bunch of rpmdb/db5 errors on the console? It looks like the installation transaction crashes and installs only a portion of the rpms even when it then says "Verified" for all of them. I still suspect some emulation issue. So the actual problem can be seen in the console log ... Installing ebtables.s390x (269/352) Installing iputils.s390x (270/352) Installing libkcapi.s390x (271/352) Installing libkcapi-hmaccalc.s390x (272/352) Installing systemd-bootchart.s390x (273/352) Installing NetworkManager-libnm.s390x (274/352) Performing post-installation setup tasks Verifying NetworkManager.s390x (1/352) Verifying NetworkManager-libnm.s390x (2/352) Verifying acl.s390x (3/352) Verifying alternatives.s390x (4/352) Verifying audit.s390x (5/352) Verifying audit-libs.s390x (6/352) Verifying authselect.s390x (7/352) Verifying authselect-libs.s390x (8/352) Verifying basesystem.noarch (9/352) ... Verifying whois-nls.noarch (347/352) Verifying xkeyboard-config.noarch (348/352) Verifying xz.s390x (349/352) Verifying xz-libs.s390x (350/352) Verifying zchunk-libs.s390x (351/352) Verifying zlib.s390x (352/352) error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery error: db5 error(-30973) from db->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: rpmdb: BDB1581 File handles still open at environment close error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/__db.001 error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/__db.001 error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/__db.002 error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/__db.003 error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Packages error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Name error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Group error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Requirename error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Providename error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Conflictname error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Obsoletename error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Triggername error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Dirnames error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Installtid error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Sigmd5 error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Sha1header error: rpmdb: BDB1582 Open file handle: /mnt/sysimage/var/lib/rpm/Filetriggername ... There is a crash/data corruption during the installation phase, which skips some packages from installing and goes to verifying, where it clearly lies. In your case s390utils-base isn't installed (even when dnf says "Verifying"), so no /usr/sbin/zipl and the error, in my case the kernel wasn't present. My tests were run with qemu-system-s390x-4.1.0-1.fc30.ppc64le, but got confirmation from a x86 host as well. There is no such problem when using real HW, with or without KVM, so my conclusion is that there is a subtle emulation bug in qemu (TCG) causing all this. Updated summary and it's a general issue in the full system emulation of a s390x guest on any host arch (confirmed on x86_64, ppc64le). So to summarize: installing Fedora 30 in qemu s390x TCG and rpmdb hits some issue, presumably emulation related. Reproduced on x86 and ppc64le host. Reported version is qemu-system-s390x-4.1.0-1.fc30, which must be from virt-preview repo. Dan can you provide /var/log/libvirt/qemu/$vmname.log CCing some qemu s390x folks Created attachment 1612461 [details]
qemu log
Yes, the summary is correct.
For the record - dgilmore was able to install F-29 just fine (using x86_64 host). So likely there is something changed between F-29 and F-30 that breaks the emulation. A rather big change is gcc8 (F-29) vs gcc9 (F-30) and there can be more. Known, issue, please refer to https://patchew.org/QEMU/20190906075750.14791-1-david@redhat.com/ Triggered by recent glibc memmove changes (that triggers under rpm and rpmbuild so far only). I am able to install Fedora Rawhide with these patches applied just fine. new version available at https://patchew.org/QEMU/20190916135806.1269-1-david@redhat.com/ Latest version available at https://github.com/davidhildenbrand/qemu/tree/mvc (kept updated). Just did an install of Fedora 30 and Fedora 31 (rawhide) without running into issues. Fixes are upstream. Thanks David. Does anyone know if this is a regression from previous Fedora releases, or just something that never worked? I'm trying to decide whether to backport that big series, or just point people at virt-preview repo when the next qemu release is out If I understood right, then the bug was always there, but the newer Fedora (glibc) exposed it. I'm running our virt stack from virt-preview, so no backporting needed for me :-) Dan is correct, just to state what used to work: Upstream QEMU was able to install+run at least Fedora 27,28,29. Fedora 30 and 31 could never have worked correctly under any QEMU version, due to the glibc thingy. Depending on which QEMU version we had in the Fedora releases, installing Fedora 27,28,29 worked, but never 30,31. I'll move this to rawhide then, and close when I rebase the next qemu release. It will trickle into virt-preview after that This is in qemu 4.2.0 which is in rawhide/virt-preview now *** Bug 1763111 has been marked as a duplicate of this bug. *** |