Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1397087
Summary: | Rpm hangs after updating to glibc >= 2.25 | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Lukas Slebodnik <lslebodn> | ||||||||
Component: | rpm | Assignee: | Packaging Maintenance Team <packaging-team-maint> | ||||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
Severity: | high | Docs Contact: | |||||||||
Priority: | high | ||||||||||
Version: | 26 | CC: | aliakc, awilliam, eblake, fweimer, ignatenko, jmontleo, kardos.lubos, lsatenstein, mcatanzaro+wrong-account-do-not-cc, michal.halenka, mruckman, novyjindrich, packaging-team-maint, pknirsch, pkubat, pmatilai, redhat-bugzilla, rharwood, rjones, rmatos, robatino, samuel-rhbugs, sgallagh, striker | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | Accepted0Day AcceptedPreviousRelease | ||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2017-06-23 19:34:04 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | 1394862 | ||||||||||
Bug Blocks: | 1349186, 1443415 | ||||||||||
Attachments: |
|
Description
Lukas Slebodnik
2016-11-21 15:11:10 UTC
The valgrind complaint is an ages old false positive caused by BDB internal optimization, its even documented somewhere in BDB doco. Try looking at what it's actually doing when it appears stuck - is it burning cpu/io cycles or just hanging there etc. (In reply to Panu Matilainen from comment #1) > The valgrind complaint is an ages old false positive caused by BDB internal > optimization, its even documented somewhere in BDB doco. > > Try looking at what it's actually doing when it appears stuck - is it > burning cpu/io cycles or just hanging there etc. There is not any activity, but I can see also different valgrind error which does not contain 'dbiSync' Steps to Reproduce: # docker run -ti --rm fedora:rawhide bash Run docker images for fedora rawhide and run following commands inside container 1 dnf update -y 'rpm*' 2 dnf install -y 'dnf-command(builddep)' valgrind 3 dnf debuginfo-install -y rpm libdb 4 valgrind --track-origins=yes dnf update -y glibc [root@d609e6427820 /]# valgrind --track-origins=yes dnf update -y glibc ==80== Memcheck, a memory error detector ==80== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==80== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==80== Command: /usr/bin/dnf update -y glibc ==80== Last metadata expiration check: 0:02:12 ago on Mon Nov 21 15:15:36 2016 UTC. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Upgrading: glibc x86_64 2.24.90-17.fc26 rawhide 3.3 M glibc-all-langpacks x86_64 2.24.90-17.fc26 rawhide 7.0 M glibc-common x86_64 2.24.90-17.fc26 rawhide 881 k libcrypt-nss x86_64 2.24.90-17.fc26 rawhide 46 k Transaction Summary ================================================================================ Upgrade 4 Packages Total download size: 11 M Downloading Packages: (1/4): glibc-common-2.24.90-17.fc26.x86_64.rpm 1.8 MB/s | 881 kB 00:00 (2/4): libcrypt-nss-2.24.90-17.fc26.x86_64.rpm 1.6 MB/s | 46 kB 00:00 (3/4): glibc-2.24.90-17.fc26.x86_64.rpm 2.2 MB/s | 3.3 MB 00:01 (4/4): glibc-all-langpacks-2.24.90-17.fc26.x86_ 2.4 MB/s | 7.0 MB 00:02 -------------------------------------------------------------------------------- Total 1.7 MB/s | 11 MB 00:06 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Upgrading : glibc-common-2.24.90-17.fc26.x86_64 1/8 ==80== Syscall param pwrite64(buf) points to uninitialised byte(s) ==80== at 0x5317DE3: ??? (in /usr/lib64/libpthread-2.24.90.so) ==80== by 0x12720C30: __os_io (in /usr/lib64/libdb-5.3.so) ==80== by 0x1270C731: ??? (in /usr/lib64/libdb-5.3.so) ==80== by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==80== by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==80== by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so) ==80== by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==80== by 0x11F53FF5: dbiSync (db3.c:547) ==80== by 0x11F53FF5: db3_dbiCursorFree (db3.c:606) ==80== by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411) ==80== by 0x11F72A0C: dbAdd (psm.c:551) ==80== by 0x11F72A0C: rpmPackageInstall (psm.c:682) ==80== by 0x11F72A0C: rpmpsmRun (psm.c:844) ==80== by 0x11F862A4: rpmteProcess (rpmte.c:756) ==80== by 0x11F8CBBD: rpmtsProcess (transaction.c:1361) ==80== by 0x11F8CBBD: rpmtsRun (transaction.c:1541) ==80== Address 0x7017596 is 2,278 bytes inside a block of size 4,096 alloc'd ==80== at 0x4C2DB8D: malloc (vg_replace_malloc.c:299) ==80== by 0x1271E1F4: __os_malloc (in /usr/lib64/libdb-5.3.so) ==80== by 0x1270C897: ??? (in /usr/lib64/libdb-5.3.so) ==80== by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==80== by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==80== by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so) ==80== by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==80== by 0x11F53FF5: dbiSync (db3.c:547) ==80== by 0x11F53FF5: db3_dbiCursorFree (db3.c:606) ==80== by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411) ==80== by 0x11F72A0C: dbAdd (psm.c:551) ==80== by 0x11F72A0C: rpmPackageInstall (psm.c:682) ==80== by 0x11F72A0C: rpmpsmRun (psm.c:844) ==80== by 0x11F862A4: rpmteProcess (rpmte.c:756) ==80== by 0x11F8CBBD: rpmtsProcess (transaction.c:1361) ==80== by 0x11F8CBBD: rpmtsRun (transaction.c:1541) ==80== Uninitialised value was created by a stack allocation ==80== at 0x125EB590: ??? (in /usr/lib64/libdb-5.3.so) ==80== Upgrading : glibc-all-langpacks-2.24.90-17.fc26.x86_64 2/8 ==80== Conditional jump or move depends on uninitialised value(s) ==80== at 0x1260C694: __bam_stkrel (in /usr/lib64/libdb-5.3.so) ==80== by 0x125FD257: ??? (in /usr/lib64/libdb-5.3.so) ==80== by 0x126B71DB: __dbc_iput (in /usr/lib64/libdb-5.3.so) ==80== by 0x126C69C0: __dbc_put_pp (in /usr/lib64/libdb-5.3.so) ==80== by 0x11F53234: dbiCursorPut.isra.5 (db3.c:626) ==80== by 0x11F551C0: updateIndex.part.8 (db3.c:1097) ==80== by 0x11F55590: updateIndex (db3.c:1144) ==80== by 0x11F55590: db3_idxdbPut (db3.c:1140) ==80== by 0x11F5B0B2: tag2index (rpmdb.c:2348) ==80== by 0x11F5ED4A: indexPut (rpmdb.c:2377) ==80== by 0x11F5ED4A: rpmdbAdd (rpmdb.c:2421) ==80== by 0x11F72A0C: dbAdd (psm.c:551) ==80== by 0x11F72A0C: rpmPackageInstall (psm.c:682) ==80== by 0x11F72A0C: rpmpsmRun (psm.c:844) ==80== by 0x11F862A4: rpmteProcess (rpmte.c:756) ==80== by 0x11F8CBBD: rpmtsProcess (transaction.c:1361) ==80== by 0x11F8CBBD: rpmtsRun (transaction.c:1541) ==80== Uninitialised value was created by a stack allocation ==80== at 0x125E9C50: ??? (in /usr/lib64/libdb-5.3.so) ==80== Upgrading : glibc-2.24.90-17.fc26.x86_64 3/8 Upgrading : libcrypt-nss-2.24.90-17.fc26.x86_64 4/8 Cleanup : libcrypt-nss-2.24.90-2.fc26.x86_64 5/8 Cleanup : glibc-common-2.24.90-2.fc26.x86_64 6/8 Cleanup : glibc-all-langpacks-2.24.90-2.fc26.x86_64 7/8 Cleanup : glibc-2.24.90-2.fc26.x86_64 8/8 Verifying : glibc-2.24.90-17.fc26.x86_64 1/8 Verifying : glibc-common-2.24.90-17.fc26.x86_64 2/8 Verifying : glibc-all-langpacks-2.24.90-17.fc26.x86_64 3/8 Verifying : libcrypt-nss-2.24.90-17.fc26.x86_64 4/8 Verifying : glibc-2.24.90-2.fc26.x86_64 5/8 Verifying : glibc-all-langpacks-2.24.90-2.fc26.x86_64 6/8 Verifying : glibc-common-2.24.90-2.fc26.x86_64 7/8 Verifying : libcrypt-nss-2.24.90-2.fc26.x86_64 8/8 Upgraded: glibc.x86_64 2.24.90-17.fc26 glibc-all-langpacks.x86_64 2.24.90-17.fc26 glibc-common.x86_64 2.24.90-17.fc26 libcrypt-nss.x86_64 2.24.90-17.fc26 Complete! ==80== ==80== HEAP SUMMARY: ==80== in use at exit: 58,807,031 bytes in 56,957 blocks ==80== total heap usage: 1,264,661 allocs, 1,207,704 frees, 4,144,098,835 bytes allocated ==80== ==80== LEAK SUMMARY: ==80== definitely lost: 1,521 bytes in 23 blocks ==80== indirectly lost: 125,140 bytes in 66 blocks ==80== possibly lost: 3,288,601 bytes in 19,571 blocks ==80== still reachable: 55,391,001 bytes in 37,287 blocks ==80== of which reachable via heuristic: ==80== length64 : 426,096 bytes in 504 blocks ==80== newarray : 1,536 bytes in 16 blocks ==80== suppressed: 0 bytes in 0 blocks ==80== Rerun with --leak-check=full to see details of leaked memory ==80== ==80== For counts of detected and suppressed errors, rerun with: -v ==80== ERROR SUMMARY: 34 errors from 2 contexts (suppressed: 0 from 0) Created attachment 1222428 [details] Dockerfile which reveal the bug very often Copy attached file to empty directory and build and image sh# docker build -t temp_rm_me . The installation of build dependencies stuck for me with following output Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: 389-ds-base-devel x86_64 1.3.6.1-1.fc26 rawhide 222 k autoconf noarch 2.69-23.fc26 rawhide 710 k automake noarch 1.15-8.fc26 rawhide 694 k cyrus-sasl-devel x86_64 2.1.26-27.fc26 rawhide 313 k dbus-python x86_64 1.2.4-2.fc25 rawhide 129 k //snip Transaction Summary ================================================================================ Install 209 Packages Upgrade 28 Packages Skip 14 Packages Total download size: 99 M Downloading Packages: -------------------------------------------------------------------------------- Total 28 MB/s | 99 MB 00:03 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Upgrading : libcom_err-1.43.3-1.fc26.x86_64 1/265 Installing : libtalloc-2.1.8-1.fc26.x86_64 2/265 Upgrading : nspr-4.13.1-1.fc26.x86_64 3/265 Installing : libtevent-0.9.31-1.fc26.x86_64 4/265 Installing : libtdb-1.3.11-1.fc26.x86_64 5/265 Upgrading : nss-util-3.27.0-2.fc26.x86_64 6/265 Installing : python2-six-1.10.0-4.fc26.noarch 7/265 Upgrading : libuuid-2.29-1.fc26.x86_64 8/265 Installing : libldb-1.1.27-1.fc26.x86_64 9/265 Upgrading : pcre-8.39-6.fc26.x86_64 10/265 Installing : nspr-devel-4.13.1-1.fc26.x86_64 11/265 Upgrading : libblkid-2.29-1.fc26.x86_64 12/265 Installing : xmlrpc-c-1.32.5-1909.svn2451.fc24.x86_64 13/265 and there are just 2 processes inside container shell and python with dnf CGroup: /system.slice/docker-04f608509c75dc43bb7a1c5e86fae071397b4969ba03d179cc0b3466229bfd37.scope ├─24379 /bin/sh -c dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 --setopt=install_weak_deps=false --setopt=tsflags=nodocs install -y sudo git 'dnf-command(builddep)' @buildsys-build && curl https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in -o freeipa.spec.in && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 --setopt=install_weak_deps=false --setopt=tsflags=nodocs builddep -y --spec freeipa.spec.in && rm -f freeipa.spec.in && dnf clean all └─24589 /usr/libexec/system-python /usr/bin/dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 --setopt=install_weak_deps=false --setopt=tsflags=nodocs builddep -y --spec freeipa.spec.in The strace for python is: sh# strace -p 24589 strace: Process 24589 attached futex(0x7ff6f2a11064, FUTEX_WAIT, 4294967295, NULL As I mention in the description The same problem is with yum-deprecated. And if I try to install some packages manually the installation is stuck on different package. BTW my workaround is to remove && rebuild rpm-db before calling builddep --- a/Dockerfile 2016-11-21 16:24:25.994623950 +0100 +++ b/Dockerfile 2016-11-21 14:53:04.466568545 +0100 @@ -11,6 +11,8 @@ @buildsys-build \ && curl https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in \ -o freeipa.spec.in \ + && rm -f /var/lib/rpm/__db* \ + && rpm --rebuilddb \ && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 \ --setopt=install_weak_deps=false --setopt=tsflags=nodocs \ builddep -y --spec freeipa.spec.in \ (In reply to Lukas Slebodnik from comment #5) > BTW my workaround is to remove && rebuild rpm-db before calling builddep > > --- a/Dockerfile 2016-11-21 16:24:25.994623950 +0100 > +++ b/Dockerfile 2016-11-21 14:53:04.466568545 +0100 > @@ -11,6 +11,8 @@ > @buildsys-build \ > && curl > https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in \ > -o freeipa.spec.in \ > + && rm -f /var/lib/rpm/__db* \ > + && rpm --rebuilddb \ > && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 \ > --setopt=install_weak_deps=false --setopt=tsflags=nodocs \ > builddep -y --spec freeipa.spec.in \ Florian F. mentioned on IRC, that simpler workaround is just to remove var/lib/rpm/__db* files. Calling "rpm --rebuilddb" is not necessary. That sounds like there's a db cursor not getting freed somewhere. (too early in the morning...) Output of 'db_stat -CA -h /var/lib/rpm' as root when it hangs would be interesting. Created attachment 1222632 [details]
Output of db_stat -CA -h /var/lib/rpm
Here you are
FYI I also noticed that fedora rawhide has never version of libdb 5.3.28-15 which should fix mutexes not being released properly (#1272680). I downgraded libdb to older version without this patch but it did not fix hangs. I hit this bug yesterday on my laptop when I was upgrading f25 -> rawhide. I had to cancel/kill dnf and as a side effect I had installed more version of packages because upgrade transaction has not been finished. Okay I can reproduce something remotely resembling that now. I'm on F24'ish system so details will differ but there are hints at least. # yum-deprecated --releasever=25 --installroot /srv/test --setopt=keepcache=true install bash [...] # db_stat -CA -h /srv/test/var/lib/rpm db_stat: DB_ENV->open: /srv/test/var/lib/rpm: No such file or directory That is how it should be, rpm removes the BDB environment after chroot operations. That's what happens with rpm itself (obviously). However doing the same with dnf: # dnf --releasever=25 --installroot /srv/test --setopt=keepcache=true install bash [...] # db_stat -CA -h /srv/test/var/lib/rpm|wc -l 162 In this case they're all READ locks which rpm will mope up on the next run but it'll complain loudly. I know I've seen fleeting glimpses of those complaints where they shouldn't have existed, just haven't had a chance to investigate. But the screwy thing here is that the environment should not even exist after dnf exits! So it seems dnf is (re)opening the rpmdb from inside the chroot which is ... not a good idea. This is with dnf 1.1.10 on F24, IIRC F25 has 2.0 and newer hawkey/whatever so there's plenty of room for different behavior. You said you can reproduce it with yum-deprecated, but can you actually reproduce the hang if you *only* use yum-deprecated all the way? (ie after using "rm -f /var/lib/rpm/__db*" to clear any cruft from earlier runs) Another datapoint: for my testcase the problem is related to importing pubkeys. If I run dnf with --nogpgcheck or pre-import the keys to the test root, there are no leftover locks. Please test whether that makes a difference for you - its possible I'm seeing a different issue entirely, but its also quite possible it's the same issue but only worse with newer dnf. Right, never mind about the pubkey case, that appears to be relatively harmless. What kills it is any outside access to rpmdb during the transaction where glibc is updated to the rawhide version. For example 'db_stat -CA -h /var/lib/rpm' is fine until glibc has gotten updated, after that trying to run it causes both rpm and the db_stat process tohang. Ditto for running 'rpm -qa' or such (as root) durin the update. That does not happen during eg F24 -> F25 glibc update. fweimer (cc'd): has there been some major futex-related change in rawhide glibc? Oh, actually it's simpler than that: after updating to rawhide glibc, new rpm is dead until you nuke the BDB environment: 'rm -f /var/lib/rpm/__db*' Oops, not "new rpm". Just "rpm is dead" until... (In reply to Panu Matilainen from comment #14) > Right, never mind about the pubkey case, that appears to be relatively > harmless. > > What kills it is any outside access to rpmdb during the transaction where > glibc is updated to the rawhide version. For example 'db_stat -CA -h > /var/lib/rpm' is fine until glibc has gotten updated, after that trying to > run it causes both rpm and the db_stat process tohang. Ditto for running > 'rpm -qa' or such (as root) durin the update. That does not happen during eg > F24 -> F25 glibc update. > > fweimer (cc'd): has there been some major futex-related change in rawhide > glibc? This looks like bug 1394862. Berkeley DB assumes that it's safe to use process-shared mutexes and condition variables after all processes have detached from a mapping, something which POSIX does not guarantee. (In reply to Florian Weimer from comment #17) > This looks like bug 1394862. Oh, indeed. Thanks for the pointer. > Berkeley DB assumes that it's safe to use > process-shared mutexes and condition variables after all processes have > detached from a mapping, something which POSIX does not guarantee. Right. It's not the first time such glibc changes cause trouble for rpmdb, just been a while since the last one and BDB locks are still just as ill-suited for rpm as they ever were... Technically this is a dupe of bug 1394862 but I'm leaving this open for now: 1) people are more likely to find an rpm bug than a libdb one when installation hangs 2) we might need to come up with some kind of workaround, we can't have every other upgrade blowing up in the middle As for 2), we could have a trigger on glibc to always nuke the environment in that case, but I worry what it does to the already running rpm instance. Or we could have rpm use private bdb locking but that opens up other issues. Or something... This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle. Changing version to '26'. *** Bug 1435927 has been marked as a duplicate of this bug. *** To interject with a bit of blocker bureaucracy: I suggest we make this an Accepted0Day+AcceptedPreviousRelease blocker rather than an AcceptedBlocker. It would still prevent us shipping Beta, but we could at least build and test an RC and ensure there are no *other* blockers. To refresh folks' memories: Accepted0Day means an update must be pushed for the release in question by Beta release day. AcceptedPreviousRelease means an update must be pushed for relevant *previous* releases by Beta release day. This would be both since it's likely we need updates for both the previous stable releases (24/25) *and* the new release (26) to ensure a successful upgrade. If that winds up not being the case with the solution we pick, we can drop whichever status is no longer relevant. Sound OK to everyone? Any objections? +1 Accepted0Day +1 AcceptedPreviousRelease +1 on both counts for me as well. +1 Accepted0Day +1 AcceptedPreviousRelease Whoops, this one isn't actually the blocker at present...but I think it makes more sense to make this specifically a blocker, rather than the tracker bug. Giving blocker status to a tracker bug doesn't make a lot of sense as its 'meaning' can change over time. So I'll transfer the status from the tracker bug and make it accepted0day and acceptedpreviousrelease. Thanks folks! For the record: we're transferring blocker status from https://bugzilla.redhat.com/show_bug.cgi?id=1443415 to this bug, as it was really this issue specifically we voted on as a blocker. Yes, dnf is merely a passanger in all this. The deeper bug here is bug 1394862 of course but if we can't find a workable solution there we might need to (additionally) apply some workaround on rpm-side so it's just as well this is the actualy blocker. Yep, that was my thought exactly in making this rather than 1394862 the blocker. libdb-5.3.28-21.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27 libdb-5.3.28-21.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27 does this also need to be sent out for 24/25? It'd be really great if people could help test the new update hard: test that it fixes the actual bug, test it in the scenarios that caused trouble with -18 and -19, etc. We want to get this fix out for Beta, but we don't want to send it unless we're sure it's good. Thanks a lot! (In reply to Adam Williamson from comment #31) > It'd be really great if people could help test the new update hard: test > that it fixes the actual bug, test it in the scenarios that caused trouble > with -18 and -19, etc. We want to get this fix out for Beta, but we don't > want to send it unless we're sure it's good. Thanks a lot! I tried dnf update https://kojipkgs.fedoraproject.org//packages/libdb/5.3.28/21.fc26/x86_64/libdb-5.3.28-21.fc26.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/libdb/5.3.28/21.fc26/x86_64/libdb-utils-5.3.28-21.fc26.x86_64.rpm And I could not see any problem with installed package openldap-servers Hi! I am not sure whether this might be related to the issue reported here. But in doubt I better put it on here so it can be verified. I am running what will become Fedora 26 (with all the latest updates that got pushed to the repositories). Around 2 weeks ago I was compiling some Xfce packages using an automated bash script. What it does is pulling in the tar files, adjusting some Fedora specs and pulling in needed dependencies using dnf builddep. This script runs for a few years without errors but recently I saw that the RPM database often gets locked and crashed. Today I tried to re-run the script again and ran into the same RPM database crash again. Please allow me to add the logfile so you can see whether this might be related to the RPM issue (directly or indirectly). The log is full of this: RPM build errors: db5 error(5) from dbenv->open: Input/output error cannot open Packages index using db5 - Input/output error (5) cannot open Packages database in /var/lib/rpm Bad exit status from /var/tmp/rpm-tmp.GjSmLj (%build) error: db5 error(5) from dbenv->open: Input/output error error: cannot open Packages index using db5 - Input/output error (5) error: cannot open Packages database in /var/lib/rpm error: db5 error(5) from dbenv->open: Input/output error error: cannot open Packages index using db5 - Input/output error (5) error: cannot open Packages database in /var/lib/rpm error: db5 error(5) from dbenv->open: Input/output error error: cannot open Packages index using db5 - Input/output error (5) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed The only thing happens here is just running dnf builddep that pulls in all the requirements to build the packages.... Created attachment 1284488 [details]
Please grep for "db5 error"
Upgraded: libdb.x86_64 5.3.28-21.fc26 libdb-utils.x86_64 5.3.28-21.fc26 Complete! [root@f26beta14 folder]# rpm -qa | tail -n 2 fedora-release-notes-25.01-2.fc26.noarch passwdqc-lib-1.3.0-7.fc26.x86_64 [root@f26beta14 folder]# Karma given I just like to inform that I updated to the latest libdb packages (manually). Unfortunately within the process of building packages, I found out that some Fedora 26 related packages (during resolving builddep) downgrades libdb back to the -17 version, which in the flow of building the packages lead into a db5 error again. It would be nice if you could guarantee, that no Fedora 26 package enforces a downgrade (in my case I've instructed dnf builddep to use --allowerasing). What I mean by this can be described as follows: Packages and versions of packages tend to change over time. But quite often building (or re-building) an RPM SRC packages have a strict dependency for older versions of a package (most likely the one it initially was compiled against with). In a flow of dependency resolving (one package resolves the requirements of another) this might cause (quite often) the issue that one or other *required* packages (for building things) wants an older version. dnf can keep up doing this by using --allowerasing so it can resolve conflicts on its own and downgrade the packages as needed. Usually the downgrades are required for some header files. After compiling the packages they usually run perfectly with never versions of any libraries. But if the downgrading also requires that an older version of the library must be installed back (as the example below) then there is a risk, that on building farms and building machines, that the issue with db5 crashes comes back (as it happened a few moments ago). So one of the packages below (in the dependency tree) enforces a downgrade of libdb (I haven't verified this yet!!!). Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: dbus-glib-devel x86_64 0.108-2.fc26 fedora 73 k perl-ExtUtils-Depends noarch 0.405-4.fc26 fedora 30 k perl-ExtUtils-MakeMaker noarch 7.24-2.fc26 fedora 295 k perl-ExtUtils-PkgConfig noarch 1.15-7.fc26 fedora 19 k perl-Glib x86_64 1.324-2.fc26 fedora 368 k perl-Glib-devel x86_64 1.324-2.fc26 fedora 70 k perl-devel x86_64 4:5.24.1-390.fc26 updates-testing 575 k perl-generators noarch 1.10-2.fc26 fedora 16 k Installing dependencies: gdbm-devel x86_64 1.13-1.fc26 updates-testing 62 k libdb-devel x86_64 5.3.28-17.fc26 fedora 42 k perl-ExtUtils-Command noarch 7.24-2.fc26 fedora 17 k perl-ExtUtils-Install noarch 2.04-367.fc26 fedora 44 k perl-ExtUtils-Manifest noarch 1.70-366.fc26 fedora 35 k perl-ExtUtils-ParseXS noarch 1:3.31-368.fc26 fedora 81 k perl-Fedora-VSP noarch 0.001-5.fc26 fedora 23 k perl-Storable x86_64 1:2.56-368.fc26 fedora 83 k perl-Test-Harness noarch 3.39-1.fc26 updates-testing 276 k perl-version x86_64 5:0.99.18-1.fc26 updates-testing 64 k systemtap-sdt-devel x86_64 3.1-5.fc26 updates-testing 71 k Downgrading: libdb x86_64 5.3.28-17.fc26 fedora 748 k libdb-utils x86_64 5.3.28-17.fc26 fedora 137 k Transaction Summary ================================================================================ Install 19 Packages Downgrade 2 Packages (In reply to Ali Akcaagac from comment #36) > So one of the packages below (in the dependency tree) enforces a downgrade > of libdb (I haven't verified this yet!!!). > Installing dependencies: > gdbm-devel x86_64 1.13-1.fc26 updates-testing > 62 k > libdb-devel x86_64 5.3.28-17.fc26 fedora > 42 k It's likely that libdb-devel in its newer version just isn't available from the configured repositories, so in order to install the package, the already-upgraded subpackages are downgraded to match the available package version. (In reply to Florian Weimer from comment #37) > It's likely that libdb-devel in its newer version just isn't available from > the configured repositories, so in order to install the package, the > already-upgraded subpackages are downgraded to match the available package > version. Yes you are right! This was exactly the case here after re-investigating into the issue... I am a a bit overworked recently and was focussing on the db5 issue only... libdb-5.3.28-21.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27 So by all accounts -21 is looking pretty good as a fix for this. I propose we go ahead and push it stable for all releases. Could people please karma the F24 and F25 updates, so long as you're happy with them, so they can be pushed out? Thanks. libdb-5.3.28-21.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf libdb-5.3.28-21.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444 libdb-5.3.28-21.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report. libdb-5.3.28-21.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report. libdb-5.3.28-21.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report. Unfortunately, while no-one reported such issues in the last week or so, now -21 has gone stable, I've seen several reports of problems very much along the lines we saw with -18 and -19. Here's one from Eric Blake (hope you don't mind me quoting it, Eric): ========== On an F25 box, I ran 'dnf update', and it installed libdb.x86_64 5.3.28-21.fc25 among others. That run of 'dnf' dumped core, and all subsequent runs of dnf were unhappy: # dnf update ... Verifying : libdb-5.3.28-21.fc25.x86_64 18/80 ... Complete! Segmentation fault (core dumped) # dnf update error: rpmdb: BDB0113 Thread/process 6173/140680480651008 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed I don't know how many others will have hosed systems now that libdb appears to have been pushed to stable, but it's a bummer that it happened to me. ========= Can we please look into this again as a matter of urgency? It's obviously bad to be breaking F24/F25 systems. Perhaps we should look into removing the update and/or doing a reverted -22 as an exceptional step also. Eric, can you get a backtrace from the core dump if possible? Can you also provide a full list of what was in the update transaction (the entire console log would be great if you still have it, otherwise dnf history should do)? Thanks a lot. [root@red eblake]# dnf update Last metadata expiration check: 0:47:27 ago on Fri Jun 9 09:07:41 2017. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: kernel x86_64 4.11.3-202.fc25 updates 100 k kernel-core x86_64 4.11.3-202.fc25 updates 21 M kernel-debug-devel x86_64 4.11.3-202.fc25 updates 11 M kernel-modules x86_64 4.11.3-202.fc25 updates 23 M python-qpid-proton x86_64 0.17.0-2.fc25 updates 220 k python-rhmsg noarch 0.5-1.fc25 rhpkg 27 k python2-simplejson x86_64 3.10.0-1.fc25 updates 285 k qpid-proton-c x86_64 0.17.0-2.fc25 updates 159 k rpmdiff x86_64 4.1.1-1.fc25 rhpkg 274 k rpmdiff-remote x86_64 4.1.1-1.fc25 rhpkg 132 k Upgrading: google-noto-emoji-fonts noarch 20170523-1.fc25 updates 4.4 M hplip x86_64 3.17.4-2.fc25 updates 13 M hplip-common x86_64 3.17.4-2.fc25 updates 102 k hplip-libs x86_64 3.17.4-2.fc25 updates 191 k hwdata noarch 0.301-1.fc25 updates 1.4 M kernel-headers x86_64 4.11.3-202.fc25 updates 1.1 M libdb x86_64 5.3.28-21.fc25 updates 749 k libdb-devel x86_64 5.3.28-21.fc25 updates 43 k libdb-utils x86_64 5.3.28-21.fc25 updates 138 k libmicrohttpd x86_64 1:0.9.55-1.fc25 updates 77 k libsane-hpaio x86_64 3.17.4-2.fc25 updates 120 k mesa-dri-drivers x86_64 17.0.5-3.fc25 updates 12 M mesa-filesystem x86_64 17.0.5-3.fc25 updates 25 k mesa-libEGL x86_64 17.0.5-3.fc25 updates 104 k mesa-libEGL-devel x86_64 17.0.5-3.fc25 updates 41 k mesa-libGL x86_64 17.0.5-3.fc25 updates 165 k mesa-libGL-devel x86_64 17.0.5-3.fc25 updates 161 k mesa-libGLES x86_64 17.0.5-3.fc25 updates 22 k mesa-libGLES-devel x86_64 17.0.5-3.fc25 updates 63 k mesa-libOpenCL x86_64 17.0.5-3.fc25 updates 581 k mesa-libgbm x86_64 17.0.5-3.fc25 updates 43 k mesa-libgbm-devel x86_64 17.0.5-3.fc25 updates 27 k mesa-libglapi x86_64 17.0.5-3.fc25 updates 50 k mesa-libwayland-egl x86_64 17.0.5-3.fc25 updates 26 k mesa-libwayland-egl-devel x86_64 17.0.5-3.fc25 updates 23 k mesa-libxatracker x86_64 17.0.5-3.fc25 updates 1.4 M ntfs-3g x86_64 2:2017.3.23-1.fc25 updates 273 k ntfsprogs x86_64 2:2017.3.23-1.fc25 updates 382 k perl-Net-HTTP noarch 6.16-1.fc25 updates 41 k perl-threads x86_64 1:2.16-1.fc25 updates 58 k qrencode-libs x86_64 3.4.4-1.fc25 updates 55 k rhpkg noarch 1.32-2.fc25 rhpkg 121 k vim-minimal x86_64 2:8.0.617-1.fc25 updates 520 k Removing: kernel x86_64 4.10.16-200.fc25 @updates 0 kernel-core x86_64 4.10.16-200.fc25 @updates 53 M kernel-debug-devel x86_64 4.10.16-200.fc25 @updates 42 M kernel-modules x86_64 4.10.16-200.fc25 @updates 22 M Transaction Summary ================================================================================ Install 10 Packages Upgrade 33 Packages Remove 4 Packages Total download size: 93 M Is this ok [y/N]: y Downloading Packages: (1/43): hplip-libs-3.17.4-1.fc25_3.17.4-2.fc25. 88 kB/s | 51 kB 00:00 (2/43): hplip-common-3.17.4-1.fc25_3.17.4-2.fc2 320 kB/s | 48 kB 00:00 [DRPM] hplip-libs-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done (3/43): hplip-3.17.4-1.fc25_3.17.4-2.fc25.x86_6 353 kB/s | 346 kB 00:00 [DRPM] hplip-common-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done (4/43): libsane-hpaio-3.17.4-1.fc25_3.17.4-2.fc 144 kB/s | 51 kB 00:00 (5/43): hwdata-0.300-1.fc25_0.301-1.fc25.noarch 45 kB/s | 41 kB 00:00 [DRPM] libsane-hpaio-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done (6/43): kernel-debug-devel-4.11.3-200.fc25_4.11 1.2 MB/s | 2.7 MB 00:02 (7/43): libdb-devel-5.3.28-16.fc25_5.3.28-21.fc 138 kB/s | 10 kB 00:00 (8/43): libdb-5.3.28-16.fc25_5.3.28-21.fc25.x86 260 kB/s | 129 kB 00:00 [DRPM] libdb-devel-5.3.28-16.fc25_5.3.28-21.fc25.x86_64.drpm: done (9/43): libmicrohttpd-0.9.54-1.fc25_0.9.55-1.fc 243 kB/s | 32 kB 00:00 (10/43): mesa-libEGL-17.0.5-2.fc25_17.0.5-3.fc2 343 kB/s | 28 kB 00:00 [DRPM] libmicrohttpd-0.9.54-1.fc25_0.9.55-1.fc25.x86_64.drpm: done (11/43): mesa-libGL-17.0.5-2.fc25_17.0.5-3.fc25 438 kB/s | 39 kB 00:00 [DRPM] mesa-libEGL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done (12/43): mesa-libglapi-17.0.5-2.fc25_17.0.5-3.f 265 kB/s | 21 kB 00:00 (13/43): mesa-libGL-devel-17.0.5-2.fc25_17.0.5- 263 kB/s | 22 kB 00:00 [DRPM] mesa-libGL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done [DRPM] mesa-libglapi-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done (14/43): kernel-headers-4.11.3-200.fc25_4.11.3- 116 kB/s | 208 kB 00:01 (15/43): mesa-libGLES-devel-17.0.5-2.fc25_17.0. 222 kB/s | 22 kB 00:00 [DRPM] libdb-5.3.28-16.fc25_5.3.28-21.fc25.x86_64.drpm: done (16/43): mesa-libgbm-17.0.5-2.fc25_17.0.5-3.fc2 215 kB/s | 21 kB 00:00 [DRPM] hwdata-0.300-1.fc25_0.301-1.fc25.noarch.drpm: done [DRPM] mesa-libGL-devel-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done [DRPM] mesa-libGLES-devel-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done (17/43): mesa-libOpenCL-17.0.5-2.fc25_17.0.5-3. 114 kB/s | 22 kB 00:00 [DRPM] mesa-libgbm-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done (18/43): perl-Net-HTTP-6.15-1.fc25_6.16-1.fc25. 124 kB/s | 18 kB 00:00 (19/43): mesa-libxatracker-17.0.5-2.fc25_17.0.5 632 kB/s | 161 kB 00:00 [DRPM] perl-Net-HTTP-6.15-1.fc25_6.16-1.fc25.noarch.drpm: done (20/43): qrencode-libs-3.4.2-6.fc24_3.4.4-1.fc2 219 kB/s | 18 kB 00:00 (21/43): perl-threads-2.15-1.fc25_2.16-1.fc25.x 131 kB/s | 23 kB 00:00 [DRPM] mesa-libOpenCL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done [DRPM] qrencode-libs-3.4.2-6.fc24_3.4.4-1.fc25.x86_64.drpm: done (22/43): kernel-4.11.3-202.fc25.x86_64.rpm 176 kB/s | 100 kB 00:00 (23/43): mesa-dri-drivers-17.0.5-2.fc25_17.0.5- 502 kB/s | 1.1 MB 00:02 [DRPM] kernel-headers-4.11.3-200.fc25_4.11.3-202.fc25.x86_64.drpm: done [DRPM] mesa-libxatracker-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done [DRPM] perl-threads-2.15-1.fc25_2.16-1.fc25.x86_64.drpm: done (24/43): python-rhmsg-0.5-1.fc25.noarch.rpm 60 kB/s | 27 kB 00:00 (25/43): rpmdiff-remote-4.1.1-1.fc25.x86_64.rpm 376 kB/s | 132 kB 00:00 (26/43): rpmdiff-4.1.1-1.fc25.x86_64.rpm 615 kB/s | 274 kB 00:00 (27/43): python2-simplejson-3.10.0-1.fc25.x86_6 411 kB/s | 285 kB 00:00 (28/43): python-qpid-proton-0.17.0-2.fc25.x86_6 459 kB/s | 220 kB 00:00 (29/43): qpid-proton-c-0.17.0-2.fc25.x86_64.rpm 406 kB/s | 159 kB 00:00 (30/43): google-noto-emoji-fonts-20170523-1.fc2 700 kB/s | 4.4 MB 00:06 (31/43): libdb-utils-5.3.28-21.fc25.x86_64.rpm 531 kB/s | 138 kB 00:00 (32/43): mesa-filesystem-17.0.5-3.fc25.x86_64.r 268 kB/s | 25 kB 00:00 (33/43): mesa-libEGL-devel-17.0.5-3.fc25.x86_64 466 kB/s | 41 kB 00:00 (34/43): mesa-libGLES-17.0.5-3.fc25.x86_64.rpm 285 kB/s | 22 kB 00:00 (35/43): mesa-libgbm-devel-17.0.5-3.fc25.x86_64 333 kB/s | 27 kB 00:00 (36/43): mesa-libwayland-egl-17.0.5-3.fc25.x86_ 198 kB/s | 26 kB 00:00 (37/43): mesa-libwayland-egl-devel-17.0.5-3.fc2 164 kB/s | 23 kB 00:00 (38/43): ntfs-3g-2017.3.23-1.fc25.x86_64.rpm 562 kB/s | 273 kB 00:00 (39/43): ntfsprogs-2017.3.23-1.fc25.x86_64.rpm 734 kB/s | 382 kB 00:00 (40/43): rhpkg-1.32-2.fc25.noarch.rpm 313 kB/s | 121 kB 00:00 (41/43): vim-minimal-8.0.617-1.fc25.x86_64.rpm 708 kB/s | 520 kB 00:00 (42/43): kernel-core-4.11.3-202.fc25.x86_64.rpm 809 kB/s | 21 MB 00:26 (43/43): kernel-modules-4.11.3-202.fc25.x86_64. 910 kB/s | 23 MB 00:25 [DRPM] hplip-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done [DRPM] kernel-debug-devel-4.11.3-200.fc25_4.11.3-202.fc25.x86_64.drpm: done [DRPM] mesa-dri-drivers-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done -------------------------------------------------------------------------------- Total 1.6 MB/s | 56 MB 00:35 Delta RPMs reduced 93.2 MB of updates to 55.8 MB (40.1% saved) Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Upgrading : mesa-libglapi-17.0.5-3.fc25.x86_64 1/80 Upgrading : mesa-libgbm-17.0.5-3.fc25.x86_64 2/80 Upgrading : libdb-5.3.28-21.fc25.x86_64 3/80 Installing : kernel-core-4.11.3-202.fc25.x86_64 4/80 Installing : kernel-modules-4.11.3-202.fc25.x86_64 5/80 Upgrading : mesa-libEGL-17.0.5-3.fc25.x86_64 6/80 Upgrading : mesa-libGL-17.0.5-3.fc25.x86_64 7/80 Upgrading : mesa-libGLES-17.0.5-3.fc25.x86_64 8/80 Upgrading : ntfs-3g-2:2017.3.23-1.fc25.x86_64 9/80 Upgrading : mesa-libwayland-egl-17.0.5-3.fc25.x86_64 10/80 Upgrading : mesa-filesystem-17.0.5-3.fc25.x86_64 11/80 Upgrading : hplip-common-3.17.4-2.fc25.x86_64 12/80 Upgrading : hplip-libs-3.17.4-2.fc25.x86_64 13/80 Installing : qpid-proton-c-0.17.0-2.fc25.x86_64 14/80 Installing : python-qpid-proton-0.17.0-2.fc25.x86_64 15/80 Installing : python-rhmsg-0.5-1.fc25.noarch 16/80 Installing : python2-simplejson-3.10.0-1.fc25.x86_64 17/80 Installing : rpmdiff-4.1.1-1.fc25.x86_64 18/80 Installing : rpmdiff-remote-4.1.1-1.fc25.x86_64 19/80 Upgrading : rhpkg-1.32-2.fc25.noarch 20/80 Upgrading : hplip-3.17.4-2.fc25.x86_64 21/80 Upgrading : libsane-hpaio-3.17.4-2.fc25.x86_64 22/80 Upgrading : mesa-dri-drivers-17.0.5-3.fc25.x86_64 23/80 Upgrading : mesa-libwayland-egl-devel-17.0.5-3.fc25.x86_64 24/80 Upgrading : ntfsprogs-2:2017.3.23-1.fc25.x86_64 25/80 Upgrading : mesa-libGLES-devel-17.0.5-3.fc25.x86_64 26/80 Upgrading : mesa-libGL-devel-17.0.5-3.fc25.x86_64 27/80 Upgrading : mesa-libEGL-devel-17.0.5-3.fc25.x86_64 28/80 Installing : kernel-4.11.3-202.fc25.x86_64 29/80 Upgrading : libdb-utils-5.3.28-21.fc25.x86_64 30/80 Upgrading : libdb-devel-5.3.28-21.fc25.x86_64 31/80 Upgrading : mesa-libOpenCL-17.0.5-3.fc25.x86_64 32/80 Upgrading : mesa-libgbm-devel-17.0.5-3.fc25.x86_64 33/80 Upgrading : vim-minimal-2:8.0.617-1.fc25.x86_64 34/80 Upgrading : qrencode-libs-3.4.4-1.fc25.x86_64 35/80 Upgrading : perl-threads-1:2.16-1.fc25.x86_64 36/80 Upgrading : perl-Net-HTTP-6.16-1.fc25.noarch 37/80 Upgrading : mesa-libxatracker-17.0.5-3.fc25.x86_64 38/80 Upgrading : libmicrohttpd-1:0.9.55-1.fc25.x86_64 39/80 Upgrading : kernel-headers-4.11.3-202.fc25.x86_64 40/80 Upgrading : hwdata-0.301-1.fc25.noarch 41/80 Upgrading : google-noto-emoji-fonts-20170523-1.fc25.noarch 42/80 Installing : kernel-debug-devel-4.11.3-202.fc25.x86_64 43/80 Erasing : kernel-4.10.16-200.fc25.x86_64 44/80 Cleanup : mesa-libGL-devel-17.0.5-2.fc25.x86_64 45/80 Cleanup : mesa-libwayland-egl-devel-17.0.5-2.fc25.x86_64 46/80 Cleanup : mesa-libgbm-devel-17.0.5-2.fc25.x86_64 47/80 Cleanup : mesa-libGLES-devel-17.0.5-2.fc25.x86_64 48/80 Cleanup : mesa-libGLES-17.0.5-2.fc25.x86_64 49/80 Cleanup : mesa-libEGL-devel-17.0.5-2.fc25.x86_64 50/80 Cleanup : libdb-devel-5.3.28-16.fc25.x86_64 51/80 Cleanup : libsane-hpaio-3.17.4-1.fc25.x86_64 52/80 Cleanup : hplip-3.17.4-1.fc25.x86_64 53/80 Cleanup : hplip-libs-3.17.4-1.fc25.x86_64 54/80 Cleanup : hplip-common-3.17.4-1.fc25.x86_64 55/80 Erasing : kernel-debug-devel-4.10.16-200.fc25.x86_64 56/80 Cleanup : rhpkg-1.31-1.fc25.noarch 57/80 Cleanup : perl-Net-HTTP-6.15-1.fc25.noarch 58/80 Cleanup : kernel-headers-4.11.3-200.fc25.x86_64 59/80 Cleanup : hwdata-0.300-1.fc25.noarch 60/80 Cleanup : google-noto-emoji-fonts-20170426-1.fc25.noarch 61/80 Cleanup : mesa-dri-drivers-17.0.5-2.fc25.x86_64 62/80 Cleanup : mesa-libEGL-17.0.5-2.fc25.x86_64 63/80 Cleanup : mesa-libGL-17.0.5-2.fc25.x86_64 64/80 Erasing : kernel-modules-4.10.16-200.fc25.x86_64 65/80 Cleanup : ntfsprogs-2:2016.2.22-4.fc25.x86_64 66/80 Cleanup : mesa-libOpenCL-17.0.5-2.fc25.x86_64 67/80 Cleanup : libdb-utils-5.3.28-16.fc25.x86_64 68/80 Cleanup : mesa-filesystem-17.0.5-2.fc25.x86_64 69/80 Cleanup : libdb-5.3.28-16.fc25.x86_64 70/80 Cleanup : mesa-libgbm-17.0.5-2.fc25.x86_64 71/80 Cleanup : ntfs-3g-2:2016.2.22-4.fc25.x86_64 72/80 Erasing : kernel-core-4.10.16-200.fc25.x86_64 73/80 Cleanup : mesa-libglapi-17.0.5-2.fc25.x86_64 74/80 Cleanup : mesa-libwayland-egl-17.0.5-2.fc25.x86_64 75/80 Cleanup : vim-minimal-2:8.0.600-1.fc25.x86_64 76/80 Cleanup : qrencode-libs-3.4.2-6.fc24.x86_64 77/80 Cleanup : perl-threads-1:2.15-1.fc25.x86_64 78/80 Cleanup : mesa-libxatracker-17.0.5-2.fc25.x86_64 79/80 Cleanup : libmicrohttpd-1:0.9.54-1.fc25.x86_64 80/80 Verifying : kernel-core-4.11.3-202.fc25.x86_64 1/80 Verifying : kernel-4.11.3-202.fc25.x86_64 2/80 Verifying : kernel-debug-devel-4.11.3-202.fc25.x86_64 3/80 Verifying : kernel-modules-4.11.3-202.fc25.x86_64 4/80 Verifying : python-rhmsg-0.5-1.fc25.noarch 5/80 Verifying : rpmdiff-remote-4.1.1-1.fc25.x86_64 6/80 Verifying : rpmdiff-4.1.1-1.fc25.x86_64 7/80 Verifying : python2-simplejson-3.10.0-1.fc25.x86_64 8/80 Verifying : python-qpid-proton-0.17.0-2.fc25.x86_64 9/80 Verifying : qpid-proton-c-0.17.0-2.fc25.x86_64 10/80 Verifying : google-noto-emoji-fonts-20170523-1.fc25.noarch 11/80 Verifying : hplip-3.17.4-2.fc25.x86_64 12/80 Verifying : hplip-libs-3.17.4-2.fc25.x86_64 13/80 Verifying : hplip-common-3.17.4-2.fc25.x86_64 14/80 Verifying : libsane-hpaio-3.17.4-2.fc25.x86_64 15/80 Verifying : hwdata-0.301-1.fc25.noarch 16/80 Verifying : kernel-headers-4.11.3-202.fc25.x86_64 17/80 Verifying : libdb-5.3.28-21.fc25.x86_64 18/80 Verifying : libdb-utils-5.3.28-21.fc25.x86_64 19/80 Verifying : libdb-devel-5.3.28-21.fc25.x86_64 20/80 Verifying : libmicrohttpd-1:0.9.55-1.fc25.x86_64 21/80 Verifying : mesa-dri-drivers-17.0.5-3.fc25.x86_64 22/80 Verifying : mesa-filesystem-17.0.5-3.fc25.x86_64 23/80 Verifying : mesa-libEGL-17.0.5-3.fc25.x86_64 24/80 Verifying : mesa-libEGL-devel-17.0.5-3.fc25.x86_64 25/80 Verifying : mesa-libGL-17.0.5-3.fc25.x86_64 26/80 Verifying : mesa-libglapi-17.0.5-3.fc25.x86_64 27/80 Verifying : mesa-libGL-devel-17.0.5-3.fc25.x86_64 28/80 Verifying : mesa-libGLES-17.0.5-3.fc25.x86_64 29/80 Verifying : mesa-libGLES-devel-17.0.5-3.fc25.x86_64 30/80 Verifying : mesa-libOpenCL-17.0.5-3.fc25.x86_64 31/80 Verifying : mesa-libgbm-17.0.5-3.fc25.x86_64 32/80 Verifying : mesa-libgbm-devel-17.0.5-3.fc25.x86_64 33/80 Verifying : mesa-libwayland-egl-17.0.5-3.fc25.x86_64 34/80 Verifying : mesa-libwayland-egl-devel-17.0.5-3.fc25.x86_64 35/80 Verifying : mesa-libxatracker-17.0.5-3.fc25.x86_64 36/80 Verifying : ntfs-3g-2:2017.3.23-1.fc25.x86_64 37/80 Verifying : ntfsprogs-2:2017.3.23-1.fc25.x86_64 38/80 Verifying : perl-Net-HTTP-6.16-1.fc25.noarch 39/80 Verifying : perl-threads-1:2.16-1.fc25.x86_64 40/80 Verifying : qrencode-libs-3.4.4-1.fc25.x86_64 41/80 Verifying : rhpkg-1.32-2.fc25.noarch 42/80 Verifying : vim-minimal-2:8.0.617-1.fc25.x86_64 43/80 Verifying : qrencode-libs-3.4.2-6.fc24.x86_64 44/80 Verifying : libdb-5.3.28-16.fc25.x86_64 45/80 Verifying : libsane-hpaio-3.17.4-1.fc25.x86_64 46/80 Verifying : libdb-utils-5.3.28-16.fc25.x86_64 47/80 Verifying : libdb-devel-5.3.28-16.fc25.x86_64 48/80 Verifying : ntfs-3g-2:2016.2.22-4.fc25.x86_64 49/80 Verifying : ntfsprogs-2:2016.2.22-4.fc25.x86_64 50/80 Verifying : libmicrohttpd-1:0.9.54-1.fc25.x86_64 51/80 Verifying : vim-minimal-2:8.0.600-1.fc25.x86_64 52/80 Verifying : kernel-4.10.16-200.fc25.x86_64 53/80 Verifying : rhpkg-1.31-1.fc25.noarch 54/80 Verifying : kernel-core-4.10.16-200.fc25.x86_64 55/80 Verifying : kernel-debug-devel-4.10.16-200.fc25.x86_64 56/80 Verifying : perl-Net-HTTP-6.15-1.fc25.noarch 57/80 Verifying : kernel-headers-4.11.3-200.fc25.x86_64 58/80 Verifying : kernel-modules-4.10.16-200.fc25.x86_64 59/80 Verifying : hplip-3.17.4-1.fc25.x86_64 60/80 Verifying : hplip-common-3.17.4-1.fc25.x86_64 61/80 Verifying : hplip-libs-3.17.4-1.fc25.x86_64 62/80 Verifying : hwdata-0.300-1.fc25.noarch 63/80 Verifying : google-noto-emoji-fonts-20170426-1.fc25.noarch 64/80 Verifying : mesa-dri-drivers-17.0.5-2.fc25.x86_64 65/80 Verifying : mesa-filesystem-17.0.5-2.fc25.x86_64 66/80 Verifying : mesa-libEGL-devel-17.0.5-2.fc25.x86_64 67/80 Verifying : mesa-libGL-17.0.5-2.fc25.x86_64 68/80 Verifying : mesa-libEGL-17.0.5-2.fc25.x86_64 69/80 Verifying : mesa-libGL-devel-17.0.5-2.fc25.x86_64 70/80 Verifying : mesa-libGLES-17.0.5-2.fc25.x86_64 71/80 Verifying : mesa-libGLES-devel-17.0.5-2.fc25.x86_64 72/80 Verifying : perl-threads-1:2.15-1.fc25.x86_64 73/80 Verifying : mesa-libOpenCL-17.0.5-2.fc25.x86_64 74/80 Verifying : mesa-libgbm-17.0.5-2.fc25.x86_64 75/80 Verifying : mesa-libgbm-devel-17.0.5-2.fc25.x86_64 76/80 Verifying : mesa-libglapi-17.0.5-2.fc25.x86_64 77/80 Verifying : mesa-libwayland-egl-17.0.5-2.fc25.x86_64 78/80 Verifying : mesa-libwayland-egl-devel-17.0.5-2.fc25.x86_64 79/80 Verifying : mesa-libxatracker-17.0.5-2.fc25.x86_64 80/80 Removed: kernel.x86_64 4.10.16-200.fc25 kernel-core.x86_64 4.10.16-200.fc25 kernel-debug-devel.x86_64 4.10.16-200.fc25 kernel-modules.x86_64 4.10.16-200.fc25 Installed: kernel.x86_64 4.11.3-202.fc25 kernel-core.x86_64 4.11.3-202.fc25 kernel-debug-devel.x86_64 4.11.3-202.fc25 kernel-modules.x86_64 4.11.3-202.fc25 python-qpid-proton.x86_64 0.17.0-2.fc25 python-rhmsg.noarch 0.5-1.fc25 python2-simplejson.x86_64 3.10.0-1.fc25 qpid-proton-c.x86_64 0.17.0-2.fc25 rpmdiff.x86_64 4.1.1-1.fc25 rpmdiff-remote.x86_64 4.1.1-1.fc25 Upgraded: google-noto-emoji-fonts.noarch 20170523-1.fc25 hplip.x86_64 3.17.4-2.fc25 hplip-common.x86_64 3.17.4-2.fc25 hplip-libs.x86_64 3.17.4-2.fc25 hwdata.noarch 0.301-1.fc25 kernel-headers.x86_64 4.11.3-202.fc25 libdb.x86_64 5.3.28-21.fc25 libdb-devel.x86_64 5.3.28-21.fc25 libdb-utils.x86_64 5.3.28-21.fc25 libmicrohttpd.x86_64 1:0.9.55-1.fc25 libsane-hpaio.x86_64 3.17.4-2.fc25 mesa-dri-drivers.x86_64 17.0.5-3.fc25 mesa-filesystem.x86_64 17.0.5-3.fc25 mesa-libEGL.x86_64 17.0.5-3.fc25 mesa-libEGL-devel.x86_64 17.0.5-3.fc25 mesa-libGL.x86_64 17.0.5-3.fc25 mesa-libGL-devel.x86_64 17.0.5-3.fc25 mesa-libGLES.x86_64 17.0.5-3.fc25 mesa-libGLES-devel.x86_64 17.0.5-3.fc25 mesa-libOpenCL.x86_64 17.0.5-3.fc25 mesa-libgbm.x86_64 17.0.5-3.fc25 mesa-libgbm-devel.x86_64 17.0.5-3.fc25 mesa-libglapi.x86_64 17.0.5-3.fc25 mesa-libwayland-egl.x86_64 17.0.5-3.fc25 mesa-libwayland-egl-devel.x86_64 17.0.5-3.fc25 mesa-libxatracker.x86_64 17.0.5-3.fc25 ntfs-3g.x86_64 2:2017.3.23-1.fc25 ntfsprogs.x86_64 2:2017.3.23-1.fc25 perl-Net-HTTP.noarch 6.16-1.fc25 perl-threads.x86_64 1:2.16-1.fc25 qrencode-libs.x86_64 3.4.4-1.fc25 rhpkg.noarch 1.32-2.fc25 vim-minimal.x86_64 2:8.0.617-1.fc25 etckeeper warning: hardlinked files could cause problems with git: chromium/native-messaging-hosts/browser_test_resources/mock_client_plugin.js chromium/native-messaging-hosts/browser_test_resources/mock_host_list_api.js chromium/native-messaging-hosts/browser_test_resources/mock_identity.js chromium/native-messaging-hosts/browser_test_resources/mock_signal_strategy.js chromium/native-messaging-hosts/browser_test_resources/sinon.js chromium/native-messaging-hosts/unittests/mock_client_plugin.js chromium/native-messaging-hosts/unittests/mock_host_list_api.js chromium/native-messaging-hosts/unittests/mock_identity.js chromium/native-messaging-hosts/unittests/mock_signal_strategy.js chromium/native-messaging-hosts/unittests/sinonjs/sinon.js Complete! Segmentation fault (core dumped) Where would the core dump be saved, or am I beyond hope there? ABRT didn't seem to see a recent crash If you can't find it in /var/spool/abrt or /var/tmp/abrt or / , probably no chance (I don't think the systemd coredump thingy is enabled on F25)... Other reports: https://bugzilla.redhat.com/show_bug.cgi?id=1394862#c92 "That's not true. I was just hit with -21.fc25 from updates which broke dnf until I ran rpm --rebuilddb manually." - Dominik Mierzejewski, on devel@ Dominik managed to get abrt to file a crash report, in fact, with stack trace: https://bugzilla.redhat.com/show_bug.cgi?id=1460303 I think I saw another report on IRC, but can't find it right now. Okay, I found the core dump in /var/spool/abrt/ccpp-2017-06-09-15\:14\:27-8088/, but I can't seem to make ABRT parse it, and right now I have a lame backtrace: (gdb) bt #0 0x0000559ddacdd528 in ?? () #1 0x00007f41b8cc7f30 in ?? () #2 0x0000559ddc834670 in ?? () #3 0x00007f41b8cc7f50 in ?? () #4 0x0000559ddad040e7 in ?? () #5 0x0000000000000000 in ?? () (In reply to Eric Blake from comment #52) > Okay, I found the core dump in > /var/spool/abrt/ccpp-2017-06-09-15\:14\:27-8088/, but I can't seem to make > ABRT parse it, and right now I have a lame backtrace: > > (gdb) bt > #0 0x0000559ddacdd528 in ?? () > #1 0x00007f41b8cc7f30 in ?? () > #2 0x0000559ddc834670 in ?? () > #3 0x00007f41b8cc7f50 in ?? () > #4 0x0000559ddad040e7 in ?? () > #5 0x0000000000000000 in ?? () Sorry for the noise; I just read ccpp-*/cmdline, and realized that it is an unrelated dump also from today, so it's not helpful here. There's nothing in /var/spool for my dnf crash :( Looking into this right now but from what I can see the issue should be easily fixable by running "rpmdb --rebuilddb" if you already hit the crash. I can confirm that I am seeing the same on one system with -22: $ rpm -q libdb; sudo dnf -y update libdb-5.3.28-22.fc26.x86_64 error: db5 error(5) from dbenv->open: Input/output error error: cannot open Packages index using db5 - Input/output error (5) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed rpm --rebuilddb gets it back in order, but I use dnf-automatic and have found that it's busted more than it is not over the last few days. yes, it's known that it happens on update from -21 to -22, which is why we withdrew -22. So this is an accepted F26 beta blocker, but not yet closed. What's up here? Does it need to be moved to F26 final blocker? It was being left open while we sorted through all the messy consequences of the updates, but the bug covered *here* is in fact fixed by -21. Since we decided to just leave -21 in stable and help people who hit the RPM db issue on update to -21 one-by-one, there's really not a lot left to do here, and I think we can just go ahead and close the bug. |