1397087 – Rpm hangs after updating to glibc >= 2.25

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1397087 - Rpm hangs after updating to glibc >= 2.25

Summary: Rpm hangs after updating to glibc >= 2.25

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	rpm
Sub Component:
Version:	26
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Assignee:	Packaging Maintenance Team
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:	Accepted0Day AcceptedPreviousRelease
Duplicates (1):	1435927 (view as bug list)
Depends On:	1394862
Blocks:	F26BetaBlocker 1443415
TreeView+	depends on / blocked

Reported:	2016-11-21 15:11 UTC by Lukas Slebodnik
Modified:	2017-09-13 14:44 UTC (History)
CC List:	24 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-06-23 19:34:04 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
Dockerfile which reveal the bug very often (672 bytes, text/plain) 2016-11-21 15:36 UTC, Lukas Slebodnik	no flags	Details
Output of db_stat -CA -h /var/lib/rpm (8.52 KB, text/plain) 2016-11-22 09:09 UTC, Lukas Slebodnik	no flags	Details
Please grep for "db5 error" (1.01 MB, text/plain) 2017-06-02 17:08 UTC, Ali Akcaagac	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1394862	unspecified	CLOSED	libdb: Assumes that internal condition variable layout never changes	2022-05-16 11:32:56 UTC
Red Hat Bugzilla	1443610	unspecified	CLOSED	dnf upgrades from f25 to f26 frequently go to sleep	2022-05-16 11:32:56 UTC
Red Hat Bugzilla	1460303	unspecified	CLOSED	[abrt] system-python: __env_in_api(): system-python killed by SIGSEGV	2022-05-16 11:32:56 UTC

Internal Links: 1394862 1443610 1460303

Description Lukas Slebodnik 2016-11-21 15:11:10 UTC

Description of problem:
I do not have a deterministic reproducer. But I happen for me when building
docker images based on rawhide. The bug is reproducible with dnf and yum-deprecated therefore I assume bug is in rpm. But I noticed there are valgrind errors with uninitialized memory which might explain difficult reproducer.

Version-Release number of selected component (if applicable):
rpm-4.13.0-4.fc26.x86_64

How reproducible:
Deterministic in my environment when building docker image
It happened to me that installation was stuck even for installation of simple pacakge (lftp)

Steps to Reproduce:
# docker run -ti --rm fedora:rawhide bash
Run docker images for fedora rawhide and run following commands inside container
1. dnf update -y 'rpm*'
2. dnf install -y 'dnf-command(builddep)' valgrind
3. dnf debuginfo-install -y rpm libdb
# try to install package with minimal dependencies (e.g. lftp)
4. valgrind --track-origins=yes dnf install -y lftp

Actual results:
Total                                                                                                                                                      215 kB/s | 822 kB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : lftp-4.7.3-1.fc25.x86_64                                                                                                                                               1/1 
==83== Syscall param pwrite64(buf) points to uninitialised byte(s)
==83==    at 0x5317DE3: ??? (in /usr/lib64/libpthread-2.24.90.so)
==83==    by 0x12720C30: __os_io (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1270C731: ??? (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==83==    by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so)
==83==    by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so)
==83==    by 0x11F53FF5: dbiSync (db3.c:547)
==83==    by 0x11F53FF5: db3_dbiCursorFree (db3.c:606)
==83==    by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411)
==83==    by 0x11F72A0C: dbAdd (psm.c:551)
==83==    by 0x11F72A0C: rpmPackageInstall (psm.c:682)
==83==    by 0x11F72A0C: rpmpsmRun (psm.c:844)
==83==    by 0x11F862A4: rpmteProcess (rpmte.c:756)
==83==    by 0x11F8CBBD: rpmtsProcess (transaction.c:1361)
==83==    by 0x11F8CBBD: rpmtsRun (transaction.c:1541)
==83==  Address 0x8e59248 is 2,312 bytes inside a block of size 4,096 alloc'd
==83==    at 0x4C2DB8D: malloc (vg_replace_malloc.c:299)
==83==    by 0x1271E1F4: __os_malloc (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1270C897: ??? (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==83==    by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==83==    by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so)
==83==    by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so)
==83==    by 0x11F53FF5: dbiSync (db3.c:547)
==83==    by 0x11F53FF5: db3_dbiCursorFree (db3.c:606)
==83==    by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411)
==83==    by 0x11F72A0C: dbAdd (psm.c:551)
==83==    by 0x11F72A0C: rpmPackageInstall (psm.c:682)
==83==    by 0x11F72A0C: rpmpsmRun (psm.c:844)
==83==    by 0x11F862A4: rpmteProcess (rpmte.c:756)
==83==    by 0x11F8CBBD: rpmtsProcess (transaction.c:1361)
==83==    by 0x11F8CBBD: rpmtsRun (transaction.c:1541)
==83==  Uninitialised value was created by a stack allocation
==83==    at 0x125EB590: ??? (in /usr/lib64/libdb-5.3.so)
==83== 
  Verifying   : lftp-4.7.3-1.fc25.x86_64                                                                                                                                               1/1 

Installed:
  lftp.x86_64 4.7.3-1.fc25                                                                                                                                                                 

Complete!
==83== 
==83== HEAP SUMMARY:
==83==     in use at exit: 58,246,804 bytes in 53,257 blocks
==83==   total heap usage: 447,873 allocs, 394,616 frees, 3,993,424,200 bytes allocated
==83== 
==83== LEAK SUMMARY:
==83==    definitely lost: 345 bytes in 20 blocks
==83==    indirectly lost: 0 bytes in 0 blocks
==83==      possibly lost: 3,164,151 bytes in 18,487 blocks
==83==    still reachable: 55,081,988 bytes in 34,747 blocks
==83==                       of which reachable via heuristic:
==83==                         length64           : 426,096 bytes in 504 blocks
==83==                         newarray           : 1,536 bytes in 16 blocks
==83==         suppressed: 0 bytes in 0 blocks
==83== Rerun with --leak-check=full to see details of leaked memory
==83== 
==83== For counts of detected and suppressed errors, rerun with: -v
==83== ERROR SUMMARY: 5 errors from 1 contexts (suppressed: 0 from 0)

Expected results:
ERROR SUMMARY has 0 errors

Additional info:
I can see a little bit different valgrind error with direct usage of rpm bianry

sh# rpm -e lftp
sh# valgrind --track-origins=yes rpm -ivh https://kojipkgs.fedoraproject.org//packages/lftp/4.7.3/1.fc25/x86_64/lftp-4.7.3-1.fc25.x86_64.rpm
==91== Memcheck, a memory error detector
==91== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==91== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==91== Command: rpm -ivh https://kojipkgs.fedoraproject.org//packages/lftp/4.7.3/1.fc25/x86_64/lftp-4.7.3-1.fc25.x86_64.rpm
==91== 
Retrieving https://kojipkgs.fedoraproject.org//packages/lftp/4.7.3/1.fc25/x86_64/lftp-4.7.3-1.fc25.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:lftp-4.7.3-1.fc25                ################################# [100%]
==91== Syscall param pwrite64(buf) points to uninitialised byte(s)
==91==    at 0x6FA0DC3: __pwrite64_nocancel (in /usr/lib64/libpthread-2.24.90.so)
==91==    by 0x584AC30: __os_io (in /usr/lib64/libdb-5.3.so)
==91==    by 0x5836731: ??? (in /usr/lib64/libdb-5.3.so)
==91==    by 0x5836BA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==91==    by 0x5846E14: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==91==    by 0x57DCD16: __db_sync (in /usr/lib64/libdb-5.3.so)
==91==    by 0x57EF3CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so)
==91==    by 0x4E4EFF5: dbiSync (db3.c:547)
==91==    by 0x4E4EFF5: db3_dbiCursorFree (db3.c:606)
==91==    by 0x4E5626E: tag2index (rpmdb.c:2368)
==91==    by 0x4E59D4A: indexPut (rpmdb.c:2377)
==91==    by 0x4E59D4A: rpmdbAdd (rpmdb.c:2421)
==91==    by 0x4E6DA0C: dbAdd (psm.c:551)
==91==    by 0x4E6DA0C: rpmPackageInstall (psm.c:682)
==91==    by 0x4E6DA0C: rpmpsmRun (psm.c:844)
==91==    by 0x4E812A4: rpmteProcess (rpmte.c:756)
==91==  Address 0x9036f30 is in a rw- mapped file /var/lib/rpm/__db.003 segment
==91==  Uninitialised value was created by a stack allocation
==91==    at 0x572A740: ??? (in /usr/lib64/libdb-5.3.so)
==91== 
==91== 
==91== HEAP SUMMARY:
==91==     in use at exit: 13,716 bytes in 78 blocks
==91==   total heap usage: 12,555 allocs, 12,477 frees, 15,969,502 bytes allocated
==91== 
==91== LEAK SUMMARY:
==91==    definitely lost: 0 bytes in 0 blocks
==91==    indirectly lost: 0 bytes in 0 blocks
==91==      possibly lost: 0 bytes in 0 blocks
==91==    still reachable: 13,716 bytes in 78 blocks
==91==         suppressed: 0 bytes in 0 blocks
==91== Rerun with --leak-check=full to see details of leaked memory
==91== 
==91== For counts of detected and suppressed errors, rerun with: -v
==91== ERROR SUMMARY: 4 errors from 1 contexts (suppressed: 0 from 0)

Comment 1 Panu Matilainen 2016-11-21 15:18:43 UTC

The valgrind complaint is an ages old false positive caused by BDB internal optimization, its even documented somewhere in BDB doco.

Try looking at what it's actually doing when it appears stuck - is it burning cpu/io cycles or just hanging there etc.

Comment 2 Lukas Slebodnik 2016-11-21 15:23:23 UTC

(In reply to Panu Matilainen from comment #1)
> The valgrind complaint is an ages old false positive caused by BDB internal
> optimization, its even documented somewhere in BDB doco.
> 
> Try looking at what it's actually doing when it appears stuck - is it
> burning cpu/io cycles or just hanging there etc.

There is not any activity, but I can see also different valgrind error
which does not contain 'dbiSync'

Steps to Reproduce:
# docker run -ti --rm fedora:rawhide bash
Run docker images for fedora rawhide and run following commands inside container
    1  dnf update -y 'rpm*'
    2  dnf install -y 'dnf-command(builddep)' valgrind
    3  dnf debuginfo-install -y rpm libdb
    4  valgrind --track-origins=yes dnf update -y glibc

[root@d609e6427820 /]# valgrind --track-origins=yes dnf update -y glibc        
==80== Memcheck, a memory error detector
==80== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==80== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==80== Command: /usr/bin/dnf update -y glibc
==80== 
Last metadata expiration check: 0:02:12 ago on Mon Nov 21 15:15:36 2016 UTC.
Dependencies resolved.
================================================================================
 Package                  Arch        Version                Repository    Size
================================================================================
Upgrading:
 glibc                    x86_64      2.24.90-17.fc26        rawhide      3.3 M
 glibc-all-langpacks      x86_64      2.24.90-17.fc26        rawhide      7.0 M
 glibc-common             x86_64      2.24.90-17.fc26        rawhide      881 k
 libcrypt-nss             x86_64      2.24.90-17.fc26        rawhide       46 k

Transaction Summary
================================================================================
Upgrade  4 Packages

Total download size: 11 M
Downloading Packages:
(1/4): glibc-common-2.24.90-17.fc26.x86_64.rpm  1.8 MB/s | 881 kB     00:00    
(2/4): libcrypt-nss-2.24.90-17.fc26.x86_64.rpm  1.6 MB/s |  46 kB     00:00    
(3/4): glibc-2.24.90-17.fc26.x86_64.rpm         2.2 MB/s | 3.3 MB     00:01    
(4/4): glibc-all-langpacks-2.24.90-17.fc26.x86_ 2.4 MB/s | 7.0 MB     00:02    
--------------------------------------------------------------------------------
Total                                           1.7 MB/s |  11 MB     00:06     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : glibc-common-2.24.90-17.fc26.x86_64                         1/8 
==80== Syscall param pwrite64(buf) points to uninitialised byte(s)
==80==    at 0x5317DE3: ??? (in /usr/lib64/libpthread-2.24.90.so)
==80==    by 0x12720C30: __os_io (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1270C731: ??? (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so)
==80==    by 0x11F53FF5: dbiSync (db3.c:547)
==80==    by 0x11F53FF5: db3_dbiCursorFree (db3.c:606)
==80==    by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411)
==80==    by 0x11F72A0C: dbAdd (psm.c:551)
==80==    by 0x11F72A0C: rpmPackageInstall (psm.c:682)
==80==    by 0x11F72A0C: rpmpsmRun (psm.c:844)
==80==    by 0x11F862A4: rpmteProcess (rpmte.c:756)
==80==    by 0x11F8CBBD: rpmtsProcess (transaction.c:1361)
==80==    by 0x11F8CBBD: rpmtsRun (transaction.c:1541)
==80==  Address 0x7017596 is 2,278 bytes inside a block of size 4,096 alloc'd
==80==    at 0x4C2DB8D: malloc (vg_replace_malloc.c:299)
==80==    by 0x1271E1F4: __os_malloc (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1270C897: ??? (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1270CBA2: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==80==    by 0x1271CE14: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126B2D16: __db_sync (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126C53CF: __db_sync_pp (in /usr/lib64/libdb-5.3.so)
==80==    by 0x11F53FF5: dbiSync (db3.c:547)
==80==    by 0x11F53FF5: db3_dbiCursorFree (db3.c:606)
==80==    by 0x11F5ECD9: rpmdbAdd (rpmdb.c:2411)
==80==    by 0x11F72A0C: dbAdd (psm.c:551)
==80==    by 0x11F72A0C: rpmPackageInstall (psm.c:682)
==80==    by 0x11F72A0C: rpmpsmRun (psm.c:844)
==80==    by 0x11F862A4: rpmteProcess (rpmte.c:756)
==80==    by 0x11F8CBBD: rpmtsProcess (transaction.c:1361)
==80==    by 0x11F8CBBD: rpmtsRun (transaction.c:1541)
==80==  Uninitialised value was created by a stack allocation
==80==    at 0x125EB590: ??? (in /usr/lib64/libdb-5.3.so)
==80== 
  Upgrading   : glibc-all-langpacks-2.24.90-17.fc26.x86_64                  2/8 
==80== Conditional jump or move depends on uninitialised value(s)
==80==    at 0x1260C694: __bam_stkrel (in /usr/lib64/libdb-5.3.so)
==80==    by 0x125FD257: ??? (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126B71DB: __dbc_iput (in /usr/lib64/libdb-5.3.so)
==80==    by 0x126C69C0: __dbc_put_pp (in /usr/lib64/libdb-5.3.so)
==80==    by 0x11F53234: dbiCursorPut.isra.5 (db3.c:626)
==80==    by 0x11F551C0: updateIndex.part.8 (db3.c:1097)
==80==    by 0x11F55590: updateIndex (db3.c:1144)
==80==    by 0x11F55590: db3_idxdbPut (db3.c:1140)
==80==    by 0x11F5B0B2: tag2index (rpmdb.c:2348)
==80==    by 0x11F5ED4A: indexPut (rpmdb.c:2377)
==80==    by 0x11F5ED4A: rpmdbAdd (rpmdb.c:2421)
==80==    by 0x11F72A0C: dbAdd (psm.c:551)
==80==    by 0x11F72A0C: rpmPackageInstall (psm.c:682)
==80==    by 0x11F72A0C: rpmpsmRun (psm.c:844)
==80==    by 0x11F862A4: rpmteProcess (rpmte.c:756)
==80==    by 0x11F8CBBD: rpmtsProcess (transaction.c:1361)
==80==    by 0x11F8CBBD: rpmtsRun (transaction.c:1541)
==80==  Uninitialised value was created by a stack allocation
==80==    at 0x125E9C50: ??? (in /usr/lib64/libdb-5.3.so)
==80== 
  Upgrading   : glibc-2.24.90-17.fc26.x86_64                                3/8 
  Upgrading   : libcrypt-nss-2.24.90-17.fc26.x86_64                         4/8 
  Cleanup     : libcrypt-nss-2.24.90-2.fc26.x86_64                          5/8 
  Cleanup     : glibc-common-2.24.90-2.fc26.x86_64                          6/8 
  Cleanup     : glibc-all-langpacks-2.24.90-2.fc26.x86_64                   7/8 
  Cleanup     : glibc-2.24.90-2.fc26.x86_64                                 8/8 
  Verifying   : glibc-2.24.90-17.fc26.x86_64                                1/8 
  Verifying   : glibc-common-2.24.90-17.fc26.x86_64                         2/8 
  Verifying   : glibc-all-langpacks-2.24.90-17.fc26.x86_64                  3/8 
  Verifying   : libcrypt-nss-2.24.90-17.fc26.x86_64                         4/8 
  Verifying   : glibc-2.24.90-2.fc26.x86_64                                 5/8 
  Verifying   : glibc-all-langpacks-2.24.90-2.fc26.x86_64                   6/8 
  Verifying   : glibc-common-2.24.90-2.fc26.x86_64                          7/8 
  Verifying   : libcrypt-nss-2.24.90-2.fc26.x86_64                          8/8 

Upgraded:
  glibc.x86_64 2.24.90-17.fc26                                                  
  glibc-all-langpacks.x86_64 2.24.90-17.fc26                                    
  glibc-common.x86_64 2.24.90-17.fc26                                           
  libcrypt-nss.x86_64 2.24.90-17.fc26                                           

Complete!
==80== 
==80== HEAP SUMMARY:
==80==     in use at exit: 58,807,031 bytes in 56,957 blocks
==80==   total heap usage: 1,264,661 allocs, 1,207,704 frees, 4,144,098,835 bytes allocated
==80== 
==80== LEAK SUMMARY:
==80==    definitely lost: 1,521 bytes in 23 blocks
==80==    indirectly lost: 125,140 bytes in 66 blocks
==80==      possibly lost: 3,288,601 bytes in 19,571 blocks
==80==    still reachable: 55,391,001 bytes in 37,287 blocks
==80==                       of which reachable via heuristic:
==80==                         length64           : 426,096 bytes in 504 blocks
==80==                         newarray           : 1,536 bytes in 16 blocks
==80==         suppressed: 0 bytes in 0 blocks
==80== Rerun with --leak-check=full to see details of leaked memory
==80== 
==80== For counts of detected and suppressed errors, rerun with: -v
==80== ERROR SUMMARY: 34 errors from 2 contexts (suppressed: 0 from 0)

Comment 3 Lukas Slebodnik 2016-11-21 15:36:09 UTC

Created attachment 1222428 [details]
Dockerfile which reveal the bug very often

Copy attached file to empty directory and build and image
sh# docker build -t temp_rm_me .

The installation of build dependencies stuck for me with following output
Dependencies resolved.
================================================================================
 Package                          Arch   Version                  Repository
                                                                           Size
================================================================================
Installing:
 389-ds-base-devel                x86_64 1.3.6.1-1.fc26           rawhide 222 k
 autoconf                         noarch 2.69-23.fc26             rawhide 710 k
 automake                         noarch 1.15-8.fc26              rawhide 694 k
 cyrus-sasl-devel                 x86_64 2.1.26-27.fc26           rawhide 313 k
 dbus-python                      x86_64 1.2.4-2.fc25             rawhide 129 k

//snip

Transaction Summary
================================================================================
Install  209 Packages
Upgrade   28 Packages
Skip      14 Packages

Total download size: 99 M
Downloading Packages:
--------------------------------------------------------------------------------
Total                                            28 MB/s |  99 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : libcom_err-1.43.3-1.fc26.x86_64                           1/265 
  Installing  : libtalloc-2.1.8-1.fc26.x86_64                             2/265 
  Upgrading   : nspr-4.13.1-1.fc26.x86_64                                 3/265 
  Installing  : libtevent-0.9.31-1.fc26.x86_64                            4/265 
  Installing  : libtdb-1.3.11-1.fc26.x86_64                               5/265 
  Upgrading   : nss-util-3.27.0-2.fc26.x86_64                             6/265 
  Installing  : python2-six-1.10.0-4.fc26.noarch                          7/265 
  Upgrading   : libuuid-2.29-1.fc26.x86_64                                8/265 
  Installing  : libldb-1.1.27-1.fc26.x86_64                               9/265 
  Upgrading   : pcre-8.39-6.fc26.x86_64                                  10/265 
  Installing  : nspr-devel-4.13.1-1.fc26.x86_64                          11/265 
  Upgrading   : libblkid-2.29-1.fc26.x86_64                              12/265 
  Installing  : xmlrpc-c-1.32.5-1909.svn2451.fc24.x86_64                 13/265


and there are just 2 processes inside container shell and python with dnf

   CGroup: /system.slice/docker-04f608509c75dc43bb7a1c5e86fae071397b4969ba03d179cc0b3466229bfd37.scope
           ├─24379 /bin/sh -c dnf --setopt=deltarpm=0 --setopt=fastestmirror=1         --setopt=install_weak_deps=false --setopt=tsflags=nodocs         install -y             sudo             git             'dnf-command(builddep)'             @buildsys-build     && curl https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in             -o freeipa.spec.in     && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1            --setopt=install_weak_deps=false --setopt=tsflags=nodocs            builddep -y --spec freeipa.spec.in     && rm -f freeipa.spec.in     && dnf clean all
           └─24589 /usr/libexec/system-python /usr/bin/dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 --setopt=install_weak_deps=false --setopt=tsflags=nodocs builddep -y --spec freeipa.spec.in


The strace for python is:
sh# strace -p 24589
strace: Process 24589 attached
futex(0x7ff6f2a11064, FUTEX_WAIT, 4294967295, NULL

Comment 4 Lukas Slebodnik 2016-11-21 15:37:49 UTC

As I mention in the description The same problem is with yum-deprecated.
And if I try to install some packages manually the installation is stuck on different package.

Comment 5 Lukas Slebodnik 2016-11-21 15:39:28 UTC

BTW my workaround is to remove && rebuild rpm-db before calling builddep

--- a/Dockerfile        2016-11-21 16:24:25.994623950 +0100
+++ b/Dockerfile        2016-11-21 14:53:04.466568545 +0100
@@ -11,6 +11,8 @@
             @buildsys-build \
     && curl https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in \
             -o freeipa.spec.in \
+    && rm -f /var/lib/rpm/__db* \
+    && rpm --rebuilddb \
     && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 \
            --setopt=install_weak_deps=false --setopt=tsflags=nodocs \
            builddep -y --spec freeipa.spec.in \

Comment 6 Lukas Slebodnik 2016-11-21 17:05:43 UTC

(In reply to Lukas Slebodnik from comment #5)
> BTW my workaround is to remove && rebuild rpm-db before calling builddep
> 
> --- a/Dockerfile        2016-11-21 16:24:25.994623950 +0100
> +++ b/Dockerfile        2016-11-21 14:53:04.466568545 +0100
> @@ -11,6 +11,8 @@
>              @buildsys-build \
>      && curl
> https://git.fedorahosted.org/cgit/freeipa.git/plain/freeipa.spec.in \
>              -o freeipa.spec.in \
> +    && rm -f /var/lib/rpm/__db* \
> +    && rpm --rebuilddb \
>      && dnf --setopt=deltarpm=0 --setopt=fastestmirror=1 \
>             --setopt=install_weak_deps=false --setopt=tsflags=nodocs \
>             builddep -y --spec freeipa.spec.in \

Florian F. mentioned on IRC, that simpler workaround is just to remove var/lib/rpm/__db* files. Calling "rpm --rebuilddb" is not necessary.

Comment 7 Panu Matilainen 2016-11-22 04:18:11 UTC

That sounds like there's a db cursor not getting freed somewhere.

Comment 8 Panu Matilainen 2016-11-22 05:31:44 UTC

(too early in the morning...)

Output of 'db_stat -CA -h /var/lib/rpm' as root when it hangs would be interesting.

Comment 9 Lukas Slebodnik 2016-11-22 09:09:25 UTC

Created attachment 1222632 [details]
Output of db_stat -CA -h /var/lib/rpm

Here you are

Comment 10 Lukas Slebodnik 2016-11-22 09:11:06 UTC

FYI I also noticed that fedora rawhide has never version of libdb 5.3.28-15
which should fix mutexes not being released properly (#1272680).

I downgraded libdb to older version without this patch but it did not fix hangs.

Comment 11 Lukas Slebodnik 2016-11-23 13:12:26 UTC

I hit this bug yesterday on my laptop when I was upgrading f25 -> rawhide.
I had to cancel/kill dnf and as a side effect I had installed more version of packages because upgrade transaction has not been finished.

Comment 12 Panu Matilainen 2016-11-23 16:26:34 UTC

Okay I can reproduce something remotely resembling that now. I'm on F24'ish system so details will differ but there are hints at least.

# yum-deprecated --releasever=25 --installroot /srv/test --setopt=keepcache=true install bash
[...]
# db_stat -CA -h /srv/test/var/lib/rpm
db_stat: DB_ENV->open: /srv/test/var/lib/rpm: No such file or directory

That is how it should be, rpm removes the BDB environment after chroot operations. That's what happens with rpm itself (obviously). However doing the same with dnf:

# dnf --releasever=25 --installroot /srv/test --setopt=keepcache=true install bash
[...]
# db_stat -CA -h /srv/test/var/lib/rpm|wc -l
162

In this case they're all READ locks which rpm will mope up on the next run but it'll complain loudly. I know I've seen fleeting glimpses of those complaints where they shouldn't have existed, just haven't had a chance to investigate. But the screwy thing here is that the environment should not even exist after dnf exits! So it seems dnf is (re)opening the rpmdb from inside the chroot which is ... not a good idea. This is with dnf 1.1.10 on F24, IIRC F25 has 2.0 and newer hawkey/whatever so there's plenty of room for different behavior. 

You said you can reproduce it with yum-deprecated, but can you actually reproduce the hang if you *only* use yum-deprecated all the way? (ie after using "rm -f /var/lib/rpm/__db*" to clear any cruft from earlier runs)

Comment 13 Panu Matilainen 2016-11-23 16:38:44 UTC

Another datapoint: for my testcase the problem is related to importing pubkeys. If I run dnf with --nogpgcheck or pre-import the keys to the test root, there are no leftover locks. Please test whether that makes a difference for you - its possible I'm seeing a different issue entirely, but its also quite possible it's the same issue but only worse with newer dnf.

Comment 14 Panu Matilainen 2016-11-24 09:35:35 UTC

Right, never mind about the pubkey case, that appears to be relatively harmless.

What kills it is any outside access to rpmdb during the transaction where glibc is updated to the rawhide version. For example 'db_stat -CA -h /var/lib/rpm' is fine until glibc has gotten updated, after that trying to run it causes both rpm and the db_stat process  tohang. Ditto for running 'rpm -qa' or such (as root) durin the update. That does not happen during eg F24 -> F25 glibc update.

fweimer (cc'd): has there been some major futex-related change in rawhide glibc?

Comment 15 Panu Matilainen 2016-11-24 09:49:56 UTC

Oh, actually it's simpler than that: after updating to rawhide glibc, new rpm is dead until you nuke the BDB environment: 'rm -f /var/lib/rpm/__db*'

Comment 16 Panu Matilainen 2016-11-24 09:50:47 UTC

Oops, not "new rpm". Just "rpm is dead" until...

Comment 17 Florian Weimer 2016-11-24 10:10:52 UTC

(In reply to Panu Matilainen from comment #14)
> Right, never mind about the pubkey case, that appears to be relatively
> harmless.
> 
> What kills it is any outside access to rpmdb during the transaction where
> glibc is updated to the rawhide version. For example 'db_stat -CA -h
> /var/lib/rpm' is fine until glibc has gotten updated, after that trying to
> run it causes both rpm and the db_stat process  tohang. Ditto for running
> 'rpm -qa' or such (as root) durin the update. That does not happen during eg
> F24 -> F25 glibc update.
> 
> fweimer (cc'd): has there been some major futex-related change in rawhide
> glibc?

This looks like bug 1394862.  Berkeley DB assumes that it's safe to use process-shared mutexes and condition variables after all processes have detached from a mapping, something which POSIX does not guarantee.

Comment 18 Panu Matilainen 2016-11-24 10:35:18 UTC

(In reply to Florian Weimer from comment #17)
> This looks like bug 1394862. 

Oh, indeed. Thanks for the pointer.

> Berkeley DB assumes that it's safe to use
> process-shared mutexes and condition variables after all processes have
> detached from a mapping, something which POSIX does not guarantee.

Right. It's not the first time such glibc changes cause trouble for rpmdb, just been a while since the last one and BDB locks are still just as ill-suited for rpm as they ever were...

Technically this is a dupe of bug 1394862 but I'm leaving this open for now:
1) people are more likely to find an rpm bug than a libdb one when installation hangs
2) we might need to come up with some kind of workaround, we can't have every other upgrade blowing up in the middle

As for 2), we could have a trigger on glibc to always nuke the environment in that case, but I worry what it does to the already running rpm instance. Or we could have rpm use private bdb locking but that opens up other issues. Or something...

Comment 19 Fedora End Of Life 2017-02-28 10:38:42 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 20 Panu Matilainen 2017-04-04 10:27:13 UTC

*** Bug 1435927 has been marked as a duplicate of this bug. ***

Comment 21 Adam Williamson 2017-05-30 17:44:51 UTC

To interject with a bit of blocker bureaucracy: I suggest we make this an Accepted0Day+AcceptedPreviousRelease blocker rather than an AcceptedBlocker. It would still prevent us shipping Beta, but we could at least build and test an RC and ensure there are no *other* blockers.

To refresh folks' memories: Accepted0Day means an update must be pushed for the release in question by Beta release day. AcceptedPreviousRelease means an update must be pushed for relevant *previous* releases by Beta release day. This would be both since it's likely we need updates for both the previous stable releases (24/25) *and* the new release (26) to ensure a successful upgrade. If that winds up not being the case with the solution we pick, we can drop whichever status is no longer relevant.

Sound OK to everyone? Any objections?

Comment 22 Stephen Gallagher 2017-05-30 19:05:28 UTC

+1 Accepted0Day
+1 AcceptedPreviousRelease

Comment 23 Mike Ruckman 2017-05-30 19:08:59 UTC

+1 on both counts for me as well.

Comment 24 Dennis Gilmore 2017-05-30 19:09:32 UTC

+1 Accepted0Day
+1 AcceptedPreviousRelease

Comment 25 Adam Williamson 2017-05-30 19:40:51 UTC

Whoops, this one isn't actually the blocker at present...but I think it makes more sense to make this specifically a blocker, rather than the tracker bug. Giving blocker status to a tracker bug doesn't make a lot of sense as its 'meaning' can change over time. So I'll transfer the status from the tracker bug and make it accepted0day and acceptedpreviousrelease. Thanks folks!

For the record: we're transferring blocker status from https://bugzilla.redhat.com/show_bug.cgi?id=1443415 to this bug, as it was really this issue specifically we voted on as a blocker.

Comment 26 Panu Matilainen 2017-05-31 05:59:06 UTC

Yes, dnf is merely a passanger in all this. The deeper bug here is bug 1394862 of course but if we can't find a workable solution there we might need to (additionally) apply some workaround on rpm-side so it's just as well this is the actualy blocker.

Comment 27 Adam Williamson 2017-05-31 07:22:08 UTC

Yep, that was my thought exactly in making this rather than 1394862 the blocker.

Comment 28 Fedora Update System 2017-06-01 12:44:20 UTC

libdb-5.3.28-21.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Comment 29 Fedora Update System 2017-06-01 12:45:37 UTC

libdb-5.3.28-21.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Comment 30 Adam Williamson 2017-06-01 17:57:51 UTC

does this also need to be sent out for 24/25?

Comment 31 Adam Williamson 2017-06-02 05:30:25 UTC

It'd be really great if people could help test the new update hard: test that it fixes the actual bug, test it in the scenarios that caused trouble with -18 and -19, etc. We want to get this fix out for Beta, but we don't want to send it unless we're sure it's good. Thanks a lot!

Comment 32 Lukas Slebodnik 2017-06-02 11:42:00 UTC

(In reply to Adam Williamson from comment #31)
> It'd be really great if people could help test the new update hard: test
> that it fixes the actual bug, test it in the scenarios that caused trouble
> with -18 and -19, etc. We want to get this fix out for Beta, but we don't
> want to send it unless we're sure it's good. Thanks a lot!

I tried dnf update https://kojipkgs.fedoraproject.org//packages/libdb/5.3.28/21.fc26/x86_64/libdb-5.3.28-21.fc26.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/libdb/5.3.28/21.fc26/x86_64/libdb-utils-5.3.28-21.fc26.x86_64.rpm

And I could not see any problem with installed package openldap-servers

Comment 33 Ali Akcaagac 2017-06-02 17:07:28 UTC

Hi!

I am not sure whether this might be related to the issue reported here. But in doubt I better put it on here so it can be verified.

I am running what will become Fedora 26 (with all the latest updates that got pushed to the repositories). Around 2 weeks ago I was compiling some Xfce packages using an automated bash script.

What it does is pulling in the tar files, adjusting some Fedora specs and pulling in needed dependencies using dnf builddep.

This script runs for a few years without errors but recently I saw that the RPM database often gets locked and crashed.

Today I tried to re-run the script again and ran into the same RPM database crash again.

Please allow me to add the logfile so you can see whether this might be related to the RPM issue (directly or indirectly).

The log is full of this:

RPM build errors:
db5 error(5) from dbenv->open: Input/output error
cannot open Packages index using db5 - Input/output error (5)
cannot open Packages database in /var/lib/rpm
Bad exit status from /var/tmp/rpm-tmp.GjSmLj (%build)
error: db5 error(5) from dbenv->open: Input/output error
error: cannot open Packages index using db5 - Input/output error (5)
error: cannot open Packages database in /var/lib/rpm
error: db5 error(5) from dbenv->open: Input/output error
error: cannot open Packages index using db5 - Input/output error (5)
error: cannot open Packages database in /var/lib/rpm
error: db5 error(5) from dbenv->open: Input/output error
error: cannot open Packages index using db5 - Input/output error (5)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

The only thing happens here is just running dnf builddep that pulls in all the requirements to build the packages....

Comment 34 Ali Akcaagac 2017-06-02 17:08:18 UTC

Created attachment 1284488 [details]
Please grep for "db5 error"

Comment 35 Striker Leggette 2017-06-02 17:18:42 UTC

Upgraded:
libdb.x86_64 5.3.28-21.fc26
libdb-utils.x86_64 5.3.28-21.fc26

Complete!
[root@f26beta14 folder]# rpm -qa | tail -n 2
fedora-release-notes-25.01-2.fc26.noarch
passwdqc-lib-1.3.0-7.fc26.x86_64
[root@f26beta14 folder]#

Karma given

Comment 36 Ali Akcaagac 2017-06-03 06:18:08 UTC

I just like to inform that I updated to the latest libdb packages (manually).

Unfortunately within the process of building packages, I found out that some Fedora 26 related packages (during resolving builddep) downgrades libdb back to the -17 version, which in the flow of building the packages lead into a db5 error again.

It would be nice if you could guarantee, that no Fedora 26 package enforces a downgrade (in my case I've instructed dnf builddep to use --allowerasing). What I mean by this can be described as follows:

Packages and versions of packages tend to change over time. But quite often building (or re-building) an RPM SRC packages have a strict dependency for older versions of a package (most likely the one it initially was compiled against with).

In a flow of dependency resolving (one package resolves the requirements of another) this might cause (quite often) the issue that one or other *required* packages (for building things) wants an older version. dnf can keep up doing this by using --allowerasing so it can resolve conflicts on its own and downgrade the packages as needed. Usually the downgrades are required for some header files. After compiling the packages they usually run perfectly with never versions of any libraries. But if the downgrading also requires that an older version of the library must be installed back (as the example below) then there is a risk, that on building farms and building machines, that the issue with db5 crashes comes back (as it happened a few moments ago).

So one of the packages below (in the dependency tree) enforces a downgrade of libdb (I haven't verified this yet!!!).

Dependencies resolved.
================================================================================
 Package                   Arch     Version             Repository         Size
================================================================================
Installing:
 dbus-glib-devel           x86_64   0.108-2.fc26        fedora             73 k
 perl-ExtUtils-Depends     noarch   0.405-4.fc26        fedora             30 k
 perl-ExtUtils-MakeMaker   noarch   7.24-2.fc26         fedora            295 k
 perl-ExtUtils-PkgConfig   noarch   1.15-7.fc26         fedora             19 k
 perl-Glib                 x86_64   1.324-2.fc26        fedora            368 k
 perl-Glib-devel           x86_64   1.324-2.fc26        fedora             70 k
 perl-devel                x86_64   4:5.24.1-390.fc26   updates-testing   575 k
 perl-generators           noarch   1.10-2.fc26         fedora             16 k
Installing dependencies:
 gdbm-devel                x86_64   1.13-1.fc26         updates-testing    62 k
 libdb-devel               x86_64   5.3.28-17.fc26      fedora             42 k
 perl-ExtUtils-Command     noarch   7.24-2.fc26         fedora             17 k
 perl-ExtUtils-Install     noarch   2.04-367.fc26       fedora             44 k
 perl-ExtUtils-Manifest    noarch   1.70-366.fc26       fedora             35 k
 perl-ExtUtils-ParseXS     noarch   1:3.31-368.fc26     fedora             81 k
 perl-Fedora-VSP           noarch   0.001-5.fc26        fedora             23 k
 perl-Storable             x86_64   1:2.56-368.fc26     fedora             83 k
 perl-Test-Harness         noarch   3.39-1.fc26         updates-testing   276 k
 perl-version              x86_64   5:0.99.18-1.fc26    updates-testing    64 k
 systemtap-sdt-devel       x86_64   3.1-5.fc26          updates-testing    71 k
Downgrading:
 libdb                     x86_64   5.3.28-17.fc26      fedora            748 k
 libdb-utils               x86_64   5.3.28-17.fc26      fedora            137 k

Transaction Summary
================================================================================
Install    19 Packages
Downgrade   2 Packages

Comment 37 Florian Weimer 2017-06-03 06:33:01 UTC

(In reply to Ali Akcaagac from comment #36)
> So one of the packages below (in the dependency tree) enforces a downgrade
> of libdb (I haven't verified this yet!!!).

> Installing dependencies:
>  gdbm-devel                x86_64   1.13-1.fc26         updates-testing   
> 62 k
>  libdb-devel               x86_64   5.3.28-17.fc26      fedora            
> 42 k

It's likely that libdb-devel in its newer version just isn't available from the configured repositories, so in order to install the package, the already-upgraded subpackages are downgraded to match the available package version.

Comment 38 Ali Akcaagac 2017-06-03 08:16:05 UTC

(In reply to Florian Weimer from comment #37)
> It's likely that libdb-devel in its newer version just isn't available from
> the configured repositories, so in order to install the package, the
> already-upgraded subpackages are downgraded to match the available package
> version.

Yes you are right! This was exactly the case here after re-investigating into the issue... I am a a bit overworked recently and was focussing on the db5 issue only...

Comment 39 Fedora Update System 2017-06-04 05:10:59 UTC

libdb-5.3.28-21.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Comment 40 Adam Williamson 2017-06-07 17:42:29 UTC

So by all accounts -21 is looking pretty good as a fix for this. I propose we go ahead and push it stable for all releases. Could people please karma the F24 and F25 updates, so long as you're happy with them, so they can be pushed out? Thanks.

Comment 41 Fedora Update System 2017-06-08 19:24:40 UTC

libdb-5.3.28-21.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf

Comment 42 Fedora Update System 2017-06-08 19:25:02 UTC

libdb-5.3.28-21.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

Comment 43 Fedora Update System 2017-06-08 21:07:36 UTC

libdb-5.3.28-21.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 44 Fedora Update System 2017-06-09 09:24:13 UTC

libdb-5.3.28-21.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 45 Fedora Update System 2017-06-09 11:29:32 UTC

libdb-5.3.28-21.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 46 Adam Williamson 2017-06-09 20:35:21 UTC

Unfortunately, while no-one reported such issues in the last week or so, now -21 has gone stable, I've seen several reports of problems very much along the lines we saw with -18 and -19. Here's one from Eric Blake (hope you don't mind me quoting it, Eric):

==========

On an F25 box, I ran 'dnf update', and it installed libdb.x86_64
5.3.28-21.fc25 among others.  That run of 'dnf' dumped core, and all
subsequent runs of dnf were unhappy:

# dnf update
...
  Verifying   : libdb-5.3.28-21.fc25.x86_64
 18/80
...
Complete!
Segmentation fault (core dumped)
# dnf update
error: rpmdb: BDB0113 Thread/process 6173/140680480651008 failed:
BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY:
Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

I don't know how many others will have hosed systems now that libdb
appears to have been pushed to stable, but it's a bummer that it
happened to me.

=========

Can we please look into this again as a matter of urgency? It's obviously bad to be breaking F24/F25 systems. Perhaps we should look into removing the update and/or doing a reverted -22 as an exceptional step also.

Comment 47 Adam Williamson 2017-06-09 20:36:37 UTC

Eric, can you get a backtrace from the core dump if possible? Can you also provide a full list of what was in the update transaction (the entire console log would be great if you still have it, otherwise dnf history should do)? Thanks a lot.

Comment 48 Eric Blake 2017-06-09 20:52:05 UTC

[root@red eblake]# dnf update
Last metadata expiration check: 0:47:27 ago on Fri Jun  9 09:07:41 2017.
Dependencies resolved.
================================================================================
 Package                      Arch      Version               Repository   Size
================================================================================
Installing:
 kernel                       x86_64    4.11.3-202.fc25       updates     100 k
 kernel-core                  x86_64    4.11.3-202.fc25       updates      21 M
 kernel-debug-devel           x86_64    4.11.3-202.fc25       updates      11 M
 kernel-modules               x86_64    4.11.3-202.fc25       updates      23 M
 python-qpid-proton           x86_64    0.17.0-2.fc25         updates     220 k
 python-rhmsg                 noarch    0.5-1.fc25            rhpkg        27 k
 python2-simplejson           x86_64    3.10.0-1.fc25         updates     285 k
 qpid-proton-c                x86_64    0.17.0-2.fc25         updates     159 k
 rpmdiff                      x86_64    4.1.1-1.fc25          rhpkg       274 k
 rpmdiff-remote               x86_64    4.1.1-1.fc25          rhpkg       132 k
Upgrading:
 google-noto-emoji-fonts      noarch    20170523-1.fc25       updates     4.4 M
 hplip                        x86_64    3.17.4-2.fc25         updates      13 M
 hplip-common                 x86_64    3.17.4-2.fc25         updates     102 k
 hplip-libs                   x86_64    3.17.4-2.fc25         updates     191 k
 hwdata                       noarch    0.301-1.fc25          updates     1.4 M
 kernel-headers               x86_64    4.11.3-202.fc25       updates     1.1 M
 libdb                        x86_64    5.3.28-21.fc25        updates     749 k
 libdb-devel                  x86_64    5.3.28-21.fc25        updates      43 k
 libdb-utils                  x86_64    5.3.28-21.fc25        updates     138 k
 libmicrohttpd                x86_64    1:0.9.55-1.fc25       updates      77 k
 libsane-hpaio                x86_64    3.17.4-2.fc25         updates     120 k
 mesa-dri-drivers             x86_64    17.0.5-3.fc25         updates      12 M
 mesa-filesystem              x86_64    17.0.5-3.fc25         updates      25 k
 mesa-libEGL                  x86_64    17.0.5-3.fc25         updates     104 k
 mesa-libEGL-devel            x86_64    17.0.5-3.fc25         updates      41 k
 mesa-libGL                   x86_64    17.0.5-3.fc25         updates     165 k
 mesa-libGL-devel             x86_64    17.0.5-3.fc25         updates     161 k
 mesa-libGLES                 x86_64    17.0.5-3.fc25         updates      22 k
 mesa-libGLES-devel           x86_64    17.0.5-3.fc25         updates      63 k
 mesa-libOpenCL               x86_64    17.0.5-3.fc25         updates     581 k
 mesa-libgbm                  x86_64    17.0.5-3.fc25         updates      43 k
 mesa-libgbm-devel            x86_64    17.0.5-3.fc25         updates      27 k
 mesa-libglapi                x86_64    17.0.5-3.fc25         updates      50 k
 mesa-libwayland-egl          x86_64    17.0.5-3.fc25         updates      26 k
 mesa-libwayland-egl-devel    x86_64    17.0.5-3.fc25         updates      23 k
 mesa-libxatracker            x86_64    17.0.5-3.fc25         updates     1.4 M
 ntfs-3g                      x86_64    2:2017.3.23-1.fc25    updates     273 k
 ntfsprogs                    x86_64    2:2017.3.23-1.fc25    updates     382 k
 perl-Net-HTTP                noarch    6.16-1.fc25           updates      41 k
 perl-threads                 x86_64    1:2.16-1.fc25         updates      58 k
 qrencode-libs                x86_64    3.4.4-1.fc25          updates      55 k
 rhpkg                        noarch    1.32-2.fc25           rhpkg       121 k
 vim-minimal                  x86_64    2:8.0.617-1.fc25      updates     520 k
Removing:
 kernel                       x86_64    4.10.16-200.fc25      @updates      0  
 kernel-core                  x86_64    4.10.16-200.fc25      @updates     53 M
 kernel-debug-devel           x86_64    4.10.16-200.fc25      @updates     42 M
 kernel-modules               x86_64    4.10.16-200.fc25      @updates     22 M

Transaction Summary
================================================================================
Install  10 Packages
Upgrade  33 Packages
Remove    4 Packages

Total download size: 93 M
Is this ok [y/N]: y
Downloading Packages:
(1/43): hplip-libs-3.17.4-1.fc25_3.17.4-2.fc25.  88 kB/s |  51 kB     00:00    
(2/43): hplip-common-3.17.4-1.fc25_3.17.4-2.fc2 320 kB/s |  48 kB     00:00    
[DRPM] hplip-libs-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done                
(3/43): hplip-3.17.4-1.fc25_3.17.4-2.fc25.x86_6 353 kB/s | 346 kB     00:00    
[DRPM] hplip-common-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done              
(4/43): libsane-hpaio-3.17.4-1.fc25_3.17.4-2.fc 144 kB/s |  51 kB     00:00    
(5/43): hwdata-0.300-1.fc25_0.301-1.fc25.noarch  45 kB/s |  41 kB     00:00    
[DRPM] libsane-hpaio-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done             
(6/43): kernel-debug-devel-4.11.3-200.fc25_4.11 1.2 MB/s | 2.7 MB     00:02    
(7/43): libdb-devel-5.3.28-16.fc25_5.3.28-21.fc 138 kB/s |  10 kB     00:00    
(8/43): libdb-5.3.28-16.fc25_5.3.28-21.fc25.x86 260 kB/s | 129 kB     00:00    
[DRPM] libdb-devel-5.3.28-16.fc25_5.3.28-21.fc25.x86_64.drpm: done             
(9/43): libmicrohttpd-0.9.54-1.fc25_0.9.55-1.fc 243 kB/s |  32 kB     00:00    
(10/43): mesa-libEGL-17.0.5-2.fc25_17.0.5-3.fc2 343 kB/s |  28 kB     00:00    
[DRPM] libmicrohttpd-0.9.54-1.fc25_0.9.55-1.fc25.x86_64.drpm: done             
(11/43): mesa-libGL-17.0.5-2.fc25_17.0.5-3.fc25 438 kB/s |  39 kB     00:00    
[DRPM] mesa-libEGL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done               
(12/43): mesa-libglapi-17.0.5-2.fc25_17.0.5-3.f 265 kB/s |  21 kB     00:00    
(13/43): mesa-libGL-devel-17.0.5-2.fc25_17.0.5- 263 kB/s |  22 kB     00:00    
[DRPM] mesa-libGL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done                
[DRPM] mesa-libglapi-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done             
(14/43): kernel-headers-4.11.3-200.fc25_4.11.3- 116 kB/s | 208 kB     00:01    
(15/43): mesa-libGLES-devel-17.0.5-2.fc25_17.0. 222 kB/s |  22 kB     00:00    
[DRPM] libdb-5.3.28-16.fc25_5.3.28-21.fc25.x86_64.drpm: done                   
(16/43): mesa-libgbm-17.0.5-2.fc25_17.0.5-3.fc2 215 kB/s |  21 kB     00:00    
[DRPM] hwdata-0.300-1.fc25_0.301-1.fc25.noarch.drpm: done                      
[DRPM] mesa-libGL-devel-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done          
[DRPM] mesa-libGLES-devel-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done        
(17/43): mesa-libOpenCL-17.0.5-2.fc25_17.0.5-3. 114 kB/s |  22 kB     00:00    
[DRPM] mesa-libgbm-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done               
(18/43): perl-Net-HTTP-6.15-1.fc25_6.16-1.fc25. 124 kB/s |  18 kB     00:00    
(19/43): mesa-libxatracker-17.0.5-2.fc25_17.0.5 632 kB/s | 161 kB     00:00    
[DRPM] perl-Net-HTTP-6.15-1.fc25_6.16-1.fc25.noarch.drpm: done                 
(20/43): qrencode-libs-3.4.2-6.fc24_3.4.4-1.fc2 219 kB/s |  18 kB     00:00    
(21/43): perl-threads-2.15-1.fc25_2.16-1.fc25.x 131 kB/s |  23 kB     00:00    
[DRPM] mesa-libOpenCL-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done            
[DRPM] qrencode-libs-3.4.2-6.fc24_3.4.4-1.fc25.x86_64.drpm: done               
(22/43): kernel-4.11.3-202.fc25.x86_64.rpm      176 kB/s | 100 kB     00:00    
(23/43): mesa-dri-drivers-17.0.5-2.fc25_17.0.5- 502 kB/s | 1.1 MB     00:02    
[DRPM] kernel-headers-4.11.3-200.fc25_4.11.3-202.fc25.x86_64.drpm: done        
[DRPM] mesa-libxatracker-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done         
[DRPM] perl-threads-2.15-1.fc25_2.16-1.fc25.x86_64.drpm: done                  
(24/43): python-rhmsg-0.5-1.fc25.noarch.rpm      60 kB/s |  27 kB     00:00    
(25/43): rpmdiff-remote-4.1.1-1.fc25.x86_64.rpm 376 kB/s | 132 kB     00:00    
(26/43): rpmdiff-4.1.1-1.fc25.x86_64.rpm        615 kB/s | 274 kB     00:00    
(27/43): python2-simplejson-3.10.0-1.fc25.x86_6 411 kB/s | 285 kB     00:00    
(28/43): python-qpid-proton-0.17.0-2.fc25.x86_6 459 kB/s | 220 kB     00:00    
(29/43): qpid-proton-c-0.17.0-2.fc25.x86_64.rpm 406 kB/s | 159 kB     00:00    
(30/43): google-noto-emoji-fonts-20170523-1.fc2 700 kB/s | 4.4 MB     00:06    
(31/43): libdb-utils-5.3.28-21.fc25.x86_64.rpm  531 kB/s | 138 kB     00:00    
(32/43): mesa-filesystem-17.0.5-3.fc25.x86_64.r 268 kB/s |  25 kB     00:00    
(33/43): mesa-libEGL-devel-17.0.5-3.fc25.x86_64 466 kB/s |  41 kB     00:00    
(34/43): mesa-libGLES-17.0.5-3.fc25.x86_64.rpm  285 kB/s |  22 kB     00:00    
(35/43): mesa-libgbm-devel-17.0.5-3.fc25.x86_64 333 kB/s |  27 kB     00:00    
(36/43): mesa-libwayland-egl-17.0.5-3.fc25.x86_ 198 kB/s |  26 kB     00:00    
(37/43): mesa-libwayland-egl-devel-17.0.5-3.fc2 164 kB/s |  23 kB     00:00    
(38/43): ntfs-3g-2017.3.23-1.fc25.x86_64.rpm    562 kB/s | 273 kB     00:00    
(39/43): ntfsprogs-2017.3.23-1.fc25.x86_64.rpm  734 kB/s | 382 kB     00:00    
(40/43): rhpkg-1.32-2.fc25.noarch.rpm           313 kB/s | 121 kB     00:00    
(41/43): vim-minimal-8.0.617-1.fc25.x86_64.rpm  708 kB/s | 520 kB     00:00    
(42/43): kernel-core-4.11.3-202.fc25.x86_64.rpm 809 kB/s |  21 MB     00:26    
(43/43): kernel-modules-4.11.3-202.fc25.x86_64. 910 kB/s |  23 MB     00:25    
[DRPM] hplip-3.17.4-1.fc25_3.17.4-2.fc25.x86_64.drpm: done                     
[DRPM] kernel-debug-devel-4.11.3-200.fc25_4.11.3-202.fc25.x86_64.drpm: done    
[DRPM] mesa-dri-drivers-17.0.5-2.fc25_17.0.5-3.fc25.x86_64.drpm: done          
--------------------------------------------------------------------------------
Total                                           1.6 MB/s |  56 MB     00:35     
Delta RPMs reduced 93.2 MB of updates to 55.8 MB (40.1% saved)
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : mesa-libglapi-17.0.5-3.fc25.x86_64                         1/80 
  Upgrading   : mesa-libgbm-17.0.5-3.fc25.x86_64                           2/80 
  Upgrading   : libdb-5.3.28-21.fc25.x86_64                                3/80 
  Installing  : kernel-core-4.11.3-202.fc25.x86_64                         4/80 
  Installing  : kernel-modules-4.11.3-202.fc25.x86_64                      5/80 
  Upgrading   : mesa-libEGL-17.0.5-3.fc25.x86_64                           6/80 
  Upgrading   : mesa-libGL-17.0.5-3.fc25.x86_64                            7/80 
  Upgrading   : mesa-libGLES-17.0.5-3.fc25.x86_64                          8/80 
  Upgrading   : ntfs-3g-2:2017.3.23-1.fc25.x86_64                          9/80 
  Upgrading   : mesa-libwayland-egl-17.0.5-3.fc25.x86_64                  10/80 
  Upgrading   : mesa-filesystem-17.0.5-3.fc25.x86_64                      11/80 
  Upgrading   : hplip-common-3.17.4-2.fc25.x86_64                         12/80 
  Upgrading   : hplip-libs-3.17.4-2.fc25.x86_64                           13/80 
  Installing  : qpid-proton-c-0.17.0-2.fc25.x86_64                        14/80 
  Installing  : python-qpid-proton-0.17.0-2.fc25.x86_64                   15/80 
  Installing  : python-rhmsg-0.5-1.fc25.noarch                            16/80 
  Installing  : python2-simplejson-3.10.0-1.fc25.x86_64                   17/80 
  Installing  : rpmdiff-4.1.1-1.fc25.x86_64                               18/80 
  Installing  : rpmdiff-remote-4.1.1-1.fc25.x86_64                        19/80 
  Upgrading   : rhpkg-1.32-2.fc25.noarch                                  20/80 
  Upgrading   : hplip-3.17.4-2.fc25.x86_64                                21/80 
  Upgrading   : libsane-hpaio-3.17.4-2.fc25.x86_64                        22/80 
  Upgrading   : mesa-dri-drivers-17.0.5-3.fc25.x86_64                     23/80 
  Upgrading   : mesa-libwayland-egl-devel-17.0.5-3.fc25.x86_64            24/80 
  Upgrading   : ntfsprogs-2:2017.3.23-1.fc25.x86_64                       25/80 
  Upgrading   : mesa-libGLES-devel-17.0.5-3.fc25.x86_64                   26/80 
  Upgrading   : mesa-libGL-devel-17.0.5-3.fc25.x86_64                     27/80 
  Upgrading   : mesa-libEGL-devel-17.0.5-3.fc25.x86_64                    28/80 
  Installing  : kernel-4.11.3-202.fc25.x86_64                             29/80 
  Upgrading   : libdb-utils-5.3.28-21.fc25.x86_64                         30/80 
  Upgrading   : libdb-devel-5.3.28-21.fc25.x86_64                         31/80 
  Upgrading   : mesa-libOpenCL-17.0.5-3.fc25.x86_64                       32/80 
  Upgrading   : mesa-libgbm-devel-17.0.5-3.fc25.x86_64                    33/80 
  Upgrading   : vim-minimal-2:8.0.617-1.fc25.x86_64                       34/80 
  Upgrading   : qrencode-libs-3.4.4-1.fc25.x86_64                         35/80 
  Upgrading   : perl-threads-1:2.16-1.fc25.x86_64                         36/80 
  Upgrading   : perl-Net-HTTP-6.16-1.fc25.noarch                          37/80 
  Upgrading   : mesa-libxatracker-17.0.5-3.fc25.x86_64                    38/80 
  Upgrading   : libmicrohttpd-1:0.9.55-1.fc25.x86_64                      39/80 
  Upgrading   : kernel-headers-4.11.3-202.fc25.x86_64                     40/80 
  Upgrading   : hwdata-0.301-1.fc25.noarch                                41/80 
  Upgrading   : google-noto-emoji-fonts-20170523-1.fc25.noarch            42/80 
  Installing  : kernel-debug-devel-4.11.3-202.fc25.x86_64                 43/80 
  Erasing     : kernel-4.10.16-200.fc25.x86_64                            44/80 
  Cleanup     : mesa-libGL-devel-17.0.5-2.fc25.x86_64                     45/80 
  Cleanup     : mesa-libwayland-egl-devel-17.0.5-2.fc25.x86_64            46/80 
  Cleanup     : mesa-libgbm-devel-17.0.5-2.fc25.x86_64                    47/80 
  Cleanup     : mesa-libGLES-devel-17.0.5-2.fc25.x86_64                   48/80 
  Cleanup     : mesa-libGLES-17.0.5-2.fc25.x86_64                         49/80 
  Cleanup     : mesa-libEGL-devel-17.0.5-2.fc25.x86_64                    50/80 
  Cleanup     : libdb-devel-5.3.28-16.fc25.x86_64                         51/80 
  Cleanup     : libsane-hpaio-3.17.4-1.fc25.x86_64                        52/80 
  Cleanup     : hplip-3.17.4-1.fc25.x86_64                                53/80 
  Cleanup     : hplip-libs-3.17.4-1.fc25.x86_64                           54/80 
  Cleanup     : hplip-common-3.17.4-1.fc25.x86_64                         55/80 
  Erasing     : kernel-debug-devel-4.10.16-200.fc25.x86_64                56/80 
  Cleanup     : rhpkg-1.31-1.fc25.noarch                                  57/80 
  Cleanup     : perl-Net-HTTP-6.15-1.fc25.noarch                          58/80 
  Cleanup     : kernel-headers-4.11.3-200.fc25.x86_64                     59/80 
  Cleanup     : hwdata-0.300-1.fc25.noarch                                60/80 
  Cleanup     : google-noto-emoji-fonts-20170426-1.fc25.noarch            61/80 
  Cleanup     : mesa-dri-drivers-17.0.5-2.fc25.x86_64                     62/80 
  Cleanup     : mesa-libEGL-17.0.5-2.fc25.x86_64                          63/80 
  Cleanup     : mesa-libGL-17.0.5-2.fc25.x86_64                           64/80 
  Erasing     : kernel-modules-4.10.16-200.fc25.x86_64                    65/80 
  Cleanup     : ntfsprogs-2:2016.2.22-4.fc25.x86_64                       66/80 
  Cleanup     : mesa-libOpenCL-17.0.5-2.fc25.x86_64                       67/80 
  Cleanup     : libdb-utils-5.3.28-16.fc25.x86_64                         68/80 
  Cleanup     : mesa-filesystem-17.0.5-2.fc25.x86_64                      69/80 
  Cleanup     : libdb-5.3.28-16.fc25.x86_64                               70/80 
  Cleanup     : mesa-libgbm-17.0.5-2.fc25.x86_64                          71/80 
  Cleanup     : ntfs-3g-2:2016.2.22-4.fc25.x86_64                         72/80 
  Erasing     : kernel-core-4.10.16-200.fc25.x86_64                       73/80 
  Cleanup     : mesa-libglapi-17.0.5-2.fc25.x86_64                        74/80 
  Cleanup     : mesa-libwayland-egl-17.0.5-2.fc25.x86_64                  75/80 
  Cleanup     : vim-minimal-2:8.0.600-1.fc25.x86_64                       76/80 
  Cleanup     : qrencode-libs-3.4.2-6.fc24.x86_64                         77/80 
  Cleanup     : perl-threads-1:2.15-1.fc25.x86_64                         78/80 
  Cleanup     : mesa-libxatracker-17.0.5-2.fc25.x86_64                    79/80 
  Cleanup     : libmicrohttpd-1:0.9.54-1.fc25.x86_64                      80/80 
  Verifying   : kernel-core-4.11.3-202.fc25.x86_64                         1/80 
  Verifying   : kernel-4.11.3-202.fc25.x86_64                              2/80 
  Verifying   : kernel-debug-devel-4.11.3-202.fc25.x86_64                  3/80 
  Verifying   : kernel-modules-4.11.3-202.fc25.x86_64                      4/80 
  Verifying   : python-rhmsg-0.5-1.fc25.noarch                             5/80 
  Verifying   : rpmdiff-remote-4.1.1-1.fc25.x86_64                         6/80 
  Verifying   : rpmdiff-4.1.1-1.fc25.x86_64                                7/80 
  Verifying   : python2-simplejson-3.10.0-1.fc25.x86_64                    8/80 
  Verifying   : python-qpid-proton-0.17.0-2.fc25.x86_64                    9/80 
  Verifying   : qpid-proton-c-0.17.0-2.fc25.x86_64                        10/80 
  Verifying   : google-noto-emoji-fonts-20170523-1.fc25.noarch            11/80 
  Verifying   : hplip-3.17.4-2.fc25.x86_64                                12/80 
  Verifying   : hplip-libs-3.17.4-2.fc25.x86_64                           13/80 
  Verifying   : hplip-common-3.17.4-2.fc25.x86_64                         14/80 
  Verifying   : libsane-hpaio-3.17.4-2.fc25.x86_64                        15/80 
  Verifying   : hwdata-0.301-1.fc25.noarch                                16/80 
  Verifying   : kernel-headers-4.11.3-202.fc25.x86_64                     17/80 
  Verifying   : libdb-5.3.28-21.fc25.x86_64                               18/80 
  Verifying   : libdb-utils-5.3.28-21.fc25.x86_64                         19/80 
  Verifying   : libdb-devel-5.3.28-21.fc25.x86_64                         20/80 
  Verifying   : libmicrohttpd-1:0.9.55-1.fc25.x86_64                      21/80 
  Verifying   : mesa-dri-drivers-17.0.5-3.fc25.x86_64                     22/80 
  Verifying   : mesa-filesystem-17.0.5-3.fc25.x86_64                      23/80 
  Verifying   : mesa-libEGL-17.0.5-3.fc25.x86_64                          24/80 
  Verifying   : mesa-libEGL-devel-17.0.5-3.fc25.x86_64                    25/80 
  Verifying   : mesa-libGL-17.0.5-3.fc25.x86_64                           26/80 
  Verifying   : mesa-libglapi-17.0.5-3.fc25.x86_64                        27/80 
  Verifying   : mesa-libGL-devel-17.0.5-3.fc25.x86_64                     28/80 
  Verifying   : mesa-libGLES-17.0.5-3.fc25.x86_64                         29/80 
  Verifying   : mesa-libGLES-devel-17.0.5-3.fc25.x86_64                   30/80 
  Verifying   : mesa-libOpenCL-17.0.5-3.fc25.x86_64                       31/80 
  Verifying   : mesa-libgbm-17.0.5-3.fc25.x86_64                          32/80 
  Verifying   : mesa-libgbm-devel-17.0.5-3.fc25.x86_64                    33/80 
  Verifying   : mesa-libwayland-egl-17.0.5-3.fc25.x86_64                  34/80 
  Verifying   : mesa-libwayland-egl-devel-17.0.5-3.fc25.x86_64            35/80 
  Verifying   : mesa-libxatracker-17.0.5-3.fc25.x86_64                    36/80 
  Verifying   : ntfs-3g-2:2017.3.23-1.fc25.x86_64                         37/80 
  Verifying   : ntfsprogs-2:2017.3.23-1.fc25.x86_64                       38/80 
  Verifying   : perl-Net-HTTP-6.16-1.fc25.noarch                          39/80 
  Verifying   : perl-threads-1:2.16-1.fc25.x86_64                         40/80 
  Verifying   : qrencode-libs-3.4.4-1.fc25.x86_64                         41/80 
  Verifying   : rhpkg-1.32-2.fc25.noarch                                  42/80 
  Verifying   : vim-minimal-2:8.0.617-1.fc25.x86_64                       43/80 
  Verifying   : qrencode-libs-3.4.2-6.fc24.x86_64                         44/80 
  Verifying   : libdb-5.3.28-16.fc25.x86_64                               45/80 
  Verifying   : libsane-hpaio-3.17.4-1.fc25.x86_64                        46/80 
  Verifying   : libdb-utils-5.3.28-16.fc25.x86_64                         47/80 
  Verifying   : libdb-devel-5.3.28-16.fc25.x86_64                         48/80 
  Verifying   : ntfs-3g-2:2016.2.22-4.fc25.x86_64                         49/80 
  Verifying   : ntfsprogs-2:2016.2.22-4.fc25.x86_64                       50/80 
  Verifying   : libmicrohttpd-1:0.9.54-1.fc25.x86_64                      51/80 
  Verifying   : vim-minimal-2:8.0.600-1.fc25.x86_64                       52/80 
  Verifying   : kernel-4.10.16-200.fc25.x86_64                            53/80 
  Verifying   : rhpkg-1.31-1.fc25.noarch                                  54/80 
  Verifying   : kernel-core-4.10.16-200.fc25.x86_64                       55/80 
  Verifying   : kernel-debug-devel-4.10.16-200.fc25.x86_64                56/80 
  Verifying   : perl-Net-HTTP-6.15-1.fc25.noarch                          57/80 
  Verifying   : kernel-headers-4.11.3-200.fc25.x86_64                     58/80 
  Verifying   : kernel-modules-4.10.16-200.fc25.x86_64                    59/80 
  Verifying   : hplip-3.17.4-1.fc25.x86_64                                60/80 
  Verifying   : hplip-common-3.17.4-1.fc25.x86_64                         61/80 
  Verifying   : hplip-libs-3.17.4-1.fc25.x86_64                           62/80 
  Verifying   : hwdata-0.300-1.fc25.noarch                                63/80 
  Verifying   : google-noto-emoji-fonts-20170426-1.fc25.noarch            64/80 
  Verifying   : mesa-dri-drivers-17.0.5-2.fc25.x86_64                     65/80 
  Verifying   : mesa-filesystem-17.0.5-2.fc25.x86_64                      66/80 
  Verifying   : mesa-libEGL-devel-17.0.5-2.fc25.x86_64                    67/80 
  Verifying   : mesa-libGL-17.0.5-2.fc25.x86_64                           68/80 
  Verifying   : mesa-libEGL-17.0.5-2.fc25.x86_64                          69/80 
  Verifying   : mesa-libGL-devel-17.0.5-2.fc25.x86_64                     70/80 
  Verifying   : mesa-libGLES-17.0.5-2.fc25.x86_64                         71/80 
  Verifying   : mesa-libGLES-devel-17.0.5-2.fc25.x86_64                   72/80 
  Verifying   : perl-threads-1:2.15-1.fc25.x86_64                         73/80 
  Verifying   : mesa-libOpenCL-17.0.5-2.fc25.x86_64                       74/80 
  Verifying   : mesa-libgbm-17.0.5-2.fc25.x86_64                          75/80 
  Verifying   : mesa-libgbm-devel-17.0.5-2.fc25.x86_64                    76/80 
  Verifying   : mesa-libglapi-17.0.5-2.fc25.x86_64                        77/80 
  Verifying   : mesa-libwayland-egl-17.0.5-2.fc25.x86_64                  78/80 
  Verifying   : mesa-libwayland-egl-devel-17.0.5-2.fc25.x86_64            79/80 
  Verifying   : mesa-libxatracker-17.0.5-2.fc25.x86_64                    80/80 

Removed:
  kernel.x86_64 4.10.16-200.fc25                                                
  kernel-core.x86_64 4.10.16-200.fc25                                           
  kernel-debug-devel.x86_64 4.10.16-200.fc25                                    
  kernel-modules.x86_64 4.10.16-200.fc25                                        

Installed:
  kernel.x86_64 4.11.3-202.fc25                                                 
  kernel-core.x86_64 4.11.3-202.fc25                                            
  kernel-debug-devel.x86_64 4.11.3-202.fc25                                     
  kernel-modules.x86_64 4.11.3-202.fc25                                         
  python-qpid-proton.x86_64 0.17.0-2.fc25                                       
  python-rhmsg.noarch 0.5-1.fc25                                                
  python2-simplejson.x86_64 3.10.0-1.fc25                                       
  qpid-proton-c.x86_64 0.17.0-2.fc25                                            
  rpmdiff.x86_64 4.1.1-1.fc25                                                   
  rpmdiff-remote.x86_64 4.1.1-1.fc25                                            

Upgraded:
  google-noto-emoji-fonts.noarch 20170523-1.fc25                                
  hplip.x86_64 3.17.4-2.fc25                                                    
  hplip-common.x86_64 3.17.4-2.fc25                                             
  hplip-libs.x86_64 3.17.4-2.fc25                                               
  hwdata.noarch 0.301-1.fc25                                                    
  kernel-headers.x86_64 4.11.3-202.fc25                                         
  libdb.x86_64 5.3.28-21.fc25                                                   
  libdb-devel.x86_64 5.3.28-21.fc25                                             
  libdb-utils.x86_64 5.3.28-21.fc25                                             
  libmicrohttpd.x86_64 1:0.9.55-1.fc25                                          
  libsane-hpaio.x86_64 3.17.4-2.fc25                                            
  mesa-dri-drivers.x86_64 17.0.5-3.fc25                                         
  mesa-filesystem.x86_64 17.0.5-3.fc25                                          
  mesa-libEGL.x86_64 17.0.5-3.fc25                                              
  mesa-libEGL-devel.x86_64 17.0.5-3.fc25                                        
  mesa-libGL.x86_64 17.0.5-3.fc25                                               
  mesa-libGL-devel.x86_64 17.0.5-3.fc25                                         
  mesa-libGLES.x86_64 17.0.5-3.fc25                                             
  mesa-libGLES-devel.x86_64 17.0.5-3.fc25                                       
  mesa-libOpenCL.x86_64 17.0.5-3.fc25                                           
  mesa-libgbm.x86_64 17.0.5-3.fc25                                              
  mesa-libgbm-devel.x86_64 17.0.5-3.fc25                                        
  mesa-libglapi.x86_64 17.0.5-3.fc25                                            
  mesa-libwayland-egl.x86_64 17.0.5-3.fc25                                      
  mesa-libwayland-egl-devel.x86_64 17.0.5-3.fc25                                
  mesa-libxatracker.x86_64 17.0.5-3.fc25                                        
  ntfs-3g.x86_64 2:2017.3.23-1.fc25                                             
  ntfsprogs.x86_64 2:2017.3.23-1.fc25                                           
  perl-Net-HTTP.noarch 6.16-1.fc25                                              
  perl-threads.x86_64 1:2.16-1.fc25                                             
  qrencode-libs.x86_64 3.4.4-1.fc25                                             
  rhpkg.noarch 1.32-2.fc25                                                      
  vim-minimal.x86_64 2:8.0.617-1.fc25                                           

etckeeper warning: hardlinked files could cause problems with git:
chromium/native-messaging-hosts/browser_test_resources/mock_client_plugin.js
chromium/native-messaging-hosts/browser_test_resources/mock_host_list_api.js
chromium/native-messaging-hosts/browser_test_resources/mock_identity.js
chromium/native-messaging-hosts/browser_test_resources/mock_signal_strategy.js
chromium/native-messaging-hosts/browser_test_resources/sinon.js
chromium/native-messaging-hosts/unittests/mock_client_plugin.js
chromium/native-messaging-hosts/unittests/mock_host_list_api.js
chromium/native-messaging-hosts/unittests/mock_identity.js
chromium/native-messaging-hosts/unittests/mock_signal_strategy.js
chromium/native-messaging-hosts/unittests/sinonjs/sinon.js
Complete!
Segmentation fault (core dumped)

Comment 49 Eric Blake 2017-06-09 20:54:14 UTC

Where would the core dump be saved, or am I beyond hope there? ABRT didn't seem to see a recent crash

Comment 50 Adam Williamson 2017-06-09 20:56:17 UTC

If you can't find it in /var/spool/abrt or /var/tmp/abrt or / , probably no chance (I don't think the systemd coredump thingy is enabled on F25)...

Comment 51 Adam Williamson 2017-06-09 21:02:06 UTC

Other reports:

https://bugzilla.redhat.com/show_bug.cgi?id=1394862#c92

"That's not true. I was just hit with -21.fc25 from updates which broke
dnf until I ran rpm --rebuilddb manually." - Dominik Mierzejewski, on devel@

Dominik managed to get abrt to file a crash report, in fact, with stack trace: https://bugzilla.redhat.com/show_bug.cgi?id=1460303

I think I saw another report on IRC, but can't find it right now.

Comment 52 Eric Blake 2017-06-09 21:07:43 UTC

Okay, I found the core dump in /var/spool/abrt/ccpp-2017-06-09-15\:14\:27-8088/, but I can't seem to make ABRT parse it, and right now I have a lame backtrace:

(gdb) bt
#0  0x0000559ddacdd528 in ?? ()
#1  0x00007f41b8cc7f30 in ?? ()
#2  0x0000559ddc834670 in ?? ()
#3  0x00007f41b8cc7f50 in ?? ()
#4  0x0000559ddad040e7 in ?? ()
#5  0x0000000000000000 in ?? ()

Comment 53 Eric Blake 2017-06-09 21:09:41 UTC

(In reply to Eric Blake from comment #52)
> Okay, I found the core dump in
> /var/spool/abrt/ccpp-2017-06-09-15\:14\:27-8088/, but I can't seem to make
> ABRT parse it, and right now I have a lame backtrace:
> 
> (gdb) bt
> #0  0x0000559ddacdd528 in ?? ()
> #1  0x00007f41b8cc7f30 in ?? ()
> #2  0x0000559ddc834670 in ?? ()
> #3  0x00007f41b8cc7f50 in ?? ()
> #4  0x0000559ddad040e7 in ?? ()
> #5  0x0000000000000000 in ?? ()

Sorry for the noise; I just read ccpp-*/cmdline, and realized that it is an unrelated dump also from today, so it's not helpful here. There's nothing in /var/spool for my dnf crash :(

Comment 54 Petr Kubat 2017-06-12 07:34:09 UTC

Looking into this right now but from what I can see the issue should be easily fixable by running "rpmdb --rebuilddb" if you already hit the crash.

Comment 55 Michal Halenka 2017-06-13 08:30:45 UTC

I can confirm that

Comment 56 Jason Montleon 2017-06-16 04:01:06 UTC

I am seeing the same on one system with -22:

$ rpm -q libdb; sudo dnf -y update
libdb-5.3.28-22.fc26.x86_64
error: db5 error(5) from dbenv->open: Input/output error
error: cannot open Packages index using db5 - Input/output error (5)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

rpm --rebuilddb gets it back in order, but I use dnf-automatic and have found that it's busted more than it is not over the last few days.

Comment 57 Adam Williamson 2017-06-16 07:03:43 UTC

yes, it's known that it happens on update from -21 to -22, which is why we withdrew -22.

Comment 58 Michael Catanzaro 2017-06-23 19:03:07 UTC

So this is an accepted F26 beta blocker, but not yet closed. What's up here? Does it need to be moved to F26 final blocker?

Comment 59 Adam Williamson 2017-06-23 19:34:04 UTC

It was being left open while we sorted through all the messy consequences of the updates, but the bug covered *here* is in fact fixed by -21. Since we decided to just leave -21 in stable and help people who hit the RPM db issue on update to -21 one-by-one, there's really not a lot left to do here, and I think we can just go ahead and close the bug.

Note You need to log in before you can comment on or make changes to this bug.

aliakc
awilliam
eblake
fweimer
ignatenko
jmontleo
kardos.lubos
lsatenstein
mcatanzaro+wrong-account-do-not-cc
michal.halenka
mruckman
novyjindrich
packaging-team-maint
pknirsch
pkubat
pmatilai
redhat-bugzilla
rharwood
rjones
rmatos
robatino
samuel-rhbugs
sgallagh
striker