We saw a similar case (arch agnostic) with texlive, where a scriptlet was wrong and entered an endless loop. Guy, can't you check from console if there is something running when the dnf update process is stuck?

Dan, unfortunately I didn't find a way to do that. After the "dnf system-upgrade reboot", the upgrade process takes the focus of the console on the boot screen and I am not able to use this console or to connect to the guest.

Well, that's odd.  I don't know why it would hang here; there isn't anything that can loop.  The only change from -4 to -6 is the addition of this trigger https://github.com/frozencemetery/krb5_fedora/blob/master/krb5.spec#L515-L529 so I assume it's hanging there.

You mention it's failing on ppc64{,le}; if it's not failing on any other architectures, I don't think this is a krb5 problem (maybe dnf?).

ok I will try to add code to get a trace of what is running by krb5 at reboot during the "system-upgrade" to see if krb5 is involved.

I'm hitting this on aarch64 and armhfp during server upgrade. Seemingly hanging during cleanup of 'crypto-policies-20160921-4.gitf3018dd.fc25.noarch'

armhfp:
[968/1030] Cleanup gzip-1.8-1.fc25.armv7hl...
[969/1030] Cleanup crypto-policies-20160921-4.gitf3018dd.fc25.noarch...
[970/1030] Cleanup krb5-libs-1.14.4-7.fc25.armv7hl...

aarch64:
[979/1118] Cleanup python3-libs-3.5.3-4.fc25.aarch64...
[980/1118] Cleanup system-python-libs-3.5.3-4.fc25.aarch64...
[981/1118] Cleanup crypto-policies-20160921-4.gitf3018dd.fc25.noarch...

That's interesting: you're both on an older version of Fedora, and seeing this on different packages.  Let's ask the dnf folks.

you could run --rpmverbosity=debug and see what's going on there..

Created attachment 1274331 [details]
dnf rpm log

Upgrade attempted with:

dnf system-upgrade download --releasever=26 --debuglevel 9 --rpmverbosity=debug
dnf system-upgrade --debuglevel 9 --rpmverbosity=debug reboot

Logs attached.

Created attachment 1274333 [details]
dnf.log

Created attachment 1274977 [details]
truncated strace log with maxcpus=1

This hang is also reproducible with 'dnf distro-sync --releasever=26' and allows for network access for troubleshooting. 

It does not always happen with the same package during Cleanup, and can also occur during the 'Upgrade' phase. When it does hang, it ends with 'FUTEX_WAIT' even when limiting maxcpus=1.

Hanging in the futex(FUTEX_WAIT) syscall often means a deadlock situation in a multi-threaded application ... The kernel bug with the same symptom should be fixed for a long time.

Also reproducible by attempting to upgrade dnf and deps to f26

dnf --releasever=26 update dnf -y
...
 Installing  : libXft-2.3.2-5.fc26.aarch64                              60/127 
 Installing  : tk-1:8.6.6-1.fc26.aarch64                                61/127 
 Installing  : python35-3.5.2-9.fc26.aarch64                            62/127 



[root@mustang ~]# strace -f -p 1030
strace: Process 1030 attached
futex(0xffff8fc717d0, FUTEX_WAIT, 2, NULL


Upgrading dnf was successful 5/6 attempts. After dnf is upgraded, distro-sync completed 5/5.

Proposing as a blocker for Beta, criteria:

"... it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

Discussed during the 2017-05-01 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker was made as it violates the following blocker criteria:

"... it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2017-05-01/f26-blocker-review.2017-05-01-16.02.txt

*** Bug 1443610 has been marked as a duplicate of this bug. ***

Also a problem when upgrading on armhfp.

s390x is affected as well

So Igor says this is basically https://bugzilla.redhat.com/show_bug.cgi?id=1394862 , which upstream has a possible fix for. He has done F25 and F26 scratch builds with the fix:

https://koji.fedoraproject.org/koji/taskinfo?taskID=19706601
https://koji.fedoraproject.org/koji/taskinfo?taskID=19706649

can affected folks on armhfp please test in the following way?

1) Install the patched F25 scratch build
2) Do your upgrade in such a way that the patched F26 scratch build will be included

In other words, the fix needs to be applied 'at both ends'.

Due to Koji's F25 config, there are no builds for aarch64 or ppc64 - if you want to test on those arches you'll have to grab the SRPM and rebuild it. To test on s390 you'll have to rebuild both F25 and F26 from SRPM.

using those scratch builds, f25 minimal successfully upgraded to f26 on armhfp.

libdb-5.3.28-18.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

libdb-5.3.28-18.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf

libdb-5.3.28-18.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Thanks. I've now submitted official updates for 24, 25 and 26 with the patch applied. If more folks could test and confirm the fix (and provide karma on the updates), that'd be great. We should ensure all three updates are pushed stable before closing this bug.

So it seems like this libdb update is causing problems in the compose process; I'm going to unpush all the updates for safety:

<adamw> there seem to be some issues in the compose process; both the RC compose attempts got stuck in STARTED state
<jkurik> afaik some builders went down and needed to be restarted, that slowed down the compose
<dgilmore> adamw: from looking there may be issues witha  libdb update
<nirik> right... they are hanging in lorax... and several failed in lorax -> libdb
<adamw> ooh.
<adamw> there's a libdb blocker fix which is supposed to *prevent* update hangs...
<dgilmore> it may be causeing compose hangs
<nirik> yep.
<adamw> maybe the fix has problems
<adamw> can you tell what libdb nevr is involved?
<adamw> well, maybe we should do this in #fedora-releng...
<dgilmore> DEBUG util.py:439:  (68/714) libdb-5.3.28-19.fc26.aarch64
<adamw> mmf, that's the latest one :/

CCing dgilmore and nirik for further details.

Here's a failed koji task from the attempted rc compose: 

https://koji.fedoraproject.org/koji/taskinfo?taskID=19719675

Looking at root.log there: 

DEBUG util.py:439:  creating the runtime image
DEBUG util.py:439:  /bin/sh: line 1: 12545 Floating point exception(core dumped) lorax --product=Fedora --version=26 --release=26 --source=file:///mnt/koji/compose/26/Fedora-26-20170524.1/work/x86_64/repo --variant=Everything --noupgrade --buildarch=x86_64 --volid=Fedora-E-dvd-x86_64-26 --logfile=/mnt/koji/compose/26/Fedora-26-20170524.1/logs/x86_64/buildinstall-Everything-logs/lorax.log /mnt/koji/compose/26/Fedora-26-20170524.1/work/x86_64/buildinstall/Everything
DEBUG util.py:577:  Child return code was: 136

Looking on that builder ( buildhw-01.phx2.fedoraproject.org ):

[Wed May 24 04:34:45 2017] traps: lorax[37508] trap divide error ip:7f5d7af41b04 sp:7ffe7afae730 error:0
[Wed May 24 04:34:45 2017]  in libdb-5.3.so[7f5d7ae64000+1b5000]
[Wed May 24 04:37:17 2017] traps: lorax[37259] trap divide error ip:f56eff8c sp:ffd6f1c0 error:0
[Wed May 24 04:37:17 2017]  in libdb-5.3.so[f5604000+1d1000]
[Wed May 24 04:41:25 2017] traps: lorax[36579] trap divide error ip:7fb5ecf68b04 sp:7ffe0644ed70 error:0
[Wed May 24 04:41:25 2017]  in libdb-5.3.so[7fb5ece8b000+1b5000]
[Wed May 24 23:30:05 2017] traps: lorax[12545] trap divide error ip:7f0cbd3cfb04 sp:7ffed72fd010 error:0
[Wed May 24 23:30:05 2017]  in libdb-5.3.so[7f0cbd2f2000+1b5000]
[Wed May 24 23:32:07 2017] traps: lorax[12320] trap divide error ip:f569cf8c sp:ff8e5610 error:0
[Wed May 24 23:32:07 2017]  in libdb-5.3.so[f55b1000+1d1000]
[Wed May 24 23:32:23 2017] traps: lorax[12313] trap divide error ip:f56b5f8c sp:ffc11490 error:0
[Wed May 24 23:32:23 2017]  in libdb-5.3.so[f55ca000+1d1000]
[Wed May 24 23:38:25 2017] traps: lorax[12491] trap divide error ip:7fed6c14bb04 sp:7ffd94c8aef0 error:0
[Wed May 24 23:38:25 2017]  in libdb-5.3.so[7fed6c06e000+1b5000]

It's worth noting that all compose tasks run in mock chroots, so the issue here could be something missing from the mock environment that the code assumes will be available.

(In reply to Adam Williamson from comment #26)
> It's worth noting that all compose tasks run in mock chroots, so the issue
> here could be something missing from the mock environment that the code
> assumes will be available.

Or the RPM database was created from outside of the chroot in the old libdb format, and the first attempt to touch it triggers an upgrade attempt, which fails.

libdb-5.3.28-19.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Created attachment 1282477 [details]
upgrade_fail

This could be relate to this bug. During upgrade my dnf crashed with:

...
  Running scriptlet: qpdf-libs-6.0.0-4.fc26.x86_64                                                                                                                                                      107/112 
  Cleanup          : microcode_ctl-2:2.1-14.1.fc26.x86_64                                                                                                                                               108/112 
  Cleanup          : libv4l-1.12.5-1.fc26.x86_64                                                                                                                                                        109/112 
  Running scriptlet: libv4l-1.12.5-1.fc26.x86_64                                                                                                                                                        109/112 
  Running scriptlet: gssproxy-0.7.0-7.fc26.x86_64                                                                                                                                                       110/112 
Floating point exception (core dumped)

Now even 'rpm' doesn't work it crashes.

> Now even 'rpm' doesn't work it crashes.

That happened to me yesterday during a yum update of a F26 guest. It looks like the rpm database format changed. You have to rebuild the rpm database. I followed the instructions at https://ma.ttias.be/yum-update-db_runrecovery-fatal-error-run-database-recovery/ (you'd use the corresponding instructions for dnf) and then ran yum-complete-transaction (don't know the dnf command) and then everything was fine again.

I take it back - AFAIK the rpm database format didn't change, but as I said, rebuilding the rpm database did fix my problem.

It certainly seems plausible that you ran into basically the same problem the composes are hitting, whatever it is, and a database rebuild would shift it. I've now unpushed the update for F26 as well, so hopefully no-one else will run into it.

I think there is some scriptlet magic involved in this.
Just tested to a newer libdb+glibc version with having openldap-servers package installed (which I know for a fact has a triggerin on libdb and touches the rpmdb) and failed the same way. Most likely due to the fact that the trigger rebuilds the db with newer version of libdb and rpm then tries to open it with the older version... But will have to look into this further.

The database *environment* changes with the new libdb version, and once the database has been accessed with the newer version the older version can no longer access it and will barf in one way or the other. At least that's what I saw on Wednesday in my brief testing of the new libdb.

Anyway, you don't need a full --rebuilddb in this situation, "rm -f /var/lib/rpm/__*.db" to nuke the environment is enough.

Moving to assigned because upgrade to libdb-5.3.28-19.fc26
still does not work as expected and rpmdb is broken

Hi,

I'm seeing this too, not sure if it is related to the libdb update, I've updated to F26 a while ago and I already have manually run rpm --rebuilddb, yet after sucking in just 2 days of F26 updates today my dnf update is stuck at:

  Running scriptlet: gssproxy-0.7.0-7.fc26.x86_64                     1029/1040 

(yes lots of updates in the last 2 days, texlive is in the set)

So it seems that the gssproxy scriptlet is doing something which triggers this.

Note no fp exception for me just dnf stuck using 100% CPU.

Regards,

Hans

A bit more input, I had to kill -9 dnf, after that dnf repoquery --duplicates showed a few duplicates so I did
dnf remove --duplicates, this time the gssproxy scriptlet did run successfully.

I did do a rpm --rebuild after they kill -9 as I was not sure the rpmdb would still be in a sane state after that.

Not sure if this is helpful, feel free to ignore.

Hans,
It is not related to gssproxy. It crashed even with updating just a libdb

sh# dnf update libdb*
Last metadata expiration check: 0:13:01 ago on Fri May 26 12:36:04 2017 CEST.
Dependencies resolved.
================================================================================
 Package          Arch        Version                Repository            Size
================================================================================
Upgrading:
 libdb            x86_64      5.3.28-19.fc26         updates-testing      749 k
 libdb-utils      x86_64      5.3.28-19.fc26         updates-testing      137 k

Transaction Summary
================================================================================
Upgrade  2 Packages

Total download size: 886 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): libdb-utils-5.3.28-19.fc26.x86_64.rpm    4.8 MB/s | 137 kB     00:00    
(2/2): libdb-5.3.28-17.fc26_5.3.28-19.fc26.x86_ 7.8 MB/s | 236 kB     00:00    
[DRPM] libdb-5.3.28-17.fc26_5.3.28-19.fc26.x86_64.drpm: done                   
--------------------------------------------------------------------------------
Total                                           337 kB/s | 373 kB     00:01     
Delta RPMs reduced 0.9 MB of updates to 0.4 MB (57.1% saved)
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : libdb-5.3.28-19.fc26.x86_64                            1/4 
  Running scriptlet: libdb-5.3.28-19.fc26.x86_64                            1/4 
Segmentation fault (core dumped)

sh# rpm -q libdb
Segmentation fault (core dumped)

*** Bug 1455823 has been marked as a duplicate of this bug. ***

*** Bug 1455864 has been marked as a duplicate of this bug. ***

I hit submit too fast.

sh# rpm -q libdb
Segmentation fault (core dumped)

sh# rpm --rebuilddb 

sh# rpm -q libdb
libdb-5.3.28-19.fc26.x86_64
libdb-5.3.28-17.fc26.x86_64

sh# rpm -q --scripts libdb
postinstall program: /sbin/ldconfig
postuninstall program: /sbin/ldconfig
postinstall program: /sbin/ldconfig
postuninstall program: /sbin/ldconfig

I doubt anyone tested package before pushing to updates-testing.

BTW it is not bug in dnf because similar problems are with yum-deprecated or microdnf. It is bug in libdb

Lukas, try to understand that not everybody's experience equals yours. 

For example I did test the thing and saw ZERO crashes, only when downgrading back to previous libdb version did I see problems (which doesn't always happen but is not unexpected). 

The actual bug is that librpm doesn't sufficiently isolate libdb from the rest of the world, and s*** like this hits the fan once every few years in that one transaction where some deep magic like glibc futex internals change. Truly fixing that would likely require drastic architectural change in rpm to force all database access through a single daemon so there can never be two different versions trying to access the same db.

(In reply to Panu Matilainen from comment #42)
> Lukas, try to understand that not everybody's experience equals yours. 
> 
> For example I did test the thing and saw ZERO crashes, only when downgrading
> back to previous libdb version did I see problems (which doesn't always
> happen but is not unexpected). 
> 
> The actual bug is that librpm doesn't sufficiently isolate libdb from the
> rest of the world, and s*** like this hits the fan once every few years in
> that one transaction where some deep magic like glibc futex internals
> change. Truly fixing that would likely require drastic architectural change
> in rpm to force all database access through a single daemon so there can
> never be two different versions trying to access the same db.

Pannu, what did you test and where/which machine?

Because upgrade on fedora 26 does not work for me with rpm, yum-deprecated (upgrade does not finish) snd -> crash. And I tested with 3 different fedora 26 machines. It crashed even in f26 docker containers(x86_64, i686) which I use for integration testing of unrelated project. It does not look like a corner-case to me. Or only very low change that only I hit bug everywhere.

BTW If you really cannot reproduce yourself and need more logs/info/valgrind output ... then I will be glad to provide them.

I don't know if that's the original issue, but I also experienced dnf crash on update of Fedora 26 that required "rpm --rebuilddb". I experienced the same issue on two different systems in the recent hours. Both track Fedora 26. One was upgraded two months ago (when alpha was released), another was installed as Fedora 26 around the same time. Both systems have fedora-updates-testing enabled. The symptoms are:

"dnf upgrade" tries to upgrade a large number of packages and crashes:

  Running scriptlet: qpdf-libs-6.0.0-4.fc26.x86_64
  Cleanup          : microcode_ctl-2:2.1-14.1.fc26.x86_64
  Cleanup          : libv4l-1.12.5-1.fc26.x86_64
  Running scriptlet: libv4l-1.12.5-1.fc26.x86_64
  Running scriptlet: gssproxy-0.7.0-7.fc26.x86_64
Floating point exception

Kernel log shows divide error in deleted (!!!) libdb-5.3.so

[  340.774423] traps: dnf[2177] trap divide error ip:7fdd2eaa9f32 sp:7ffc3f07cb00 error:0 in libdb-5.3.so (deleted)[7fdd2e9d2000+1b5000]

"package-cleanup --cleandupes" segfaults

[  413.885836] traps: package-cleanup[4076] general protection ip:7f86c15d6683 sp:7ffcc0cf2190 error:0 in libdb-5.3.so[7f86c14b1000+1b6000]

"rpm --rebuilddb" works

"package-cleanup --cleandupes" works

  Erasing    : 1:dmidecode-3.0-8.fc26.x86_64
  Erasing    : gssproxy-0.7.0-7.fc26.x86_64
  Erasing    : gperftools-libs-2.5.91-1.fc26.x86_64

"dnf check" passes

Lukas, Pavel and anyone other hitting this issue:
Please provide us with result of either "dnf --rpmverbosity debug update", or "rpm -Uvv" when upgrading through rpm. This will tell us much more about where in the update process the failure takes place.

Thank you!

I cannot reproduce the issue by downgrading the packages that were upgraded today and redoing the upgrade. But I noticed that if I downgrade libdb, I need to run "rpm --rebuilddb". I don't know if it's normal.

(In reply to Pavel Roskin from comment #46)
> I cannot reproduce the issue by downgrading the packages that were upgraded
> today and redoing the upgrade. But I noticed that if I downgrade libdb, I
> need to run "rpm --rebuilddb". I don't know if it's normal.

It looks like you have reproduced the issue, if your RPM database broke. Sometimes the crash happens after dnf finishes or in a separate process using RPM, e.g. dnf makecache.

[root@host rpms]# rpm -qa "libdb*"
libdb-utils-5.3.28-17.fc26.x86_64
libdb-5.3.28-17.fc26.x86_64

[root@host rpms]# rpm -Uvvv libdb-5.3.28-19.fc26.x86_64.rpm  libdb-utils-5.3.28-19.fc26.x86_64.rpm
ufdio:       1 reads,    16960 total bytes in 0.000016 secs
ufdio:       1 reads,     4874 total bytes in 0.000001 secs
ufdio:       1 reads,    16960 total bytes in 0.000006 secs
D: ============== libdb-5.3.28-19.fc26.x86_64.rpm
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Name 0x400 mode=0x0
D:  read h#      50 Header SHA1 digest: OK (3faba9ffbffc3a2d283452d115959d3d0c6fe2fa)
D: added key gpg-pubkey-fdb19c98-56fd6333 to keyring
D:  read h#     983 Header SHA1 digest: OK (44dfac3a3ed2ad52f6139c5313da593614f56803)
D: added key gpg-pubkey-34ec9cba-54e38751 to keyring
D:  read h#    1020 Header SHA1 digest: OK (cfe21d873624cf8171e50d34136dbc005df076ed)
D: added key gpg-pubkey-81b46521-55b3ca9a to keyring
D:  read h#    1894 Header SHA1 digest: OK (5e75131638942274458bbc47adc744c3f502a8e8)
D: added key gpg-pubkey-64dab85d-57d33e22 to keyring
D:  read h#    1923 Header SHA1 digest: OK (10c972a7cd84e939380e135e42e6566833a140f6)
D: added key gpg-pubkey-f5282ee4-58ac92a3 to keyring
D:  read h#    2062 Header SHA1 digest: OK (6ece1dbaaa4150376442430d5a07f4d54c135dec)
D: added key gpg-pubkey-98ab5139-4bf2d0b0 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: Expected size:       767222 = lead(96)+sigs(4324)+pad(4)+data(762798)
D:   Actual size:       767222
D: libdb-5.3.28-19.fc26.x86_64.rpm: Header SHA1 digest: OK (cc6f9b160a1c8a2f128b48d77a88bd8c1009d710)
ufdio:       6 reads,    14370 total bytes in 0.000009 secs
D: Plugin: calling hook init in systemd_inhibit plugin
D:  read h#     750 Header SHA1 digest: OK (fd008120d31275ac46e0759e0e41ef7d9d4f9fbc)
D:      added binary package [0]
D: ============== libdb-utils-5.3.28-19.fc26.x86_64.rpm
D: Expected size:       140406 = lead(96)+sigs(4324)+pad(4)+data(135982)
D:   Actual size:       140406
D: libdb-utils-5.3.28-19.fc26.x86_64.rpm: Header SHA1 digest: OK (863b47cdec17d7cda4df885dd1438a8629c31f04)
ufdio:       6 reads,    20738 total bytes in 0.000014 secs
D:  read h#    1547 Header SHA1 digest: OK (137fb7a99f7ae0f5a59fd12b724a29c348be1e60)
D:      added binary package [1]
D: found 0 source and 2 binary packages
D: opening  db index       /var/lib/rpm/Conflictname 0x400 mode=0x0
D: opening  db index       /var/lib/rpm/Requirename 0x400 mode=0x0
D: ========== +++ libdb-5.3.28-19.fc26 x86_64/linux 0x2
D: opening  db index       /var/lib/rpm/Basenames 0x400 mode=0x0
D:  read h#    1413 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: /sbin/ldconfig                                YES (db files)
D:  Requires: /sbin/ldconfig                                YES (cached)
D: opening  db index       /var/lib/rpm/Providename 0x400 mode=0x0
D:  Requires: libc.so.6()(64bit)                            YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.14)(64bit)                  YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.15)(64bit)                  YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.17)(64bit)                  YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.2.5)(64bit)                 YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3)(64bit)                   YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3.4)(64bit)                 YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.4)(64bit)                   YES (db provides)
D:  Requires: libpthread.so.0()(64bit)                      YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.2.5)(64bit)           YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.3.2)(64bit)           YES (db provides)
D:  Requires: rpmlib(CompressedFileNames) <= 3.0.4-1        YES (rpmlib provides)
D:  Requires: rpmlib(FileDigests) <= 4.6.0-1                YES (rpmlib provides)
D:  Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1       YES (rpmlib provides)
D:  Requires: rpmlib(PayloadIsXz) <= 5.2-1                  YES (rpmlib provides)
D:  Requires: rtld(GNU_HASH)                                YES (db provides)
D:  read h#     851 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D: Conflicts: filesystem < 3                                NO  
D: opening  db index       /var/lib/rpm/Obsoletename 0x400 mode=0x0
D: ========== +++ libdb-utils-5.3.28-19.fc26 x86_64/linux 0x2
D:  Requires: libc.so.6()(64bit)                            YES (cached)
D:  Requires: libc.so.6(GLIBC_2.14)(64bit)                  YES (cached)
D:  Requires: libc.so.6(GLIBC_2.2.5)(64bit)                 YES (cached)
D:  Requires: libc.so.6(GLIBC_2.3)(64bit)                   YES (cached)
D:  Requires: libc.so.6(GLIBC_2.3.4)(64bit)                 YES (cached)
D:  Requires: libc.so.6(GLIBC_2.4)(64bit)                   YES (cached)
D:  Requires: libdb(x86-64) = 5.3.28-19.fc26                YES (added provide)
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  Requires: libpthread.so.0()(64bit)                      YES (cached)
D:  Requires: libpthread.so.0(GLIBC_2.2.5)(64bit)           YES (cached)
D:  Requires: rpmlib(CompressedFileNames) <= 3.0.4-1        YES (rpmlib provides)
D:  Requires: rpmlib(FileDigests) <= 4.6.0-1                YES (rpmlib provides)
D:  Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1       YES (rpmlib provides)
D:  Requires: rpmlib(PayloadIsXz) <= 5.2-1                  YES (rpmlib provides)
D:  Requires: rtld(GNU_HASH)                                YES (cached)
D: ========== --- libdb-5.3.28-17.fc26 x86_64/linux 0x2
D:  read h#      70 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     228 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     376 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     417 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     446 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     453 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     501 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     550 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     683 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     763 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     801 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#     827 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1063 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1360 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1575 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1639 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1658 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1814 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    1860 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  read h#    2108 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D: ========== --- libdb-utils-5.3.28-17.fc26 x86_64/linux 0x2
D:  read h#    1021 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
D:  Requires: libdb-utils                                   YES (added provide)
D:  Requires: libdb-utils                                   YES (added provide)
D:  Requires: /usr/bin/db_stat                              YES (added files)
D: ========== recording tsort relations
D:  Requires: libdb(x86-64) = 5.3.28-19.fc26                YES (added provide)
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D:  Requires: libdb(x86-64) = 5.3.28-17.fc26                YES (added provide)
D:  Requires: libdb-5.3.so()(64bit)                         YES (added provide)
D: ========== tsorting packages (order, #predecessors, #succesors, depth)
D:     0    0    1    1   +libdb-5.3.28-19.fc26.x86_64
D:     1    0    0    2     +libdb-utils-5.3.28-19.fc26.x86_64
D:     2    0    1    1   -libdb-utils-5.3.28-17.fc26.x86_64
D:     3    0    0    2     -libdb-5.3.28-17.fc26.x86_64
D: installing binary packages
D: closed   db index       /var/lib/rpm/Packages
D: closed   db index       /var/lib/rpm/Obsoletename
D: closed   db index       /var/lib/rpm/Conflictname
D: closed   db index       /var/lib/rpm/Providename
D: closed   db index       /var/lib/rpm/Requirename
D: closed   db index       /var/lib/rpm/Basenames
D: closed   db index       /var/lib/rpm/Name
D: closed   db environment /var/lib/rpm
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages (none) mode=0x42
D: sanity checking 4 elements
D: opening  db index       /var/lib/rpm/Name (none) mode=0x42
D:  read h#     750 Header SHA1 digest: OK (fd008120d31275ac46e0759e0e41ef7d9d4f9fbc)
D:  read h#    1547 Header SHA1 digest: OK (137fb7a99f7ae0f5a59fd12b724a29c348be1e60)
D: Plugin: calling hook tsm_pre in selinux plugin
D: Plugin: calling hook tsm_pre in systemd_inhibit plugin
D: System shutdown blocked (fd 12)
D: opening  db index       /var/lib/rpm/Transfiletriggername (none) mode=0x42
D: opening  db index       /var/lib/rpm/Dirnames (none) mode=0x42
D: unknown: libdb-utils-5.3.28-17.fc26.x86_64 has 30 files
D: Plugin: calling hook psm_pre in selinux plugin
D: unknown: libdb-5.3.28-17.fc26.x86_64 has 7 files
D: Plugin: calling hook psm_pre in selinux plugin
D: running pre-transaction scripts
D: computing 74 file fingerprints
D: opening  db index       /var/lib/rpm/Basenames (none) mode=0x42
D: opening  db index       /var/lib/rpm/Group (none) mode=0x42
D: opening  db index       /var/lib/rpm/Requirename (none) mode=0x42
D: opening  db index       /var/lib/rpm/Providename (none) mode=0x42
D: opening  db index       /var/lib/rpm/Conflictname (none) mode=0x42
D: opening  db index       /var/lib/rpm/Obsoletename (none) mode=0x42
D: opening  db index       /var/lib/rpm/Triggername (none) mode=0x42
D: opening  db index       /var/lib/rpm/Installtid (none) mode=0x42
D: opening  db index       /var/lib/rpm/Sigmd5 (none) mode=0x42
D: opening  db index       /var/lib/rpm/Sha1header (none) mode=0x42
D: opening  db index       /var/lib/rpm/Filetriggername (none) mode=0x42
D: opening  db index       /var/lib/rpm/Recommendname (none) mode=0x42
D: opening  db index       /var/lib/rpm/Suggestname (none) mode=0x42
D: opening  db index       /var/lib/rpm/Supplementname (none) mode=0x42
D: opening  db index       /var/lib/rpm/Enhancename (none) mode=0x42
Preparing packages...
D: computing file dispositions
D: 0x00000028     4096      2542694           -1 /
D: ========== +++ libdb-5.3.28-19.fc26 x86_64-linux 0x2
D: Expected size:       767222 = lead(96)+sigs(4324)+pad(4)+data(762798)
D:   Actual size:       767222
D: libdb-5.3.28-19.fc26.x86_64: Header SHA1 digest: OK (cc6f9b160a1c8a2f128b48d77a88bd8c1009d710)
D:   install: libdb-5.3.28-19.fc26.x86_64 has 7 files
D: Plugin: calling hook psm_pre in selinux plugin
D:  read h#    1814 Header V3 RSA/SHA256 Signature, key ID 64dab85d: OK
libdb-5.3.28-19.fc26.x86_64
D: ========== Directories not explicitly included in package:
D:          0 /usr/lib64/
D:          1 /usr/share/doc/
D:          3 /usr/share/licenses/
D: ==========
D: create     100755  1 (   0,   0)1853896 /usr/lib64/libdb-5.3.so;59288a17
ufdio:      57 writes,  1853896 total bytes in 0.000611 secs
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/lib64/libdb-5.3.so;59288a17, system_u:object_r:lib_t:s0) 
D: create     120777  1 (   0,   0)    12 /usr/lib64/libdb-5.so;59288a17
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/lib64/libdb-5.so;59288a17, system_u:object_r:lib_t:s0) 
D: create     040755  1 (   0,   0)     0 /usr/share/doc/libdb
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/share/doc/libdb, system_u:object_r:usr_t:s0) 
D: create     100644  1 (   0,   0)   240 /usr/share/doc/libdb/README;59288a17
ufdio:       1 writes,      240 total bytes in 0.000014 secs
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/share/doc/libdb/README;59288a17, system_u:object_r:usr_t:s0) 
D: create     040755  1 (   0,   0)     0 /usr/share/licenses/libdb
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/share/licenses/libdb, system_u:object_r:usr_t:s0) 
D: create     100644  1 (   0,   0)  7310 /usr/share/licenses/libdb/LICENSE;59288a17
ufdio:       1 writes,     7310 total bytes in 0.000011 secs
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/share/licenses/libdb/LICENSE;59288a17, system_u:object_r:usr_t:s0) 
D: create     100644  1 (   0,   0) 26530 /usr/share/licenses/libdb/lgpl-2.1.txt;59288a17
ufdio:       1 writes,    26530 total bytes in 0.000018 secs
D: Plugin: calling hook fsm_file_prepare in selinux plugin
D: lsetfilecon: (/usr/share/licenses/libdb/lgpl-2.1.txt;59288a17, system_u:object_r:usr_t:s0) 
fdio:      94 reads,  1889100 total bytes in 0.053953 secs
D: adding "libdb" to Name index.
D: adding 7 entries to Basenames index.
D: adding "System Environment/Libraries" to Group index.
D: adding 18 entries to Requirename index.
D: adding 3 entries to Providename index.
D: adding 1 entries to Conflictname index.
D: adding 5 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "cc6f9b160a1c8a2f128b48d77a88bd8c1009d710" to Sha1header index.
D: %post(libdb-5.3.28-19.fc26.x86_64): scriptlet start
D: %post(libdb-5.3.28-19.fc26.x86_64): execv(/sbin/ldconfig) pid 32495
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/sbin/ldconfig) 
D: %post(libdb-5.3.28-19.fc26.x86_64): waitpid(32495) rc 32495 status 0
D: %triggerin(openldap-servers-2.4.44-10.fc26.x86_64): scriptlet start
fdio:       2 writes,      368 total bytes in 0.000032 secs
D: %triggerin(openldap-servers-2.4.44-10.fc26.x86_64): execv(/bin/sh) pid 32496
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/bin/sh) 
+ '[' 2 -eq 2 ']'
++ rpm -q '--qf=%{version}\n' libdb
++ wc -l
++ sort -u
++ sed 's/\.[0-9]*$//'
+ '[' 1 '!=' 1 ']'
+ rm -f /var/lib/ldap/rpm_upgrade_libdb
+ exit 0
D: %triggerin(openldap-servers-2.4.44-10.fc26.x86_64): waitpid(32496) rc 32496 status 0
error: rpmdb: DB_LOCK->lock_put: Lock is no longer valid
error: db5 error(22) from dbcursor->c_close: Invalid argument


And upgrade has not finished yet

rpm is busy (100%CPU) even after 60 minutes; so I killed it.

So it looks like I was really lucky that openldap-servers was installed in all my cases even though it was not running anywhere. I am so sorry that I was so pessimistic and a little bit frustrated.

(In reply to Lukas Slebodnik from comment #43)
> Pannu, what did you test and where/which machine?

It's Panu, with a single n, BTW.

I'm sure I tested something different than you did. Which is the whole point: people HAVE tested the fix in good faith without seeing the crashes you're seeing. Not everybody has openldap-servers installed, and (thank goodness) not many packages are naughty enough to call rpm from scriptlets. 

Running rpm queries manually during the upgrade would have the same effect so it's not all just about naughty packages either.

In any case, good thing is it did get caught before going further, updates-testing served its purpose.

*** Bug 1455750 has been marked as a duplicate of this bug. ***

For the record, we're transferring the blocker status from this bug to https://bugzilla.redhat.com/show_bug.cgi?id=1397087 , which is the specific bug we really voted on as a blocker issue. It's not really appropriate to grant blocker status to tracker bugs, as their 'meaning' could change over time - say in this case, we could resolve the current issue but then some much smaller bug which caused an issue only very rarely (and maybe only on a secondary arch) could be set as blocking this tracker, and then that would instantly 'become' a release blocker. Which we don't want.

*** Bug 1456788 has been marked as a duplicate of this bug. ***

*** Bug 1456002 has been marked as a duplicate of this bug. ***

libdb-5.3.28-21.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

using libdb-5.3.28-21.fc26 armhfp and aarch64 successfully upgraded with dnf-plugin-system-upgrade+local repo

It'd be really great if people could help test the new update hard: test that it fixes the actual bug, test it in the scenarios that caused trouble with -18 and -19, etc. We want to get this fix out for Beta, but we don't want to send it unless we're sure it's good. Thanks a lot!

libdb-5.3.28-21.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf

libdb-5.3.28-21.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

I did a check with ppc64le and the issue is still here. This is what I did
- installed F-25 "minimal"
- updated libdb to libdb-5.3.28-21.fc25.ppc64le (downloaded from ppc koji)
- created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le (downloaded from primary koji)
- run "dnf distro-sync --releasever 26" with the side repo enabled
- and got a hung after
...
  Cleanup     : crypto-policies-20160921-4.gitf3018dd.fc25.noarch  614/708 
  Cleanup     : krb5-libs-1.14.4-7.fc25.ppc64le                    615/708

(In reply to Dan Horák from comment #60)
> I did a check with ppc64le and the issue is still here. This is what I did
> - installed F-25 "minimal"
> - updated libdb to libdb-5.3.28-21.fc25.ppc64le (downloaded from ppc koji)
> - created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le
> (downloaded from primary koji)
> - run "dnf distro-sync --releasever 26" with the side repo enabled
> - and got a hung after
> ...
>   Cleanup     : crypto-policies-20160921-4.gitf3018dd.fc25.noarch  614/708 
>   Cleanup     : krb5-libs-1.14.4-7.fc25.ppc64le                    615/708

Dan, do you get hangs with dnf system-upgrade? It's just doing an online update that is problematic?

Well, my recognition is that 
* -18 and -19 applied a patch which are supposed to fix hang issue in -17, but it caused segv
* -21 removed the patch applied on -18 and -19, so -21 is effencially same as -17, and while segv is gone, hang issue in -17 "should" reappear.

Am I right?

(In reply to Mamoru TASAKA from comment #62)
> Well, my recognition is that 
> * -18 and -19 applied a patch which are supposed to fix hang issue in -17,
> but it caused segv
> * -21 removed the patch applied on -18 and -19, so -21 is effencially same
> as -17, and while segv is gone, hang issue in -17 "should" reappear.
> 
> Am I right?

Oops, removing patch was done in -20, -21 introduced other patch, sorry.

(In reply to Matthew Miller from comment #61)
> (In reply to Dan Horák from comment #60)
> > I did a check with ppc64le and the issue is still here. This is what I did
> > - installed F-25 "minimal"
> > - updated libdb to libdb-5.3.28-21.fc25.ppc64le (downloaded from ppc koji)
> > - created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le
> > (downloaded from primary koji)
> > - run "dnf distro-sync --releasever 26" with the side repo enabled
> > - and got a hung after
> > ...
> >   Cleanup     : crypto-policies-20160921-4.gitf3018dd.fc25.noarch  614/708 
> >   Cleanup     : krb5-libs-1.14.4-7.fc25.ppc64le                    615/708
> 
> Dan, do you get hangs with dnf system-upgrade? It's just doing an online
> update that is problematic?

- installed F-25 "minimal"
- created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le (downloaded from primary koji)
- run "dnf system-upgrade download --releasever 26" with the side repo enabled
- run "dnf system-upgrade reboot"
- and got a hung at
 [620/714]  Cleanup : crypto-policies-20160921-4.gitf3018dd.fc25.noarch
on the console

Well, that answers that. :( Thanks Dan.

Dan: did you check the correct libdb version was actually included in the distro-sync / upgrade transactions?

(In reply to Adam Williamson from comment #66)
> Dan: did you check the correct libdb version was actually included in the
> distro-sync / upgrade transactions?

it's listed in the "download" phase, so it should be then also installed/updated

...
 libcroco                                                  ppc64le                         0.6.12-1.fc26                                                 fedora                         121 k
 libcrypt-nss                                              ppc64le                         2.25-4.fc26                                                   fedora                          53 k
 libcurl                                                   ppc64le                         7.53.1-7.fc26                                                 fedora                         294 k
 libdb                                                     ppc64le                         5.3.28-21.fc26                                                libdb                          798 k
 libdb-utils                                               ppc64le                         5.3.28-21.fc26                                                libdb                          150 k
 libedit                                                   ppc64le                         3.1-17.20160618cvs.fc26                                       fedora                         107 k
 libfdisk                                                  ppc64le                         2.29.1-2.fc26                                                 fedora                         248 k
 libffi                                                    ppc64le                         3.1-10.fc26                                                   fedora                          35 k
 libgcc                                                    ppc64le                         7.1.1-1.fc26                                                  fedora                          76 k
...

libdb-5.3.28-21.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a4c41ecc27

Hi Dan,
can you reproduce this issue using debugging output so that we can see where dnf freezes? Running with "--rpmverbosity debug" should do the trick.

Also are you able to reproduce this even after updating libdb to the latest version for F25 (https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf) before running the upgrade to F26?

Looking at https://bugzilla.redhat.com/show_bug.cgi?id=1443415#c60 and https://bugzilla.redhat.com/show_bug.cgi?id=1443415#c64, Dan tried both options: with updating to -21 libdb on F25 and without.

>Dan tried both options: with updating to -21 libdb on F25 and without.

Missed that, thanks!

In any case the debug output would still be immensely helpful as I am currently unable to reproduce the hang on a ppc64le box.

(In reply to Petr Kubat from comment #69)
> Hi Dan,
> can you reproduce this issue using debugging output so that we can see where
> dnf freezes? Running with "--rpmverbosity debug" should do the trick.
> 
> Also are you able to reproduce this even after updating libdb to the latest
> version for F25
> (https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf) before
> running the upgrade to F26?

Petr, I'll check with you off-line and give you login info for the machine I use for reproducing the problem.

For debugging purposes I think we should stick to the online update using "dnf distro-sync", I couldn't access the console when running the off-line upgrade.

Seems I can't reproduce the issue any more with an up-to-date F-25 system, maybe something in the updates pushed over the weekend made the change.

For the record - the first in my procedure in comment #60 should read as "install and update F-25 minimal".

I cannot reproduce any hangs during the dist-upgrade when updating from an up-2-date fc24 or fc25, when having -21 libdb installed *before* upgrading to fc26.

libdb-5.3.28-21.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

(In reply to Björn "besser82" Esser from comment #75)
> I cannot reproduce any hangs during the dist-upgrade when updating from an
> up-2-date fc24 or fc25, when having -21 libdb installed *before* upgrading
> to fc26.

Right, and that's exactly the procedure that MUST be used, testing anything else is useless because it WILL fail unless libdb is updated to -21 first in a separate transaction.

Based on the messages this seems to be working except with ppc64le from which we're seeing a bit mixed results. Which makes me suspect maybe ppc64le has a different issue on top of the one being fixed here.

To extend upon Panu's comment:

The difference between upgrading from F24/25 with libdb updated to -21 and upgrading without the -21 libdb update already installed is that in the latter case (newly installed) libdb relies only on rpm's transaction lock to see whether it should rebuild rpmdb's environment due to VERSION_MISMATCH (change of libdb regions' internal structures) fixing the original glibc issue of bug 1394862. However since this is a feature of the -21 build there is a window in the update process after the new glibc is already installed and before the libdb -21 build gets installed during which an external process might freeze when accessing rpmdb's environment due to not being able to work with new glibc internals.

In the former case, having the libdb -21 build already installed on the system before running the upgrade to F26 makes libdb able to use its own internal flock-style lock of the environment (that is part of the fix for bug 1394862) during the upgrade instead of relying on the rpm's transaction lock (though the hack is still there to be able to install libdb -21 f24/25 build in the first place) so it would definitely be more safer running the upgrade this way.

In either of those cases I have yet to see a hang/crash get reproduced (if you can, let me know!).

I still have the problem in my side,
1 - installed F-25 ppc64le "minimal"
2 - created a side repo for F-25 libdb containing libdb-5.3.28-21.fc25.ppc64le 
and libdb-utils-5.3.28-21.fc25.ppc64le (downloaded from secondary koji)
3 - do a "dnf update" and check new libdbxx -21 fc25 are installed
4 - created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le and libdb-utils-5.3.28-21.fc26.ppc64le (downloaded from primary koji)
5 - run "dnf system-upgrade download --releasever 26" with the side repo F-26 enabled and check libdbxx -21 fc25 will be upgraded by libdbxx -21 fc26
6 - run "dnf system-upgrade reboot"
- and got a hung at 
[643/739] Cleanup crypto-policies-20160921-4.gitf3018dd.fc25.noarch...

Note that I hung the same way without creation of F-26 repo of step 4 (only f25 updated with libdb -21 fc25 and no libdb repo F-26 before the system-upgrade).

I tried today again and can't reproduce it on ppoc64le. Guy, could you store the list of installed rpms from the F-25 system before starting the update? I suspect something changed between Friday and Monday in the F-25 updates, so I would compare my list with yours.

Created attachment 1285467 [details]
dnf list installed after "dnf update"

dnf_list_installed.after_update is the list done just after the fc25 ppc64le installation update

Created attachment 1285468 [details]
before update

dnf_list_installed.before_upgrade is the list done after the fc25 ppc64le installation update + install of packages "createrepo, wget, libdb -21 and libdb-utils -21" (and dependencies).

libdb-5.3.28-21.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

libdb-5.3.28-21.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf

(In reply to Dan Horák from comment #17)
> s390x is affected as well

Upgrade issue seems to be fixed for s390x as well. Did following:
- Installed F25 and "dnf update"
- created side repo for F-26 with latest libdb and libdb-utils package from s390-koji  (libdb-utils-5.3.28-21.fc26.s390x and libdb-5.3.28-21.fc26.s390x)
- Ran "dnf distro-sync --releasever 26" with the side repo enabled
- distro-sync finished successfully
- Rebooted system successfully

Seems I can reproduce Guy's hang, with libdb-5.3.28-21.fc25.ppc64le updated before the upgrade starts.

Upgrade of f25 ppc64le to f26 succeed when I use "dnf distro-sync" process described by Sinny (comment #85) instead of "dnf system-upgrade" process used by openqa test (comment #1) and described by Paul (comment #8)

this is log from my stuck upgrade on ppc64le with libdb-5.3.28-21.fc25.ppc64le installed

...
D: ========== +++ crypto-policies-20160921-4.gitf3018dd.fc25 noarch-linux 0x0
D:     erase: crypto-policies-20160921-4.gitf3018dd.fc25.noarch has 46 files
D: Plugin: calling hook psm_pre in selinux plugin
D: skip       100644  1 (   0,   0)  2856 /usr/share/man/man8/update-crypto-policies.8.gz
D: skip       100644  1 (   0,   0) 26432 /usr/share/licenses/crypto-policies/COPYING.LESSER
D: skip       040755  2 (   0,   0)  4096 /usr/share/licenses/crypto-policies
D: skip       100644  1 (   0,   0)    70 /usr/share/crypto-policies/reload-cmds.sh
D: skip       100644  1 (   0,   0)   606 /usr/share/crypto-policies/default-config
D: skip       100644  1 (   0,   0)    87 /usr/share/crypto-policies/LEGACY/openssl.txt
D: skip       100644  1 (   0,   0)   431 /usr/share/crypto-policies/LEGACY/nss.txt
D: skip       100644  1 (   0,   0)   142 /usr/share/crypto-policies/LEGACY/krb5.txt
D: skip       100644  1 (   0,   0)   312 /usr/share/crypto-policies/LEGACY/java.txt
D: skip       100644  1 (   0,   0)   438 /usr/share/crypto-policies/LEGACY/gnutls28.txt
D: skip       100644  1 (   0,   0)   496 /usr/share/crypto-policies/LEGACY/gnutls.txt
D: skip       100644  1 (   0,   0)    64 /usr/share/crypto-policies/LEGACY/bind.txt
D: skip       100644  1 (   0,   0)   103 /usr/share/crypto-policies/FUTURE/openssl.txt
D: skip       100644  1 (   0,   0)   400 /usr/share/crypto-policies/FUTURE/nss.txt
D: skip       100644  1 (   0,   0)   111 /usr/share/crypto-policies/FUTURE/krb5.txt
D: skip       100644  1 (   0,   0)   386 /usr/share/crypto-policies/FUTURE/java.txt
D: skip       100644  1 (   0,   0)   420 /usr/share/crypto-policies/FUTURE/gnutls28.txt
D: skip       100644  1 (   0,   0)   465 /usr/share/crypto-policies/FUTURE/gnutls.txt
D: skip       100644  1 (   0,   0)   105 /usr/share/crypto-policies/FUTURE/bind.txt
D: skip       100644  1 (   0,   0)   160 /usr/share/crypto-policies/EMPTY/openssl.txt
D: skip       100644  1 (   0,   0)   145 /usr/share/crypto-policies/EMPTY/nss.txt
D: skip       100644  1 (   0,   0)    22 /usr/share/crypto-policies/EMPTY/krb5.txt
D: skip       100644  1 (   0,   0)   594 /usr/share/crypto-policies/EMPTY/java.txt
D: skip       100644  1 (   0,   0)    36 /usr/share/crypto-policies/EMPTY/gnutls28.txt
D: skip       100644  1 (   0,   0)    36 /usr/share/crypto-policies/EMPTY/gnutls.txt
D: skip       100644  1 (   0,   0)   123 /usr/share/crypto-policies/EMPTY/bind.txt
D: skip       100644  1 (   0,   0)    97 /usr/share/crypto-policies/DEFAULT/openssl.txt
D: skip       100644  1 (   0,   0)   418 /usr/share/crypto-policies/DEFAULT/nss.txt
D: skip       100644  1 (   0,   0)   125 /usr/share/crypto-policies/DEFAULT/krb5.txt
D: skip       100644  1 (   0,   0)   345 /usr/share/crypto-policies/DEFAULT/java.txt
D: skip       100644  1 (   0,   0)   420 /usr/share/crypto-policies/DEFAULT/gnutls28.txt
D: skip       100644  1 (   0,   0)   465 /usr/share/crypto-policies/DEFAULT/gnutls.txt
D: skip       100644  1 (   0,   0)    70 /usr/share/crypto-policies/DEFAULT/bind.txt
D: skip       040755  6 (   0,   0)  4096 /usr/share/crypto-policies
D: skip       100755  1 (   0,   0)  2362 /usr/bin/update-crypto-policies
D: skip       040755  2 (   0,   0)  4096 /etc/crypto-policies/local.d
D: skip       100644  1 (   0,   0)   606 /etc/crypto-policies/config
D: skip       120777  1 (   0,   0)    46 /etc/crypto-policies/back-ends/openssl.config
D: skip       120777  1 (   0,   0)    42 /etc/crypto-policies/back-ends/nss.config
D: skip       120777  1 (   0,   0)    43 /etc/crypto-policies/back-ends/krb5.config
D: skip       120777  1 (   0,   0)    43 /etc/crypto-policies/back-ends/java.config
D: skip       120777  1 (   0,   0)    47 /etc/crypto-policies/back-ends/gnutls28.config
D: skip       120777  1 (   0,   0)    45 /etc/crypto-policies/back-ends/gnutls.config
D: skip       120777  1 (   0,   0)    43 /etc/crypto-policies/back-ends/bind.config
D: skip       040755  2 (   0,   0)  4096 /etc/crypto-policies/back-ends
D: skip       040755  5 (   0,   0)  4096 /etc/crypto-policies
fdio:       2 writes,      759 total bytes in 0.000020 secs
D:   --- h#     186 crypto-policies-20160921-4.gitf3018dd.fc25.noarch
D: adding "crypto-policies" to Name index.
D: adding 46 entries to Basenames index.
D: adding "Unspecified" to Group index.
D: adding 8 entries to Requirename index.
D: adding 2 entries to Providename index.
D: adding 14 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "9f33e91a20fa838c5e85706350d2b9c7e5d4ac2c" to Sha1header index.
D: ========== +++ krb5-libs-1.14.4-7.fc25 ppc64le-linux 0x2
D:     erase: krb5-libs-1.14.4-7.fc25.ppc64le has 41 files
D: Plugin: calling hook psm_pre in selinux plugin
D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): scriptlet start
D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): execv(/bin/sh) pid 3867
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/bin/sh) 
++ rpm -q --qf '%{VERSION}\n' krb5-libs
++ sort -V
++ tr -d '\n'
++ head -n 1
+ old_ver=1.14.4
++ rpm -q --qf '%{RELEASE}\n' krb5-libs
++ sort -V
++ head -n 1
++ tr -d '\n'
+ old_rel=7.fc25
+ old_rel=7
+ [[ 1.14.4 < 1.15.1 ]]
+ grep -q 'includedir /etc/krb5.conf.d' /etc/krb5.conf
+ exit 0
D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): waitpid(3867) rc 3867 status 0
  Cleanup     : krb5-libs-1.14.4-7.fc25.ppc64le                                                                                                                                       613/706 
D: skip       040755  2 (   0,   0)  4096 /var/kerberos/krb5/user
D: skip       040755  3 (   0,   0)  4096 /var/kerberos/krb5
D: skip       040755  3 (   0,   0)  4096 /var/kerberos
D: skip       100644  1 (   0,   0) 13970 /usr/share/man/man5/krb5.conf.5.gz
D: skip       100644  1 (   0,   0)  1176 /usr/share/man/man5/k5login.5.gz
D: skip       100644  1 (   0,   0)  1207 /usr/share/man/man5/k5identity.5.gz
D: skip       100644  1 (   0,   0)    39 /usr/share/man/man5/.k5login.5.gz
D: skip       100644  1 (   0,   0)    42 /usr/share/man/man5/.k5identity.5.gz
D: skip       100644  1 (   0,   0)   410 /usr/share/locale/en_US/LC_MESSAGES/mit-krb5.mo
D: skip       100644  1 (   0,   0) 61301 /usr/share/licenses/krb5-libs/LICENSE
D: skip       040755  2 (   0,   0)  4096 /usr/share/licenses/krb5-libs
D: skip       100644  1 (   0,   0) 15120 /usr/share/doc/krb5-libs/README
D: skip       100644  1 (   0,   0) 61301 /usr/share/doc/krb5-libs/NOTICE
D: skip       040755  2 (   0,   0)  4096 /usr/share/doc/krb5-libs
D: skip       100755  1 (   0,   0)135408 /usr/lib64/libkrb5support.so.0.1
D: skip       120777  1 (   0,   0)    21 /usr/lib64/libkrb5support.so.0
D: skip       100755  1 (   0,   0)1199384 /usr/lib64/libkrb5.so.3.3
D: skip       120777  1 (   0,   0)    14 /usr/lib64/libkrb5.so.3
D: skip       100755  1 (   0,   0) 68576 /usr/lib64/libkrad.so.0.0
D: skip       120777  1 (   0,   0)    14 /usr/lib64/libkrad.so.0
D: skip       100755  1 (   0,   0)135224 /usr/lib64/libkdb5.so.8.0
D: skip       120777  1 (   0,   0)    14 /usr/lib64/libkdb5.so.8
D: skip       100755  1 (   0,   0)267296 /usr/lib64/libk5crypto.so.3.1
D: skip       120777  1 (   0,   0)    18 /usr/lib64/libk5crypto.so.3
D: skip       100755  1 (   0,   0)202424 /usr/lib64/libgssrpc.so.4.2
D: skip       120777  1 (   0,   0)    16 /usr/lib64/libgssrpc.so.4
D: skip       100755  1 (   0,   0)402960 /usr/lib64/libgssapi_krb5.so.2.2
D: skip       120777  1 (   0,   0)    21 /usr/lib64/libgssapi_krb5.so.2
D: skip       100755  1 (   0,   0) 68584 /usr/lib64/krb5/plugins/tls/k5tls.so
D: skip       040755  2 (   0,   0)  4096 /usr/lib64/krb5/plugins/tls
D: skip       040755  2 (   0,   0)  4096 /usr/lib64/krb5/plugins/preauth
D: skip       040755  2 (   0,   0)  4096 /usr/lib64/krb5/plugins/libkrb5
D: skip       040755  2 (   0,   0)  4096 /usr/lib64/krb5/plugins/kdb
D: skip       040755  2 (   0,   0)  4096 /usr/lib64/krb5/plugins/authdata
D: skip       040755  7 (   0,   0)  4096 /usr/lib64/krb5/plugins
D: skip       040755  3 (   0,   0)  4096 /usr/lib64/krb5
D: skip       120777  1 (   0,   0)    42 /etc/krb5.conf.d/crypto-policies
D: skip       040755  2 (   0,   0)  4096 /etc/krb5.conf.d
D: skip       100644  1 (   0,   0)   677 /etc/krb5.conf
D: skip       040755  2 (   0,   0)  4096 /etc/gss/mech.d
D: skip       040755  3 (   0,   0)  4096 /etc/gss
D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): scriptlet start
D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): execv(/sbin/ldconfig) pid 3879
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/sbin/ldconfig) 
D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): waitpid(3879) rc 3879 status 0

libdb-5.3.28-21.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d674be444

libdb-5.3.28-21.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e056b68bf

As this is supposed to be a *tracking* bug, can someone file a new bug for the outstanding ppc64le issue and mark it as fixing this bug? I'll edit the updates to *not* close this bug.

sorry, of course I meant "mark it as blocking this bug" not "mark it as fixing this bug".

*** Bug 1460147 has been marked as a duplicate of this bug. ***

*** Bug 1476493 has been marked as a duplicate of this bug. ***

This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.