The error message comes from libdb, reassigning.

Thanks for reporting this. Looks like fallout from https://bugzilla.redhat.com/show_bug.cgi?id=1394862

Something seems to be trying to access the rpmdb after the rpm transaction goes through (before dnf's verify step) and finds itself unable to do so as glibc got updated.
I am unable to reproduce this on a clean F26 box so it is possibly something specific to your box that is causing issues. Your best bet is to find that 'something' and stop it from accessing the rpmdb when it is not safe to do.

Created attachment 1316338 [details]
Journal listing at time of problem

As per the journal, something unusual happened during the middle of that upgrade.

After today's large update, including kernel 4.12.8-300.fc26.x86_64, I again received the same error:

BDB1539 Build signature doesn't match environment
failed loading RPMDB

However, as far as I can tell, the upgrade was successful.
Looking at the journal log, I didn't notice anything unusual.

Just now I got:

...
Running scriptlet: kernel-core-4.11.11-300.fc26.x86_64                                            114/114 
BDB1539 Build signature doesn't match environment
failed loading RPMDB
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

This can very easily be reproduced in F27 container.

Just do:
1. docker run --rm -ti fedora:27 bash
2. dnf upgrade -y glibc

I'm also seeing this on both f26 and f27.

(In reply to Stanislav Laznicka from comment #6)
>This can very easily be reproduced in F27 container.

Thanks Standa for providing the reproducer. This can even be reproduced exactly the same way with f26 version of the image... Will look into this.

This is also affecting our CI infra which runs builds in Docker containers. To clarify, it looks like the scriptlet of 2.25-9 is causing the failure, which is present in the latest fedora:26 image.

Looks like a container specific issue.

The reason this is happening is because one of the hacks introduced via the fix for bug 1394862 depends on being able to get pid of the process that has rpmdb's environment locked for reading, which is done by parsing /proc/locks looking for the inode of the file of interest.

Unfortunately, for some reason (possibly a feature of OverlayFS?), in docker containers the inode of the file that is actually locked does not correspond to its entry in /proc/locks:

$ touch test; inode=`ls test -i | awk '{print $1}'`; flock test bash -c "cat /proc/locks" | grep $inode

prints nothing inside a docker container, while working correctly outside of it.

(In reply to Petr Kubat from comment #10)
> Looks like a container specific issue.

Occurred at the end of a DNF update on a normal (non-container) install for me.

>Occurred at the end of a DNF update on a normal (non-container) install for me.

Not saying this might not occur on a non-container system, but the causes for it would likely be different.

I have so far been unable to reproduce this outside containers on a clean system with only a glibc update so if you are running into the same failure on your system it might be something specific to your configuration that is the culprit.

FYI: Not using overlay2 as the storage driver for your docker containers can be used as a workaround for the time being.

I also had this happen on with a f26 docker container and on f27 with a regular upgrade. 

Changing the docker storage driver is not an option right now.

At least for the docker case, this should presumably be fixed once a new base container with the glibc update comes out?  Or is it related to updating glibc *at all*, not just this particular glibc update?

Also reproduced with a f25 container. I guess since the fix in bug 1394862 went into f25 as well.

(In reply to Colin Walters from comment #14)
>Or is it related to updating glibc *at all*, not just this particular glibc update?

Exactly. Since we do not know if a glibc update might change some internal structures libdb is using (and introduce another bug 1394862), libdb tries to rebuild its environment every time a new glibc rpm is installed and fails if it finds itself not being able to do this safely.

(In reply to Peter from comment #15)
> Also reproduced with a f25 container. I guess since the fix in bug 1394862
> went into f25 as well.

This issue is a combination of three things - one is the libdb bug you have linked, which tries to look on /proc/locks, and went into all supported Fedora version at that time, to make sure the upgrade process goes through.
Second is the recent F26 change to make overlay2 the default storage engine for docker and the last is overlay itself, since the inodes it uses for locking (and populating /proc/locks) do not correspond to the real inodes of the files being locked, which as I understand it now might be a feature of overlay (I have asked the maintainer for comment).

I think the best way of moving this forward is to assign the bug to dnf as it is in dnf where a fix makes the most sense.

To sum up for dnf maintainers what this bug is about:
The upstream fix for bug 1394862 (back-ported to our version of libdb) made libdb able to automatically rebuild its environment when trying to attach to it in case the environment was created by an older version of libdb, based on internal structure signature, version numbers, or pthread modify timestamp change.
Of course the rebuild would only be safe when there is no other process accessing the environment, which is why libdb has been modified to also try to lock its first region (__db.001) via a flock-style lock (since fcntl-style locks get released on close(2)).

Unfortunately, since dnf still accesses the rpmdb in its verification step, after the transaction goes through successfully, and libdb only looks at the modify timestamp (which has by that time changed due to the update), a specific hack was needed to make sure dnf does not fail in its verification (post-transaction) phase because of libdb thinking it is trying to access an old environment based on the timestamp (which is not true, as the process should still have the old pthreads library loaded at that point).
The hack itself tries to look-up the inode of the first region file in `/proc/locks` to get the PID of the process holding the lock (since that information cannot be obtained any other way for flock-style locks afaik) and compares it with the PID of the running process to make sure it is still safe to access the environment.

Ugly as this hack is, it has worked until now when it was found that things still break when trying to do an update in containerized fedora as in Fedora 26 there was a change of the default docker storage driver from devicemapper to overlay, which has the unfortunate feature of not having the inodes of locked files found in `/proc/locks` correspond to the inodes of real files. It is my understanding that this is by design, so that file locks are preserved even during copy-up of files from lower layers. The combination of these things makes the hack fail to make sure the process is safe to access the environment and bail immediately.

Additionally, there have been reports of similar failures even in non-containerized Fedora system (comment 11, comment 13) but I have not been able to reproduce these failures so far. Fixing the issue in dnf would, however, most likely result in getting rid of these failures as well.

Adding a "Regression" keyword as building containers that contain "dnf upgrade -y" was possible previously but is not anymore.

This error appears in Fedora 25 containers as well.

(In reply to Stanislav Laznicka from comment #6)
> This can very easily be reproduced in F27 container.
> 
> Just do:
> 1. docker run --rm -ti fedora:27 bash
> 2. dnf upgrade -y glibc

Do you have latest images?

(In reply to Stanislav Laznicka from comment #18)
> Adding a "Regression" keyword as building containers that contain "dnf
> upgrade -y" was possible previously but is not anymore.

I tested on f27 and following image works for me
registry.fedoraproject.org/fedora           27                  1e635fb211cd

I do not want to say it is not a bug. But there might be also another missing part for triggering it. And glic was updates :-) in my case

(In reply to Lukas Slebodnik from comment #20)
> I tested on f27 and following image works for me
> registry.fedoraproject.org/fedora           27                  1e635fb211cd
> 
> I do not want to say it is not a bug. But there might be also another
> missing part for triggering it. And glic was updates :-) in my case

I have been able to reproduce this on every clean-installed post f25 system I have tried, including rawhide.

Is your docker daemon using the default overlay as its storage driver? Can check with:
$ docker info | grep Storage

---

Panu: I have also checked that updating to newest rpm release first does not help.

(In reply to Lukas Slebodnik from comment #20)
> (In reply to Stanislav Laznicka from comment #6)
> > This can very easily be reproduced in F27 container.
> > 
> > Just do:
> > 1. docker run --rm -ti fedora:27 bash
> > 2. dnf upgrade -y glibc
> 
> Do you have latest images?
> 
Yes. I have the same image you do.

# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
registry.fedoraproject.org/fedora   27                  1e635fb211cd        13 days ago         233 MB

> (In reply to Stanislav Laznicka from comment #18)
> > Adding a "Regression" keyword as building containers that contain "dnf
> > upgrade -y" was possible previously but is not anymore.
> 
> I tested on f27 and following image works for me
> registry.fedoraproject.org/fedora           27                  1e635fb211cd

I don't know about f27, this happens on F26 for me. If it works in F27, it might be worth checking out for anyone fixing this.

> 
> I do not want to say it is not a bug. But there might be also another
> missing part for triggering it. And glic was updates :-) in my case

I think we can safely say it's a bug. I am not the only person who reproduced it should you skim through this BZ. It's also reproducible 100% of the time on F26.

(In reply to Petr Kubat from comment #21)
> (In reply to Lukas Slebodnik from comment #20)
> > I tested on f27 and following image works for me
> > registry.fedoraproject.org/fedora           27                  1e635fb211cd
> > 
> > I do not want to say it is not a bug. But there might be also another
> > missing part for triggering it. And glic was updates :-) in my case
> 
> I have been able to reproduce this on every clean-installed post f25 system
> I have tried, including rawhide.
> 
> Is your docker daemon using the default overlay as its storage driver? Can
> check with:
> $ docker info | grep Storage
> 

That is the case. I tested on two different machines and both of them have different storage driver :-)

[root@f27 ~]# docker info | grep Storage
Storage Driver: btrfs

[root@f25 ~]# docker info | grep Storage
Storage Driver: devicemapper

(In reply to Petr Kubat from comment #17)

> Ugly as this hack is, it has worked until now when it was found that things
> still break when trying to do an update in containerized fedora as in Fedora
> 26 there was a change of the default docker storage driver from devicemapper
> to overlay, which has the unfortunate feature of not having the inodes of
> locked files found in `/proc/locks` correspond to the inodes of real files.
> It is my understanding that this is by design,

One thing that might help is to force a copy up the first time we open the rpmdb for write.  Then we only lock the overlay file.

This is effectively what rpm-ostree does: https://github.com/projectatomic/rpm-ostree/blob/79086f8c3d3037828e81372439f3a68c49e6d049/src/libpriv/rpmostree-core.c#L3086
(rpm-ostree uses ostree, not overlayfs, but the idea is the same)

Something like this in pseudocode:

lock_and_copyup() {
 lock(/var/lib/rpm)
 if is_overlayfs(/var/lib/rpm) && !exists(/var/lib/rpm/.copyup) {
     for file in /var/lib/rpm {
       force_copyup (file)
     }
     touch(/var/lib/rpm/.copyup)
   }
 }
 unlock(/var/lib/rpm)
 lock(/var/lib/rpm)
}

(Worth noting that host systems managed via rpm-ostree are pretty much systematically immune to issues like this one due to the way we do offline updates/installs by default, and only optionally make them live after everything is complete - that plus the fact that the rpm database always appears read-only to everything except rpm-ostree (and we always regenerate it fresh) and there are no plugins etc.)

(In reply to Colin Walters from comment #24)
>One thing that might help is to force a copy up the first time we open the rpmdb for write.  Then we only lock the overlay file.

The file that is being locked by libdb is already present in the upper layer so I guess this will not work for overlay.

---

The maintainer for the overlay part of the kernel already sent me a patch (I have yet to test) so I guess at least the containerized part of this issue can be thought of as a bug in the kernel.

(In reply to Petr Kubat from comment #21)
> I have been able to reproduce this on every clean-installed post f25 system
> I have tried, including rawhide.

Maybe "clean-installed" has something to do with it (this laptop was a fresh install of Fedora 15, many moons ago), but I cannot reproduce the docker case at all. Which is really annoying since ... well, debugging without a reproducer is.

> Is your docker daemon using the default overlay as its storage driver? Can
> check with:
> $ docker info | grep Storage

[root@sopuli ~]# docker info | grep Storage
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Storage Driver: devicemapper
  WARNING: You're not using the default seccomp profile

Whatever that means. I'm pretty sure I haven't manually changed that but then this has been updated through eleven Fedora versions... Doing a fresh install of fedora 26 in libvirt gives something different (Storage Driver: overlay2), and there it is reproducable.

Yay, progress - this means I will be able to actually look into this. Thanks for the tip, wouldn't have guessed that in a million years.

BUT. That also sounds somehow awfully ominuous. Guess it's time to go read what these different storage options actually do and why does overlay2 screw up rpm so. Not that docker and rpm ever liked each-other too much.

(after hitting mid-air collision and seeing the last update: possible kernel bug, so sounds ominuous indeed)

(In reply to Panu Matilainen from comment #26)
> BUT. That also sounds somehow awfully ominuous. Guess it's time to go read
> what these different storage options actually do and why does overlay2 screw
> up rpm so. Not that docker and rpm ever liked each-other too much.

I have already touched up on that a bit in comment 16 and comment 17. My understanding is that overlayfs stores the overlay inodes of locked files into `/proc/locks` instead of inodes of real files located in upper/lower layers of the filesystem. I do not think this is documented anywhere apart from the kernel patch that introduced this (and I do hope this is the one):

https://patchwork.kernel.org/patch/9236917/

(In reply to Petr Kubat from comment #27)
> I have already touched up on that a bit in comment 16 and comment 17. 

Right, so you have, and pointed out the exact workaround in comment 12. My bad for missing those. 

> My understanding is that overlayfs stores the overlay inodes of locked files
> into `/proc/locks` instead of inodes of real files located in upper/lower
> layers of the filesystem. I do not think this is documented anywhere apart
> from the kernel patch that introduced this (and I do hope this is the one):
> 
> https://patchwork.kernel.org/patch/9236917/

Right. In the meanwhile, now that I have a reproducer...

The source of this particular error is actually libsolv, which uses its own code for reading the rpmdb. Which explains why the band-aid in rpm wasn't working for this case.

Created attachment 1323628 [details]
libsolv patch to fall back to DB_PRIVATE on DB_VERSION_MISMATCH

The attached patch does basically the same as rpm in this situation: it falls back to using DB_PRIVATE environment. It's not really a safe thing to do so it's band-aid rather than actual solution, but it lets the operation complete:

Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : glibc-common-2.26-7.fc27.x86_64                        1/8 
  Upgrading        : glibc-langpack-en-2.26-7.fc27.x86_64                   2/8 
  Running scriptlet: glibc-2.26-7.fc27.x86_64                               3/8 
  Upgrading        : glibc-2.26-7.fc27.x86_64                               3/8 
  Running scriptlet: glibc-2.26-7.fc27.x86_64                               3/8 
  Upgrading        : libcrypt-nss-2.26-7.fc27.x86_64                        4/8 
  Running scriptlet: libcrypt-nss-2.26-7.fc27.x86_64                        4/8 
  Cleanup          : libcrypt-nss-2.26-4.fc27.x86_64                        5/8 
  Running scriptlet: libcrypt-nss-2.26-4.fc27.x86_64                        5/8 
  Cleanup          : glibc-2.26-4.fc27.x86_64                               6/8 
  Running scriptlet: glibc-2.26-4.fc27.x86_64                               6/8 
  Cleanup          : glibc-langpack-en-2.26-4.fc27.x86_64                   7/8 
  Cleanup          : glibc-common-2.26-4.fc27.x86_64                        8/8 
BDB1539 Build signature doesn't match environment
  Verifying        : glibc-2.26-7.fc27.x86_64                               1/8 
  Verifying        : glibc-common-2.26-7.fc27.x86_64                        2/8 
  Verifying        : glibc-langpack-en-2.26-7.fc27.x86_64                   3/8 
  Verifying        : libcrypt-nss-2.26-7.fc27.x86_64                        4/8 
  Verifying        : glibc-common-2.26-4.fc27.x86_64                        5/8 
  Verifying        : glibc-langpack-en-2.26-4.fc27.x86_64                   6/8 
  Verifying        : libcrypt-nss-2.26-4.fc27.x86_64                        7/8 
  Verifying        : glibc-2.26-4.fc27.x86_64                               8/8 

Upgraded:
  glibc.x86_64 2.26-7.fc27                  glibc-common.x86_64 2.26-7.fc27     
  glibc-langpack-en.x86_64 2.26-7.fc27      libcrypt-nss.x86_64 2.26-7.fc27

I can confirm the patch works in in my containers, thanks!
Hopefully this will also fix the non-containerized use-case, but I have no way to check as I lack a reproducer.
For the people that were affected - are you able to test this?

Anyway I think libsolv/dnf should also log some info message that it is trying to fall back to DB_PRIVATE. Having an error message right at the start of the verification step might scare the people that pay attention to its output.

A scratch-build of libsolv with the above patch:
https://koji.fedoraproject.org/koji/taskinfo?taskID=21724101

And yeah it needs a diagnostic message, both in rpm itself and libsolv (dnf doesn't have the slightest clue of what's going on here). The problem is how on earth to explain the mess in a oner-liner message...

Looking past immediate workarounds:
Dnf level is blissfully ignorant of all this, it just wants to open the system repo (aka rpmdb) to do the verify walk. It could also use native rpm iterators instead of libsolv to avoid this to begin with. Anyway, when invoked at this point, libsolv rpmdb extension notices the rpmdb has changed since the solv file creation and wants to recreate it. And for that it opens up a new db environment to do it because it has no other choice, it's not aware of the upper layers and doesn't even use librpm code for it.

If dnf closed ALL database references before doing this, it should actually be ok, but this is skating on thin ice no matter what.

One possibility would be turning the picture upside down by implementing the libsolv solve-file creation as an rpm plugin which could use the already existing environment within rpm, but that'd be far more work of course.

>If dnf closed ALL database references before doing this, it should actually be ok, but this is skating on thin ice no matter what.

If dnf closed all its references on rpm's environment nothing would obstruct libsolv to rebuild the environment during a glibc upgrade, still with the old pthread library loaded in, and update the pthread timestamp to the newer one's, effectively re-introducing bug 1394862, since the timestamp is the only way we know we might not have a compatible environment already.

Ugh, indeed. Forgot about libsolv still using the old library in this case - the ice is even thinner than I remembered...

So actually dnf would need to run the verification step as new, separately exec'ed process after closing all db references for it to work. Seems cumbersome but OTOH it'd make the rpmdb accessible earlier for other players too. Another radical idea is to throw out the whole verification step, I never liked it in yum either and AFAIK dnf simply inherited it from there.

(In reply to Panu Matilainen from comment #33)
> So actually dnf would need to run the verification step as new, separately
> exec'ed process after closing all db references for it to work. Seems
> cumbersome but OTOH it'd make the rpmdb accessible earlier for other players
> too.

Making the rpmdb accessible just before the verification step might make the verification fail in case some other process gets write access to the rpmdb though. Well depending on what is checked during the verification step I guess.

Making libsolv able to use dnf's environment handle would be the best way to fix this but I can see how it would be a lot more work than any other solution so far.

I installed F27 fresh on my laptop and trying to create my own docker images again on this system but the images which tries to install gcc or glibc are failing to build the image with the "BDB1539 Build signature doesn't match environment Failed loading RPMDB." error. If there is any test packages to test for fixing this issue then I will be happy to test it.

I was able to run "dnf update" (I built this image [1]) in F26AH using devicemapper. With overlay2 it is NOT working.

[1] https://github.com/projectatomic/atomic-system-containers/blob/master/kubernetes-master/Dockerfile

*** Bug 1483854 has been marked as a duplicate of this bug. ***

I've refreshed all Fedora base images in both the Docker Hub[0] and in registry.fedoraproject.org, are these issues still persisting?

[0] - https://github.com/docker-library/official-images/pull/3459

It looks like Spyros broke this case (good detective work!). 

I'm leaning more towards the non POSIX compliant parts of overlayfs now. 

To be clear, this is a problem when using overlayfs as the docker storage backend and IS NOT a problem when using devicemapper.

This single command on F26AH with overlay2 yields:

```
# docker run -it --rm registry.fedoraproject.org/fedora:rawhide dnf update -y
...   
  Running scriptlet: glibc-common-2.26.90-14.fc28.x86_64                                                                                                                             112/112 
Failed to connect to bus: No such file or directory
BDB1539 Build signature doesn't match environment
failed loading RPMDB
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
```

With devicemapper as the backend and that same command we get a successful update.

(In reply to Spyros Trigazis from comment #36)
> I was able to run "dnf update" (I built this image [1]) in F26AH using
> devicemapper. With overlay2 it is NOT working.
> 
> [1]
> https://github.com/projectatomic/atomic-system-containers/blob/master/
> kubernetes-master/Dockerfile

Indeed, as stated in comment 12.
The person maintaining ovrerlayfs upstream already created a patch for the issue. Not sure how quickly it will get integrated though...

(In reply to Adam Miller from comment #38)
> I've refreshed all Fedora base images in both the Docker Hub[0] and in
> registry.fedoraproject.org, are these issues still persisting?
> 
> [0] - https://github.com/docker-library/official-images/pull/3459

Not persisting as in there is no glibc update right now that can break things via "dnf update" but running "dnf downgrade glibc" still runs into the same issues. I guess the libsolv band-aid is not yet in?

IIUC, it is similar issue we faced in the past. That is a file on lower filesystem is opened for READ. And later another process opens it for WRITE and file get copied up. But now reader and writer are looking at two different copies and any updates by writer are not visible to reader.

This is an issue with overlay filesystem (as of now) and people found workarounds for it. Some people modified their docker file to touch rpm files.

RUN touch /var/lib/rpm/*

IIRC, people also wrote a yum/dnf plugin which touch rpm files on startup. That makes sure files get copied up before any reader opens them and avoid this situation.

Jeremy Eder will know more details. He was very involved with that work.

Jeremy, IIRC, you and yum/dnf team already solved this issue. Did all the work you guys did, make in fedora as well?

The original BZ is https://bugzilla.redhat.com/show_bug.cgi?id=1213602

The solution was a yum ovl plugin, the mechanics of which are essentially what Colin has mentioned in c#24.

And the Fedora equivalent was https://bugzilla.redhat.com/show_bug.cgi?id=1216064

one more interesting tidbit:

I tried to `touch /var/lib/rpm/*` before running `dnf update -y` and I still get the `BDB1539 Build signature doesn't match environment` error. 

So 

1 docker run -it --rm registry.fedoraproject.org/fedora:rawhide
2 touch /var/lib/rpm/*
3 dnf update -y

Still see problem.

(In reply to Dusty Mabe from comment #45)
> one more interesting tidbit:
> 
> I tried to `touch /var/lib/rpm/*` before running `dnf update -y` and I still
> get the `BDB1539 Build signature doesn't match environment` error. 
> 
> So 
> 
> 1 docker run -it --rm registry.fedoraproject.org/fedora:rawhide
> 2 touch /var/lib/rpm/*
> 3 dnf update -y
> 
> Still see problem.

That's because it is a different issue... See comment 17 for the reason the error is coming up.

I am sorry, but I cannot reproduce the issue according to Comment 6, Comment 8, Comment 39, and Comment 45. Comment 10 also describe that the issue is caused by scriplet and performing scriplet is not job of dnf, but rpm. But in such an issue even RPM cannot solve the issue but change in scriplet definitely help.

To Comment 17:
I still don't see any problem in dnf, mostly due I am unable to reproduce the issue with the present environment. If I will have a reproducer with failing dnf, I can do more for you.

What I can do, I can refactor verification in dnf. Anyway now we depend on errors reported by RPM, and verification by comparison of requested rpm changes with new status of rpm-db is redundant (but in past it discovered some issues). At the present time we do not cache system repo to prevent problem with detection of minor changes in rpm-db. Unfortunately sum plugins also ask for its copy of installed rpm, therefore I cannot promise that there will be no other opener.

>If I will have a reproducer with failing dnf, I can do more for you.

Step 1: Make sure you are using overlay2 (or overlay) as your storage driver for Docker (default in F26)

$ docker info | grep Storage
Storage Driver: overlay

Step 2: As there is no glibc update available right now you have to trigger the issue by downgrading.

$ docker run --rm -i fedora:26 dnf -y downgrade glibc

Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Downgrading      : glibc-common-2.25-6.fc26.x86_64                                                                                                                                                           1/8 
  Downgrading      : glibc-langpack-en-2.25-6.fc26.x86_64                                                                                                                                                      2/8 
  Running scriptlet: glibc-2.25-6.fc26.x86_64                                                                                                                                                                  3/8 
  Downgrading      : glibc-2.25-6.fc26.x86_64                                                                                                                                                                  3/8 
  Running scriptlet: glibc-2.25-6.fc26.x86_64                                                                                                                                                                  3/8 
  Downgrading      : libcrypt-nss-2.25-6.fc26.x86_64                                                                                                                                                           4/8 
  Running scriptlet: libcrypt-nss-2.25-6.fc26.x86_64                                                                                                                                                           4/8 
  Erasing          : libcrypt-nss-2.25-10.fc26.x86_64                                                                                                                                                          5/8 
  Running scriptlet: libcrypt-nss-2.25-10.fc26.x86_64                                                                                                                                                          5/8 
  Erasing          : glibc-common-2.25-10.fc26.x86_64                                                                                                                                                          6/8 
  Erasing          : glibc-langpack-en-2.25-10.fc26.x86_64                                                                                                                                                     7/8 
  Erasing          : glibc-2.25-10.fc26.x86_64                                                                                                                                                                 8/8 
  Running scriptlet: glibc-2.25-10.fc26.x86_64                                                                                                                                                                 8/8 
BDB1539 Build signature doesn't match environment
failed loading RPMDB
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

---

This reproducer should work for you even with the current state of the environment. However, if you still have trouble reproducing, let me know and I will try to help out.

(In reply to Jaroslav Mracek from comment #47)
> I am sorry, but I cannot reproduce the issue according to Comment 6, Comment
> 8, Comment 39, and Comment 45. Comment 10 also describe that the issue is
> caused by scriplet and performing scriplet is not job of dnf, but rpm. But
> in such an issue even RPM cannot solve the issue but change in scriplet
> definitely help.

The reason why you can't reproduce it is in Comment 38. Just do what Petr Kubat tells you in Comment 48.

(In reply to Jaroslav Mracek from comment #47)
> What I can do, I can refactor verification in dnf. Anyway now we depend on
> errors reported by RPM, and verification by comparison of requested rpm
> changes with new status of rpm-db is redundant (but in past it discovered
> some issues). At the present time we do not cache system repo to prevent
> problem with detection of minor changes in rpm-db. Unfortunately sum plugins
> also ask for its copy of installed rpm, therefore I cannot promise that
> there will be no other opener.

Don't do anything hasty, this is a complex and terribly subtle mess and it's quite easy to make things actually worse (see eg comments from 29 to 34)

libsolv-0.6.29-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1c33746e7e

libsolv-0.6.29-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd488c853f

libsolv-0.6.29-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e3e612f6a1

We need to make sure this fix gets into RHEL7.5 release. since we hope to move the default for container runtimes to OVERLAY for RHEL7.

(In reply to Daniel Walsh from comment #54)
> We need to make sure this fix gets into RHEL7.5 release. since we hope to
> move the default for container runtimes to OVERLAY for RHEL7.

Um, RHEL-7 doesn't have glibc 2.25 so it's not affected by bug 1394862, so this particular thing is not happening there at all. Unless the glibc + libdb changes have been backported that is, and I'd be quite surprised if they were.

(In reply to Daniel Walsh from comment #54)
> We need to make sure this fix gets into RHEL7.5 release. since we hope to
> move the default for container runtimes to OVERLAY for RHEL7.

I don't think that will pose any issues. The libdb hack mentioned in comment 17 (being the reason behind dnf's failuresin this BZ) is not present in rhel7 as of now. And I do hope it stays way.

Excellent thanks.

libsolv-0.6.29-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1c33746e7e

libsolv-0.6.29-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e3e612f6a1

libsolv-0.6.29-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd488c853f

libsolv-0.6.29-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

1. start docker container fedora:rawhide , dnf update, works
2. start docker container fedora:26 , upgrade to libsolv-0.6.29-2.fc26 , dnf update works
3. start docker container fedora:26 , upgrade to libsolv-0.6.29-2.fc26 , downgrade glibc works
4. start docker container fedora:26 , dnf update shows the error message "BDB1539 Build signature doesn't match environment" but running dnf update says "Nothing to do." with no errors.

kde CI using f26 docker image with (libsolv-0.6.29-2.fc26 updated) is still seeing errors as well, for example:

https://build.kde.org/view/CI%20Management/job/Docker%20Generate%20FedoraQt5.8%20Image/54/console

Re-opening

(In reply to Spyros Trigazis from comment #62)
> 1. start docker container fedora:rawhide , dnf update, works
> 2. start docker container fedora:26 , upgrade to libsolv-0.6.29-2.fc26 , dnf
> update works
> 3. start docker container fedora:26 , upgrade to libsolv-0.6.29-2.fc26 ,
> downgrade glibc works
> 4. start docker container fedora:26 , dnf update shows the error message
> "BDB1539 Build signature doesn't match environment" but running dnf update
> says "Nothing to do." with no errors.
4. start docker container fedora:26 , dnf update shows the error message "BDB1539 Build signature doesn't match environment" but running dnf update *again* says "Nothing to do." with no errors. I guess it upgraded libsolv during the first run.

Folks, see comment #29: the updated libsolv doesn't make the error message go away, but it does let the update complete.

And BTW ignatenko, this is exactly why the diagnostic message would've been important...

Yep, libdb still fails even with the libsolv update but the verification step completes since libsolv tries to continue using a private (non-shared) libdb environment.

FYI: The overlayfs mantainer got back to me with the link to the kernel patch fixing the /proc/locks content:

https://lkml.org/lkml/2017/9/20/244

libsolv-0.6.29-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

libsolv-0.6.29-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Re-opening (again) per comment #63

Hi Rex,

I believe the problem was not that dnf/libsolv were not fixed, I think you just needed a rebased container image (which @maxamillion released yesterday) that had the newer builds. I.e., I think you were running the old broken ones instead of the fixed ones.

I had this same problem on Bodhi's CI system, and I just confirmed that the new images seem to resolve it:

https://github.com/fedora-infra/bodhi/pull/1915

You can see the build passed here:

https://ci.centos.org/job/bodhi-bodhi/614/

Not sure if it's related, but I noticed that packagekitd had been consuming 100% CPU for a couple of days, since this particular issue occurred on one of my systems. Killing packagekitd resolved the problem,

$ sudo dnf clean dbcache
error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch
error: cannot open Packages index using db5 -  (-30969)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed
$ sudo dnf update
error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch
error: cannot open Packages index using db5 -  (-30969)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

$ sudo killall -KILL packagekitd
$ sudo dnf clean dbcache
90 files removed
$ dnf update
...

FWIW, the packagekitd process was holding a lock in /proc/locks

The fixed image does not appear to be on dockerhub yet - I had to use
FROM       registry.fedoraproject.org/fedora:26
to get dnf working in images again.

For sure not available from dockerhub: https://gitlab.com/minishift/minishift/-/jobs/38406683
Any ETA? (or possibility to sync these automagically?)

+1 on needing this fixed for fedora:26, which is currently failing:

$ docker run --rm -ti fedora:26 bash
[root@469218cae304 /]# dnf upgrade -y glibc              
Fedora 26 - x86_64 - Updates                                                                                        573 kB/s |  16 MB     00:28    
Fedora 26 - x86_64                                                                                                  478 kB/s |  53 MB     01:54    
Last metadata expiration check: 0:00:21 ago on Thu Nov  2 16:30:44 2017.
Dependencies resolved.
====================================================================================================================================================
 Package                                  Arch                          Version                                Repository                      Size
====================================================================================================================================================
Upgrading:
 glibc                                    x86_64                        2.25-12.fc26                           updates                        3.4 M
 glibc-common                             x86_64                        2.25-12.fc26                           updates                        890 k
 glibc-langpack-en                        x86_64                        2.25-12.fc26                           updates                        291 k
 libcrypt-nss                             x86_64                        2.25-12.fc26                           updates                         54 k

Transaction Summary
====================================================================================================================================================
Upgrade  4 Packages

Total download size: 4.6 M
Downloading Packages:
(1/4): glibc-langpack-en-2.25-12.fc26.x86_64.rpm                                                                     58 kB/s | 291 kB     00:05    
(2/4): libcrypt-nss-2.25-12.fc26.x86_64.rpm                                                                          50 kB/s |  54 kB     00:01    
(3/4): glibc-common-2.25-12.fc26.x86_64.rpm                                                                         131 kB/s | 890 kB     00:06    
(4/4): glibc-2.25-12.fc26.x86_64.rpm                                                                                184 kB/s | 3.4 MB     00:19    
----------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                               240 kB/s | 4.6 MB     00:19     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                            1/1 
  Upgrading        : glibc-common-2.25-12.fc26.x86_64                                                                                           1/8 
  Upgrading        : glibc-langpack-en-2.25-12.fc26.x86_64                                                                                      2/8 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                                                                                                  3/8 
  Upgrading        : glibc-2.25-12.fc26.x86_64                                                                                                  3/8 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                                                                                                  3/8 
  Upgrading        : libcrypt-nss-2.25-12.fc26.x86_64                                                                                           4/8 
  Running scriptlet: libcrypt-nss-2.25-12.fc26.x86_64                                                                                           4/8 
  Cleanup          : libcrypt-nss-2.25-10.fc26.x86_64                                                                                           5/8 
  Running scriptlet: libcrypt-nss-2.25-10.fc26.x86_64                                                                                           5/8 
  Cleanup          : glibc-common-2.25-10.fc26.x86_64                                                                                           6/8 
  Cleanup          : glibc-langpack-en-2.25-10.fc26.x86_64                                                                                      7/8 
  Cleanup          : glibc-2.25-10.fc26.x86_64                                                                                                  8/8 
  Running scriptlet: glibc-2.25-10.fc26.x86_64                                                                                                  8/8 
BDB1539 Build signature doesn't match environment
failed loading RPMDB
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

(In reply to Mark Hirota from comment #74)
> +1 on needing this fixed for fedora:26, which is currently failing:
> 
> $ docker run --rm -ti fedora:26 bash
[...]
> BDB1539 Build signature doesn't match environment
> failed loading RPMDB

This is the symptoms when running with libsolv-0.6.29-1. With the above reproducer command I'm getting image sha256:64907a529db7e1b25d76c84928e2e1a82889c92a23f73b8f16385d05705ecea6 in which libsolv-0.6.29-2.fc26.x86_64 is present, and the upgrade succeeds despite the "build signature doesn't match" complaint:

[...]
  Running scriptlet: glibc-2.25-10.fc26.x86_64                              8/8 
BDB1539 Build signature doesn't match environment
  Verifying        : glibc-2.25-12.fc26.x86_64                              1/8 
  Verifying        : glibc-common-2.25-12.fc26.x86_64                       2/8 
  Verifying        : glibc-langpack-en-2.25-12.fc26.x86_64                  3/8 
  Verifying        : libcrypt-nss-2.25-12.fc26.x86_64                       4/8 
  Verifying        : glibc-common-2.25-10.fc26.x86_64                       5/8 
  Verifying        : glibc-langpack-en-2.25-10.fc26.x86_64                  6/8 
  Verifying        : libcrypt-nss-2.25-10.fc26.x86_64                       7/8 
  Verifying        : glibc-2.25-10.fc26.x86_64                              8/8 
[...]

So before complaining about not working, everybody PLEASE make sure you have the updated libsolv version. Oh and there's a related issue on rpm-side too (there's an update pending stable), but hitting that would manifest in a different way.

(In reply to Panu Matilainen from comment #75)

> So before complaining about not working, everybody PLEASE make sure you have
> the updated libsolv version. Oh and there's a related issue on rpm-side too
> (there's an update pending stable), but hitting that would manifest in a
> different way.

Hi, I'm not sure what you mean "make sure you have the updated libsolv version"?  I tried both in the host environment (which happens to be F26) and inside the docker image build (Dockerfile) -- but it still fails. I am using the base docker fedora:26 image from hub.docker.com (Please see my config below).

$ sudo dnf update -y libsolv
Last metadata expiration check: 0:00:00 ago on Fri 03 Nov 2017 07:07:45 AM PDT.
Dependencies resolved.
================================================================================
 Package          Arch            Version                Repository        Size
================================================================================
Upgrading:
 libsolv          x86_64          0.6.30-1.fc26          updates          365 k

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 365 k
Downloading Packages:
libsolv-0.6.30-1.fc26.x86_64.rpm                459 kB/s | 365 kB     00:00    
--------------------------------------------------------------------------------
Total                                           357 kB/s | 365 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : libsolv-0.6.30-1.fc26.x86_64                           1/2 
  Running scriptlet: libsolv-0.6.30-1.fc26.x86_64                           1/2 
  Cleanup          : libsolv-0.6.28-5.fc26.x86_64                           2/2 
  Running scriptlet: libsolv-0.6.28-5.fc26.x86_64                           2/2 
  Verifying        : libsolv-0.6.30-1.fc26.x86_64                           1/2 
  Verifying        : libsolv-0.6.28-5.fc26.x86_64                           2/2 

Upgraded:
  libsolv.x86_64 0.6.30-1.fc26                                                  

Complete!
$ docker build --tag=disk-monitor .
Sending build context to Docker daemon   5.12kB
Step 1/9 : FROM fedora:26
 ---> b0b140824a48
Step 2/9 : WORKDIR /mnt/work
 ---> Using cache
 ---> 8058d14d1ecb
Step 3/9 : RUN dnf -y install     binutils     curl     findutils     gzip     libsolv     tar     unzip     wget     which
 ---> Using cache
 ---> 05b2c3c22b1b
Step 4/9 : RUN dnf -y install     automake     cmake     gcc     gcc-c++     glibc     make
 ---> Running in 1d88bba9b423
Last metadata expiration check: 0:04:02 ago on Fri Nov  3 14:03:53 2017.
Package glibc-2.25-10.fc26.x86_64 is already installed, skipping.
Dependencies resolved.
================================================================================
 Package                    Arch       Version                Repository   Size
================================================================================
Installing:
 automake                   noarch     1.15-9.fc26            fedora      695 k
 cmake                      x86_64     3.9.1-1.fc26           updates     7.4 M
 gcc                        x86_64     7.2.1-2.fc26           updates      21 M
 gcc-c++                    x86_64     7.2.1-2.fc26           updates      11 M
 make                       x86_64     1:4.2.1-2.fc26         fedora      492 k
Upgrading:
 glibc                      x86_64     2.25-12.fc26           updates     3.4 M
 glibc-common               x86_64     2.25-12.fc26           updates     890 k
 glibc-langpack-en          x86_64     2.25-12.fc26           updates     291 k
 libcrypt-nss               x86_64     2.25-12.fc26           updates      54 k
 libgcc                     x86_64     7.2.1-2.fc26           updates      90 k
Installing dependencies:
 autoconf                   noarch     2.69-24.fc26           fedora      709 k
 cmake-data                 noarch     3.9.1-1.fc26           updates     1.2 M
 cmake-filesystem           x86_64     3.9.1-1.fc26           updates      35 k
 cpp                        x86_64     7.2.1-2.fc26           updates     9.2 M
 gc                         x86_64     7.6.0-2.fc26           fedora      108 k
 glibc-devel                x86_64     2.25-12.fc26           updates     966 k
 glibc-headers              x86_64     2.25-12.fc26           updates     516 k
 groff-base                 x86_64     1.22.3-9.fc26          fedora      1.0 M
 guile                      x86_64     5:2.0.14-1.fc26        fedora      3.5 M
 isl                        x86_64     0.16.1-1.fc26          fedora      821 k
 jsoncpp                    x86_64     1.8.1-3.fc26           updates      83 k
 kernel-headers             x86_64     4.13.9-200.fc26        updates     1.2 M
 libatomic_ops              x86_64     7.4.4-2.fc26           fedora       31 k
 libgomp                    x86_64     7.2.1-2.fc26           updates     197 k
 libmpc                     x86_64     1.0.2-6.fc26           fedora       55 k
 libstdc++                  x86_64     7.2.1-2.fc26           updates     462 k
 libstdc++-devel            x86_64     7.2.1-2.fc26           updates     1.9 M
 libtool-ltdl               x86_64     2.4.6-17.fc26          fedora       55 k
 libuv                      x86_64     1:1.11.0-1.fc26        fedora      112 k
 m4                         x86_64     1.4.18-3.fc26          fedora      217 k
 openssl                    x86_64     1:1.1.0f-7.fc26        updates     543 k
 perl                       x86_64     4:5.24.3-395.fc26      updates     6.1 M
 perl-Carp                  noarch     1.40-366.fc26          fedora       28 k
 perl-Data-Dumper           x86_64     2.161-2.fc26           fedora       54 k
 perl-Encode                x86_64     4:2.88-6.fc26          fedora      1.5 M
 perl-Errno                 x86_64     1.25-395.fc26          updates      69 k
 perl-Exporter              noarch     5.72-367.fc26          fedora       32 k
 perl-File-Path             noarch     2.12-367.fc26          fedora       34 k
 perl-File-Temp             noarch     0.230.400-2.fc26       fedora       61 k
 perl-Getopt-Long           noarch     2.49.1-2.fc26          fedora       61 k
 perl-HTTP-Tiny             noarch     0.070-2.fc26           fedora       56 k
 perl-IO                    x86_64     1.36-395.fc26          updates     134 k
 perl-MIME-Base64           x86_64     3.15-366.fc26          fedora       28 k
 perl-Net-SSLeay            x86_64     1.81-1.fc26            fedora      333 k
 perl-PathTools             x86_64     3.63-367.fc26          fedora       87 k
 perl-Pod-Escapes           noarch     1:1.07-366.fc26        fedora       19 k
 perl-Pod-Perldoc           noarch     3.28-1.fc26            fedora       86 k
 perl-Pod-Simple            noarch     1:3.35-2.fc26          fedora      211 k
 perl-Pod-Usage             noarch     4:1.69-2.fc26          fedora       33 k
 perl-Scalar-List-Utils     x86_64     3:1.48-1.fc26          updates      65 k
 perl-Socket                x86_64     4:2.024-2.fc26         fedora       55 k
 perl-Term-ANSIColor        noarch     4.06-2.fc26            fedora       44 k
 perl-Term-Cap              noarch     1.17-366.fc26          fedora       21 k
 perl-Text-ParseWords       noarch     3.30-366.fc26          fedora       16 k
 perl-Text-Tabs+Wrap        noarch     2013.0523-366.fc26     fedora       22 k
 perl-Thread-Queue          noarch     3.12-1.fc26            fedora       22 k
 perl-Time-Local            noarch     1:1.250-2.fc26         fedora       30 k
 perl-URI                   noarch     1.71-6.fc26            fedora      115 k
 perl-Unicode-Normalize     x86_64     1.25-366.fc26          fedora       79 k
 perl-constant              noarch     1.33-368.fc26          fedora       23 k
 perl-libnet                noarch     3.10-3.fc26            fedora      119 k
 perl-libs                  x86_64     4:5.24.3-395.fc26      updates     1.5 M
 perl-macros                x86_64     4:5.24.3-395.fc26      updates      65 k
 perl-parent                noarch     1:0.236-2.fc26         fedora       18 k
 perl-podlators             noarch     4.09-2.fc26            fedora      114 k
 perl-threads               x86_64     1:2.16-1.fc26          fedora       58 k
 perl-threads-shared        x86_64     1.57-1.fc26            fedora       45 k
 rhash                      x86_64     1.3.4-2.fc26           fedora      125 k
Installing weak dependencies:
 perl-IO-Socket-IP          noarch     0.39-1.fc26            fedora       45 k
 perl-IO-Socket-SSL         noarch     2.049-1.fc26           fedora      270 k
 perl-Mozilla-CA            noarch     20160104-4.fc26        fedora       13 k

Transaction Summary
================================================================================
Install  66 Packages
Upgrade   5 Packages

Total download size: 80 M
Downloading Packages:
(1/71): make-4.2.1-2.fc26.x86_64.rpm            683 kB/s | 492 kB     00:00    
(2/71): perl-Carp-1.40-366.fc26.noarch.rpm      371 kB/s |  28 kB     00:00    
(3/71): perl-Exporter-5.72-367.fc26.noarch.rpm  419 kB/s |  32 kB     00:00    
(4/71): autoconf-2.69-24.fc26.noarch.rpm        798 kB/s | 709 kB     00:00    
(5/71): automake-1.15-9.fc26.noarch.rpm         728 kB/s | 695 kB     00:00    
(6/71): perl-File-Path-2.12-367.fc26.noarch.rpm 401 kB/s |  34 kB     00:00    
(7/71): perl-Getopt-Long-2.49.1-2.fc26.noarch.r 443 kB/s |  61 kB     00:00    
(8/71): perl-PathTools-3.63-367.fc26.x86_64.rpm 915 kB/s |  87 kB     00:00    
(9/71): perl-Thread-Queue-3.12-1.fc26.noarch.rp 161 kB/s |  22 kB     00:00    
(10/71): perl-constant-1.33-368.fc26.noarch.rpm 248 kB/s |  23 kB     00:00    
(11/71): perl-threads-2.16-1.fc26.x86_64.rpm    450 kB/s |  58 kB     00:00    
(12/71): gc-7.6.0-2.fc26.x86_64.rpm             612 kB/s | 108 kB     00:00    
(13/71): perl-Data-Dumper-2.161-2.fc26.x86_64.r 381 kB/s |  54 kB     00:00    
(14/71): m4-1.4.18-3.fc26.x86_64.rpm            891 kB/s | 217 kB     00:00    
(15/71): perl-Text-ParseWords-3.30-366.fc26.noa 217 kB/s |  16 kB     00:00    
(16/71): perl-Pod-Usage-1.69-2.fc26.noarch.rpm  442 kB/s |  33 kB     00:00    
(17/71): perl-threads-shared-1.57-1.fc26.x86_64 602 kB/s |  45 kB     00:00    
(18/71): libatomic_ops-7.4.4-2.fc26.x86_64.rpm  236 kB/s |  31 kB     00:00    
(19/71): libtool-ltdl-2.4.6-17.fc26.x86_64.rpm  428 kB/s |  55 kB     00:00    
(20/71): perl-Pod-Perldoc-3.28-1.fc26.noarch.rp 1.1 MB/s |  86 kB     00:00    
(21/71): perl-podlators-4.09-2.fc26.noarch.rpm  785 kB/s | 114 kB     00:00    
(22/71): perl-File-Temp-0.230.400-2.fc26.noarch 821 kB/s |  61 kB     00:00    
(23/71): perl-HTTP-Tiny-0.070-2.fc26.noarch.rpm 729 kB/s |  56 kB     00:00    
(24/71): perl-Pod-Simple-3.35-2.fc26.noarch.rpm 940 kB/s | 211 kB     00:00    
(25/71): perl-parent-0.236-2.fc26.noarch.rpm    240 kB/s |  18 kB     00:00    
(26/71): perl-Term-ANSIColor-4.06-2.fc26.noarch 584 kB/s |  44 kB     00:00    
(27/71): groff-base-1.22.3-9.fc26.x86_64.rpm    1.4 MB/s | 1.0 MB     00:00    
(28/71): perl-Term-Cap-1.17-366.fc26.noarch.rpm 206 kB/s |  21 kB     00:00    
(29/71): perl-MIME-Base64-3.15-366.fc26.x86_64. 375 kB/s |  28 kB     00:00    
(30/71): perl-Socket-2.024-2.fc26.x86_64.rpm    718 kB/s |  55 kB     00:00    
(31/71): perl-Time-Local-1.250-2.fc26.noarch.rp 405 kB/s |  30 kB     00:00    
(32/71): perl-Pod-Escapes-1.07-366.fc26.noarch. 254 kB/s |  19 kB     00:00    
(33/71): perl-Text-Tabs+Wrap-2013.0523-366.fc26 305 kB/s |  22 kB     00:00    
(34/71): guile-2.0.14-1.fc26.x86_64.rpm         1.9 MB/s | 3.5 MB     00:01    
(35/71): cmake-filesystem-3.9.1-1.fc26.x86_64.r 653 kB/s |  35 kB     00:00    
(36/71): cmake-data-3.9.1-1.fc26.noarch.rpm     2.3 MB/s | 1.2 MB     00:00    
(37/71): libuv-1.11.0-1.fc26.x86_64.rpm         307 kB/s | 112 kB     00:00    
(38/71): rhash-1.3.4-2.fc26.x86_64.rpm          324 kB/s | 125 kB     00:00    
(39/71): cmake-3.9.1-1.fc26.x86_64.rpm          2.6 MB/s | 7.4 MB     00:02    
(40/71): isl-0.16.1-1.fc26.x86_64.rpm           1.1 MB/s | 821 kB     00:00    
(41/71): libmpc-1.0.2-6.fc26.x86_64.rpm         684 kB/s |  55 kB     00:00    
(42/71): cpp-7.2.1-2.fc26.x86_64.rpm            2.5 MB/s | 9.2 MB     00:03    
(43/71): glibc-devel-2.25-12.fc26.x86_64.rpm    2.1 MB/s | 966 kB     00:00    
(44/71): glibc-headers-2.25-12.fc26.x86_64.rpm  2.4 MB/s | 516 kB     00:00    
(45/71): perl-Encode-2.88-6.fc26.x86_64.rpm     1.3 MB/s | 1.5 MB     00:01    
(46/71): perl-Scalar-List-Utils-1.48-1.fc26.x86 1.7 MB/s |  65 kB     00:00    
(47/71): gcc-7.2.1-2.fc26.x86_64.rpm            3.5 MB/s |  21 MB     00:06    
(48/71): perl-Errno-1.25-395.fc26.x86_64.rpm    1.2 MB/s |  69 kB     00:00    
(49/71): perl-libs-5.24.3-395.fc26.x86_64.rpm   3.0 MB/s | 1.5 MB     00:00    
(50/71): perl-Unicode-Normalize-1.25-366.fc26.x 546 kB/s |  79 kB     00:00    
(51/71): perl-IO-1.36-395.fc26.x86_64.rpm       2.2 MB/s | 134 kB     00:00    
(52/71): libstdc++-7.2.1-2.fc26.x86_64.rpm      1.7 MB/s | 462 kB     00:00    
(53/71): perl-macros-5.24.3-395.fc26.x86_64.rpm 1.5 MB/s |  65 kB     00:00    
(54/71): gcc-c++-7.2.1-2.fc26.x86_64.rpm        2.8 MB/s |  11 MB     00:03    
(55/71): kernel-headers-4.13.9-200.fc26.x86_64. 2.4 MB/s | 1.2 MB     00:00    
(56/71): libgomp-7.2.1-2.fc26.x86_64.rpm        2.6 MB/s | 197 kB     00:00    
(57/71): jsoncpp-1.8.1-3.fc26.x86_64.rpm        2.1 MB/s |  83 kB     00:00    
(58/71): libstdc++-devel-7.2.1-2.fc26.x86_64.rp 2.8 MB/s | 1.9 MB     00:00    
(59/71): perl-Mozilla-CA-20160104-4.fc26.noarch  59 kB/s |  13 kB     00:00    
(60/71): perl-IO-Socket-IP-0.39-1.fc26.noarch.r 120 kB/s |  45 kB     00:00    
(61/71): perl-IO-Socket-SSL-2.049-1.fc26.noarch 723 kB/s | 270 kB     00:00    
(62/71): openssl-1.1.0f-7.fc26.x86_64.rpm       2.9 MB/s | 543 kB     00:00    
(63/71): perl-Net-SSLeay-1.81-1.fc26.x86_64.rpm 1.2 MB/s | 333 kB     00:00    
(64/71): perl-URI-1.71-6.fc26.noarch.rpm        724 kB/s | 115 kB     00:00    
(65/71): perl-5.24.3-395.fc26.x86_64.rpm        2.8 MB/s | 6.1 MB     00:02    
(66/71): libgcc-7.2.1-2.fc26.x86_64.rpm         1.5 MB/s |  90 kB     00:00    
(67/71): perl-libnet-3.10-3.fc26.noarch.rpm     1.2 MB/s | 119 kB     00:00    
(68/71): glibc-langpack-en-2.25-12.fc26.x86_64. 2.6 MB/s | 291 kB     00:00    
(69/71): libcrypt-nss-2.25-12.fc26.x86_64.rpm   2.0 MB/s |  54 kB     00:00    
(70/71): glibc-common-2.25-12.fc26.x86_64.rpm   1.5 MB/s | 890 kB     00:00    
(71/71): glibc-2.25-12.fc26.x86_64.rpm          3.1 MB/s | 3.4 MB     00:01    
--------------------------------------------------------------------------------
Total                                           6.0 MB/s |  80 MB     00:13     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : glibc-common-2.25-12.fc26.x86_64                      1/76 
  Upgrading        : glibc-langpack-en-2.25-12.fc26.x86_64                 2/76 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                             3/76 
  Upgrading        : glibc-2.25-12.fc26.x86_64                             3/76 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                             3/76 
  Upgrading        : libgcc-7.2.1-2.fc26.x86_64                            4/76 
  Running scriptlet: libgcc-7.2.1-2.fc26.x86_64                            4/76 
  Installing       : libstdc++-7.2.1-2.fc26.x86_64                         5/76 
  Running scriptlet: libstdc++-7.2.1-2.fc26.x86_64                         5/76 
  Upgrading        : libcrypt-nss-2.25-12.fc26.x86_64                      6/76 
  Running scriptlet: libcrypt-nss-2.25-12.fc26.x86_64                      6/76 
  Installing       : perl-Exporter-5.72-367.fc26.noarch                    7/76 
  Installing       : perl-libs-4:5.24.3-395.fc26.x86_64                    8/76 
  Running scriptlet: perl-libs-4:5.24.3-395.fc26.x86_64                    8/76 
  Installing       : perl-Carp-1.40-366.fc26.noarch                        9/76 
  Installing       : perl-Scalar-List-Utils-3:1.48-1.fc26.x86_64          10/76 
  Installing       : perl-Text-ParseWords-3.30-366.fc26.noarch            11/76 
  Installing       : libmpc-1.0.2-6.fc26.x86_64                           12/76 
  Running scriptlet: libmpc-1.0.2-6.fc26.x86_64                           12/76 
  Installing       : cmake-filesystem-3.9.1-1.fc26.x86_64                 13/76 
  Installing       : cpp-7.2.1-2.fc26.x86_64                              14/76 
  Running scriptlet: cpp-7.2.1-2.fc26.x86_64                              14/76 
  Installing       : perl-Term-ANSIColor-4.06-2.fc26.noarch               15/76 
  Installing       : perl-macros-4:5.24.3-395.fc26.x86_64                 16/76 
  Installing       : perl-Socket-4:2.024-2.fc26.x86_64                    17/76 
  Installing       : perl-Errno-1.25-395.fc26.x86_64                      18/76 
  Installing       : perl-File-Path-2.12-367.fc26.noarch                  19/76 
  Installing       : perl-PathTools-3.63-367.fc26.x86_64                  20/76 
  Installing       : perl-constant-1.33-368.fc26.noarch                   21/76 
  Installing       : perl-threads-shared-1.57-1.fc26.x86_64               22/76 
  Installing       : perl-threads-1:2.16-1.fc26.x86_64                    23/76 
  Installing       : perl-parent-1:0.236-2.fc26.noarch                    24/76 
  Installing       : perl-Text-Tabs+Wrap-2013.0523-366.fc26.noarch        25/76 
  Installing       : perl-Unicode-Normalize-1.25-366.fc26.x86_64          26/76 
  Installing       : perl-IO-1.36-395.fc26.x86_64                         27/76 
  Installing       : perl-4:5.24.3-395.fc26.x86_64                        28/76 
  Installing       : perl-MIME-Base64-3.15-366.fc26.x86_64                29/76 
  Installing       : perl-Data-Dumper-2.161-2.fc26.x86_64                 30/76 
  Installing       : perl-IO-Socket-IP-0.39-1.fc26.noarch                 31/76 
  Installing       : perl-Time-Local-1:1.250-2.fc26.noarch                32/76 
  Installing       : perl-HTTP-Tiny-0.070-2.fc26.noarch                   33/76 
  Installing       : perl-libnet-3.10-3.fc26.noarch                       34/76 
  Installing       : perl-Net-SSLeay-1.81-1.fc26.x86_64                   35/76 
  Installing       : perl-File-Temp-0.230.400-2.fc26.noarch               36/76 
  Installing       : perl-Term-Cap-1.17-366.fc26.noarch                   37/76 
  Installing       : perl-Pod-Escapes-1:1.07-366.fc26.noarch              38/76 
  Installing       : perl-Thread-Queue-3.12-1.fc26.noarch                 39/76 
  Installing       : groff-base-1.22.3-9.fc26.x86_64                      40/76 
  Installing       : perl-Pod-Simple-1:3.35-2.fc26.noarch                 41/76 
  Installing       : perl-Encode-4:2.88-6.fc26.x86_64                     42/76 
  Installing       : perl-Getopt-Long-2.49.1-2.fc26.noarch                43/76 
  Installing       : perl-podlators-4.09-2.fc26.noarch                    44/76 
  Installing       : perl-Pod-Perldoc-3.28-1.fc26.noarch                  45/76 
  Installing       : perl-Pod-Usage-4:1.69-2.fc26.noarch                  46/76 
  Installing       : perl-URI-1.71-6.fc26.noarch                          47/76 
  Installing       : libstdc++-devel-7.2.1-2.fc26.x86_64                  48/76 
  Installing       : jsoncpp-1.8.1-3.fc26.x86_64                          49/76 
  Running scriptlet: jsoncpp-1.8.1-3.fc26.x86_64                          49/76 
  Installing       : m4-1.4.18-3.fc26.x86_64                              50/76 
  Running scriptlet: m4-1.4.18-3.fc26.x86_64                              50/76 
  Installing       : autoconf-2.69-24.fc26.noarch                         51/76 
  Running scriptlet: autoconf-2.69-24.fc26.noarch                         51/76 
install-info: No such file or directory for /usr/share/info/autoconf.info
  Installing       : libatomic_ops-7.4.4-2.fc26.x86_64                    52/76 
  Running scriptlet: libatomic_ops-7.4.4-2.fc26.x86_64                    52/76 
  Installing       : gc-7.6.0-2.fc26.x86_64                               53/76 
  Running scriptlet: gc-7.6.0-2.fc26.x86_64                               53/76 
  Installing       : libtool-ltdl-2.4.6-17.fc26.x86_64                    54/76 
  Running scriptlet: libtool-ltdl-2.4.6-17.fc26.x86_64                    54/76 
  Installing       : guile-5:2.0.14-1.fc26.x86_64                         55/76 
  Running scriptlet: guile-5:2.0.14-1.fc26.x86_64                         55/76 
  Installing       : make-1:4.2.1-2.fc26.x86_64                           56/76 
  Running scriptlet: make-1:4.2.1-2.fc26.x86_64                           56/76 
  Installing       : openssl-1:1.1.0f-7.fc26.x86_64                       57/76 
  Installing       : libuv-1:1.11.0-1.fc26.x86_64                         58/76 
  Running scriptlet: libuv-1:1.11.0-1.fc26.x86_64                         58/76 
  Installing       : rhash-1.3.4-2.fc26.x86_64                            59/76 
  Running scriptlet: rhash-1.3.4-2.fc26.x86_64                            59/76 
  Installing       : cmake-data-3.9.1-1.fc26.noarch                       60/76 
  Installing       : cmake-3.9.1-1.fc26.x86_64                            61/76 
  Installing       : isl-0.16.1-1.fc26.x86_64                             62/76 
  Running scriptlet: isl-0.16.1-1.fc26.x86_64                             62/76 
  Installing       : libgomp-7.2.1-2.fc26.x86_64                          63/76 
  Running scriptlet: libgomp-7.2.1-2.fc26.x86_64                          63/76 
  Installing       : kernel-headers-4.13.9-200.fc26.x86_64                64/76 
  Running scriptlet: glibc-headers-2.25-12.fc26.x86_64                    65/76 
  Installing       : glibc-headers-2.25-12.fc26.x86_64                    65/76 
  Installing       : glibc-devel-2.25-12.fc26.x86_64                      66/76 
  Running scriptlet: glibc-devel-2.25-12.fc26.x86_64                      66/76 
  Installing       : gcc-7.2.1-2.fc26.x86_64                              67/76 
  Running scriptlet: gcc-7.2.1-2.fc26.x86_64                              67/76 
  Installing       : gcc-c++-7.2.1-2.fc26.x86_64                          68/76 
  Installing       : perl-IO-Socket-SSL-2.049-1.fc26.noarch               69/76 
  Installing       : automake-1.15-9.fc26.noarch                          70/76 
  Running scriptlet: automake-1.15-9.fc26.noarch                          70/76 
install-info: No such file or directory for /usr/share/info/automake.info.gz
  Installing       : perl-Mozilla-CA-20160104-4.fc26.noarch               71/76 
  Cleanup          : libcrypt-nss-2.25-10.fc26.x86_64                     72/76 
  Running scriptlet: libcrypt-nss-2.25-10.fc26.x86_64                     72/76 
  Cleanup          : glibc-common-2.25-10.fc26.x86_64                     73/76 
  Cleanup          : glibc-langpack-en-2.25-10.fc26.x86_64                74/76 
  Cleanup          : glibc-2.25-10.fc26.x86_64                            75/76 
  Running scriptlet: glibc-2.25-10.fc26.x86_64                            75/76 
  Cleanup          : libgcc-7.1.1-3.fc26.x86_64                           76/76 
  Running scriptlet: libgcc-7.1.1-3.fc26.x86_64                           76/76 
  Running scriptlet: guile-5:2.0.14-1.fc26.x86_64                         76/76BDB1539 Build signature doesn't match environment
failed loading RPMDB
 
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
The command '/bin/sh -c dnf -y install     automake     cmake     gcc     gcc-c++     glibc     make' returned a non-zero code: 1
$ docker image inspect fedora:26
[
    {
        "Id": "sha256:b0b140824a486ccc0f7968f3c6ceb6982b4b77e82ef8b4faaf2806049fc266df",
        "RepoTags": [
            "fedora:26"
        ],
        "RepoDigests": [
            "fedora@sha256:613830d906f8897c05fd737aea8c7c00e4ecb5a518eedf71958c2c110532957c"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2017-09-15T23:28:22.816153183Z",
        "Container": "325c38d9922c257e0ccfec8503e61a156fa6ddbba9633f33b3e8acdf19d1307a",
        "ContainerConfig": {
            "Hostname": "325c38d9922c",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f26container",
                "FGC=f26",
                "FBR=f26"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ADD file:6fef8af91ffd240db5c0e394621757fc453e5f7312e33f04d98d04fef009b078 in / "
            ],
            "Image": "sha256:b01a4b5daeddf89b88f00abe8271e0e10ff1d520d227f91ce9af5d10dc94aaf1",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "17.06.2-ce",
        "Author": "[Adam Miller <maxamillion>] [Patrick Uiterwijk <patrick>]",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DISTTAG=f26container",
                "FGC=f26",
                "FBR=f26"
            ],
            "Cmd": null,
            "Image": "sha256:b01a4b5daeddf89b88f00abe8271e0e10ff1d520d227f91ce9af5d10dc94aaf1",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 231079876,
        "VirtualSize": 231079876,
        "GraphDriver": {
            "Data": {
                "MergedDir": "/var/lib/docker/overlay2/ab00f10bc15af977ffb201fd3ae1d01d41ce2f5b9aad91970c9e32ab392b2455/merged",
                "UpperDir": "/var/lib/docker/overlay2/ab00f10bc15af977ffb201fd3ae1d01d41ce2f5b9aad91970c9e32ab392b2455/diff",
                "WorkDir": "/var/lib/docker/overlay2/ab00f10bc15af977ffb201fd3ae1d01d41ce2f5b9aad91970c9e32ab392b2455/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:72e455bdda27d1fad7d0b6e4e2b7b10893e90576a5008e12af9bdaea0f926b1d"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]
$ uname -a
Linux linuxbox 4.13.9-200.fc26.x86_64 #1 SMP Mon Oct 23 13:52:45 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 17.11.0-ce-rc2, build d7062e5

fedora:26 at docker hub is out of date with respect to the one from fedoraproject
```
registry.fedoraproject.org/fedora   26    ba857ff07853        2 weeks ago         231MB
fedora                              26    b0b140824a48        6 weeks ago         231MB
```

Will dockerhub's     "Author": "[Adam Miller <maxamillion>] [Patrick Uiterwijk <patrick>]",
please update it?

(In reply to Mark Hirota from comment #76)
> (In reply to Panu Matilainen from comment #75)
> 
> > So before complaining about not working, everybody PLEASE make sure you have
> > the updated libsolv version. Oh and there's a related issue on rpm-side too
> > (there's an update pending stable), but hitting that would manifest in a
> > different way.
> 
> Hi, I'm not sure what you mean "make sure you have the updated libsolv
> version"?  I tried both in the host environment (which happens to be F26)
> and inside the docker image build (Dockerfile) -- but it still fails. I am
> using the base docker fedora:26 image from hub.docker.com (Please see my
> config below).

Inside the docker. As in:

[root@sopuli ~]# docker run --rm -ti fedora:26 bash
[root@34c95cb831ca /]# rpm -q libsolv
libsolv-0.6.29-2.fc26.x86_64

If libsolv is older than that, you need to first update libsolv in a separate transaction, ie (inside docker still):

# dnf update libsolv
[...]
# dnf update

With that, you'll still get "BDB1539 Build signature doesn't match environment" but not "failed loading RPMDB" and the operation actually succeeds. There's no indication of the libsolv version inside docker so impossible to say anything except that it's behaving exactly like the version without the workaround.

But there really should be a diagnostic message for the workaround, sigh...

(In reply to Panu Matilainen from comment #78)
> There's no indication of the libsolv version inside docker
> so impossible to say anything except that it's behaving exactly like
> the version without the workaround.

That was supposed to say:

There's no indication of the libsolv version inside docker in the log you included in comment #76 [...]

Created attachment 1348517 [details]
libsolv patch to fall back to DB_PRIVATE on DB_VERSION_MISMATCH

Updated version of the patch, now including a diagnostic message + a better explanation of what the patch is about.

libsolv-0.6.30-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d9767cf8a

libsolv-0.6.30-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f8c708fa8d

libsolv-0.6.30-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f8c708fa8d

libsolv-0.6.30-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9d9767cf8a

I have requested the container to be rebuilt here: https://pagure.io/releng/issue/7125

libsolv-0.6.30-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

libsolv-0.6.30-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.