Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1946975 - dnf installs old version of dependency even though --best is used
Summary: dnf installs old version of dependency even though --best is used
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 33
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Marek Blaha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-07 12:07 UTC by Lukáš Zachar
Modified: 2021-06-15 07:21 UTC (History)
9 users (show)

Fixed In Version: dnf-4.8.0-1.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-15 07:21:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Lukáš Zachar 2021-04-07 12:07:17 UTC
Description of problem:

Regardless of best value transaction for dnf install podman picks older `crun` version from 'fedora' instead of 'updates'.
Manual update of libcap beforehand changes selected version of crun.
For a basic user it is impossible to find out why.

Version-Release number of selected component (if applicable):
dnf-4.6.1-1.fc33

How reproducible:

Steps to Reproduce:
1. podman run --rm -ti fedora:33 bash
2. dnf install -y best podman
3. check crun version

Actual results:
0.15-5.fc33 (from fedora)

Expected results:
0.18-5.fc33 (from updates)

Additional info:

If I update `libcap` (to libcap-2.48-2.fc33) before executing `dnf install podman` I get expected (newer) crun in the transaction.
Image used is `registry.fedoraproject.org/fedora 33 9f2a56037643`

Issue was found during investigation what is happening in https://github.com/psss/tmt/issues/682

Comment 2 Marek Blaha 2021-04-07 14:33:33 UTC
I did some investigation and succeeded to create a minimal reproducer - see this draft PR on our ci-dnf-stack project: https://github.com/rpm-software-management/ci-dnf-stack/pull/982/files

This is required structure of packages to reproduce the issue:
package "top" (podman in your case) requires package "middle" (crun in your case) and the "middle" package requires "bottom" (libcap in your case).
Each of those dependencies comes in two versions (middle-0.15, middle-0.18, bottom-2.26, bottom-2.48) and the newer version of the "middle" package requires the newer version of the "bottom" package.
In case you have older "bottom-2.26" installed on system and you try to do "dnf install --best top", the older "middle-0.15" is installed instead of the best middle-0.18 version.

I didn't investigate the source code of dnf/libdnf/libsolv yet, so I'm not sure how complicated the fix for the issue would be. It is also possible that the issue is "not a bug" because libsolv is not supposed to apply the --best switch recursively to the package dependencies.

As a workaround you can either manually add crun package to the transaction (dnf install --best crun podman) or (as you suggested) run `dnf upgrade libcap` beforehand.

Comment 3 Martin Pitt 2021-04-07 16:02:48 UTC
> 2. dnf install -y best podman

I suppose that was meant to be --best.

Lukas, does tmt use `--best` for installing the FMF plan's dependencies? (See https://github.com/psss/tmt/issues/682 for my original issue)

I don't see that in the debug output:

            Run command 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 10023 -i /var/tmp/tmt/run-001/plans/all/provision/
default/id_rsa root.0.1 rpm -q --whatprovides "make" "cockpit-ws" "cockpit-podman" "cockpit-system" "libvirt-python3" "npm" "git" "rsy
nc" "python3" || dnf install -y "make" "cockpit-ws" "cockpit-podman" "cockpit-system" "libvirt-python3" "npm" "git" "rsync" "python3"'.

So it seems to happen without --best as well?

This is indeed highly irritating, and probably leads to all kinds of vulnerable or malfunctioning installations out there, if `dnf install foo` installs a package which is 7 months old (as in my case).

Comment 4 Marek Blaha 2021-04-08 06:37:11 UTC
Unfortunately the issue is reproducible with --best used. See the minimal reproducer in https://github.com/rpm-software-management/ci-dnf-stack/pull/982/files and comment#2. May be it is not clear from the PR itself so let me explain how the dnf behaves:

when command `dnf install --best top` is executed your expectations are:
- dnf installs the best candidate for the `top` package - top-3.0 (aka podman)
- dnf installs also the best candidates for its dependencies - `middle-0.18` (this is the crun package in your tmt issue)
- dnf upgrades the `bottom` package to fulfill the `middle` dependencies

But the transaction you actually see is this:


# dnf install bottom-2.26
# dnf install --best top

================================================================================
Package          Architecture     Version                 Repository      Size
================================================================================
Installing:
 top              noarch           3.0-1.fc29              best           5.8 k
Installing dependencies:
 middle           noarch           0.15-1.fc29             best           5.8 k
================================================================================

So the --best was applied only to the package actually being installed but not to its dependencies - the `middle` package is installed in older version. If we hadn't installed the `bottom` package beforehand, the newest version of the `middle` package would be installed.

Comment 5 Lukáš Zachar 2021-04-08 07:09:16 UTC
(In reply to Martin Pitt from comment #3)
> > 2. dnf install -y best podman
> 
> I suppose that was meant to be --best.
> 
> Lukas, does tmt use `--best` for installing the FMF plan's dependencies?
> (See https://github.com/psss/tmt/issues/682 for my original issue)

Not yet (no --best in tmt install) however as Marek mentioned it will not help right now either way. Let's deal with tmt in its own issue.


Marek - thank you for investigation and I still hope it can be fixed on dnf side.

Comment 6 Marek Blaha 2021-04-08 07:55:03 UTC
I'd like to first discuss it in the DNF team and then according the result either clarify the documentation of the --best option (i.e. wont fix) or fix dnf/libdnf or file an issue on libsolv.

Comment 7 Marek Blaha 2021-04-13 06:32:05 UTC
I'm sorry but I have a bad news for you. After consulting the DNF team and the libsolv I found the current behavior expected.

What's the most important - the `best` option (either the command line --best or the configuration option) affects *only* directly requested packages. I think this definitely deserves some clarification in the documentation which with the current wording suggests that each package in the transaction (including the dependencies) is guaranteed to be used in the latest version. A documentation PR will follow.

That said - in the case of `dnf install --best podman` only the latest version of the `podman` package is guaranteed to be installed and solver is not bound to fulfill its dependencies with the latest versions of other packages. On contrary - the solver prefers to fit the incoming packages to the already installed ones and avoids installation of new packages when possible. That's why it prefers to preserve the current version of libcap and rather goes with the matching version of the crun package.

So if there is a minimal version of crun needed for cockpit-podman you might condider requiring it.
The other possibility is to add crun to the command line arguments and then the newest version is guaranteed to be installed - `dnf install --best podman crun`

Comment 8 Martin Pitt 2021-04-13 07:12:17 UTC
I still don't know what the `--best` is doing here (i.e. why Lukas added it to the reproducer) -- tmt is *not* using that, nor does the reproducer in https://github.com/psss/tmt/issues/682#issuecomment-814778267 .

(In reply to Marek Blaha from comment #7)
> On contrary - the solver prefers to fit the incoming packages to
> the already installed ones and avoids installation of new packages when
> possible.

That makes sense. But in the reported case, crun was *not* installed already:

            Run command 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p 10023 -i /var/tmp/tmt/run-001/plans/all/provision/
default/id_rsa root.0.1 rpm -q --whatprovides "make" "cockpit-ws" "cockpit-podman" "cockpit-system" "libvirt-python3" "npm" "git" "rsy
nc" "python3" || dnf install -y "make" "cockpit-ws" "cockpit-podman" "cockpit-system" "libvirt-python3" "npm" "git" "rsync" "python3"'.
            environment: None
[...]
            out: Installing:
            out:  cockpit-podman              noarch 29-1.fc33                     updates 1.0 M
[...]
            out: Installing dependencies:
            out:  crun                        x86_64 0.15-5.fc33                   fedora  156 k

(and 0.15-5 is 7 months old, current is 0.18-5)

> That's why it prefers to preserve the current version of libcap
> and rather goes with the matching version of the crun package.

So this "prefers to preserve current version" applies to transitive dependencies also? That feels weird. It makes total sense for "dnf install foo" to be a no-op if any version of "foo" is already installed. But if "foo" is not installed, then IMHO it should always pick the latest version, even if that means to upgrade a dependency.

> The other possibility is to add crun to the command line arguments

That's what I have as a workaround now, but that does not scale to each test. The point of running a distro gating test is to run it against the latest available packages.

So I figure Lukas' proposal in https://github.com/psss/tmt/issues/682 to always `dnf update -y` the testbed for each test run makes more sense -- it will avoid that "keeps old transitive dependencies" trap, and generally seems useful for distro gating tests anyway.

Thanks Marek for the investigations! At least we now know why this happens, it's not just some random bug/regression in dnf.

Comment 9 Lukáš Zachar 2021-04-13 07:25:52 UTC
Thank you Marek. 

So if I want to install new package and I want to be sure that system stays up to date then I need to do it in two steps: 1. install package, 2. update whole system to be sure. 
Any chance to have 'pick latest possible dependencies' as an option?

I'm not sure how often / if packages are updated just for the sake for minimal dependency version when e.g. CVE is fixed in some dependency.
IMO this way `dnf install` can put vulnerable/buggy packages into the system as a side effect.

Comment 10 Martin Pitt 2021-04-13 07:46:15 UTC
Lukas: I'd *really* like this to be the other way around -- *first* update the system, and *then* install test dependencies. That avoids downloading/installing test deps twice, avoids upgrading pitfalls and further unnecessary slowdown. As far as I understand the root cause here, that should also be sufficient to circumvent the "pick old version" problem. Thanks!

Comment 11 Marek Blaha 2021-04-13 14:52:03 UTC
I agree that first update and then install is better order of actions.

Comment 12 Marek Blaha 2021-04-19 08:20:17 UTC
Clarification of the documentation regarding the dependencies: https://github.com/rpm-software-management/dnf/pull/1756

Comment 13 Fedora Update System 2021-06-15 07:21:32 UTC
FEDORA-2021-78c5f8c03d has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.