Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1636239
Summary: | get_best_query().filter(latest=True) is returning incorrect results | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Brian Lane <bcl> | ||||
Component: | dnf | Assignee: | rpm-software-management | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 29 | CC: | awilliam, bcl, chad.schroeder, dmach, dustymabe, extras-qa, fzatlouk, jmracek, mattdm, mblaha, mhatina, packaging-team-maint, pbrobinson, robatino, rpm-software-management, vmukhame, walters | ||||
Target Milestone: | --- | Keywords: | Triaged | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Linux | ||||||
Whiteboard: | PrioritizedBug, RejectedBlocker | ||||||
Fixed In Version: | dnf-4.0.4 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | 1548586 | Environment: | |||||
Last Closed: | 2018-11-22 17:29:18 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 1548586 | ||||||
Bug Blocks: | 1269538 | ||||||
Attachments: |
|
Description
Brian Lane
2018-10-04 19:32:43 UTC
This appears to STILL be a problem with the versions currently in Fedora-29 dnf-3.6.1-1.fc29.noarch dnf-data-3.6.1-1.fc29.noarch dnf-plugins-core-3.0.4-1.fc29.noarch dnf-yum-4.0.3.6.1-1.fc29.noarch libdnf-0.20.0-1.fc29.x86_64 python3-dnf-3.6.1-1.fc29.noarch python3-dnf-plugins-core-3.0.4-1.fc29.noarch python3-libdnf-0.20.0-1.fc29.x86_64 This prevents lorax-composer from working: [root@composer-f29 lorax]# composer blueprints depsolve example-development 2018-10-04 15:35:14,929: example-development: There was a problem depsolving [('cmake', '*'), ('curl', '*'), ('file', '*'), ('gcc', '*'), ('gcc-c++', '*'), ('gdb', '*'), ('git', '*'), ('glibc-devel', '*'), ('gnupg2', '*'), ('libcurl-devel', '*'), ('make', '*'), ('openssl-devel', '*'), ('sqlite', '*'), ('sqlite-devel', '*'), ('sudo', '*'), ('tar', '*'), ('xz', '*'), ('xz-devel', '*'), ('zlib-devel', '*')]: Problem 1: cannot install both curl-7.61.1-1.fc29.x86_64 and curl-7.61.1-1.fc29.x86_64 - conflicting requests Problem 2: cannot install both gdb-8.2-2.fc29.x86_64 and gdb-8.2-2.fc29.x86_64 - conflicting requests Problem 3: cannot install both git-2.19.0-1.fc29.x86_64 and git-2.19.0-1.fc29.x86_64 - conflicting requests Problem 4: cannot install both glibc-devel-2.28-9.fc29.i686 and glibc-devel-2.28-9.fc29.i686 - conflicting requests Problem 5: cannot install both glibc-devel-2.28-9.fc29.x86_64 and glibc-devel-2.28-9.fc29.x86_64 - conflicting requests Problem 6: cannot install both libcurl-devel-7.61.1-1.fc29.i686 and libcurl-devel-7.61.1-1.fc29.i686 - conflicting requests Problem 7: cannot install both libcurl-devel-7.61.1-1.fc29.x86_64 and libcurl-devel-7.61.1-1.fc29.x86_64 - conflicting requests Problem 8: cannot install both openssl-devel-1:1.1.1-3.fc29.i686 and openssl-devel-1:1.1.1-3.fc29.i686 - conflicting requests Problem 9: cannot install both openssl-devel-1:1.1.1-3.fc29.x86_64 and openssl-devel-1:1.1.1-3.fc29.x86_64 - conflicting requests blueprint: example-development v0.0.1 Proposed as a Blocker for 29-final by Fedora user bcl using the blocker tracking app because: Prevents lorax-composer from working. Is lorax-composer on any of Fedora's critical paths? It's not, AFAIK? Created attachment 1490729 [details]
Test script
Here's a test script to reproduce the problem. Run this on a f29 VM and observe the output:
[root@composer-f29 ~]# ./dnf-fail-bash.py
bash wants to install: [<hawkey.Package object id 2787, bash-4.4.23-4.fc29.x86_64, updates-testing>, <hawkey.Package object id 12437, bash-4.4.23-4.fc29.x86_64, fedora>]
bash-4.4.* wants to install: [<hawkey.Package object id 2787, bash-4.4.23-4.fc29.x86_64, updates-testing>, <hawkey.Package object id 12437, bash-4.4.23-4.fc29.x86_64, fedora>]
It appears to have the same package in 2 different repos, but it is returning both of them instead of just one.
(and to answer adam's question, not lorax-composer isn't in the critical path, but lorax is, and so are other things that depend on dnf. I don't think this problem is limited to lorax-composer, it's just what's hitting it at the moment).
Please can you point me to the code in lorax or other tool where query.latest function is used and where you fill the goal or how? We can also make a direct call. I can help you to fix the issue, just let us to help you. https://github.com/weldr/lorax/blob/master/src/pylorax/api/projects.py#L215 I've added the max( ... or [None]) to work around the problem for now. The correct solution would be as I mentioned above: ``` query = dnf.subject.Subject(name).get_best_query(dbo.sack).filter(version__glob=version, latest=True) if not query: install_errors.append(("%s-%s" % (name, version), "No match")) continue sltr=dnf.selector.Selector() sltr = slts.set(pkg=query) dbo.goal.install(select=sltr, optional=(not strict)) ``` From my point of view this is really not a bug and DNF returns everything correctly. If you will provide additional information about character of "name" variable (if it is name of package, provide or NEVRA, or partial nevra type), I can provide additional performance improvement. Anyway usage version__glob=version is not perfect in aspect of performance. Do you want an additional help? Brian, what Jaroslav suggests in comment#7 is the correct solution you should implement in lorax. If you need any help with changing lorax code, please reach out to us via email or (even better) schedule a call so we can work interactively on solving the problem. Discussed during the 2018-10-08 blocker review meeting: [1] The decision to classify this bug as an RejectedBlocker was made: "there does not seem to be any current violation of the release criteria here, this is not affecting F29 composes" [1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2018-10-08/f29-blocker-review.2018-10-08-16.00.log.txt (In reply to Jaroslav Mracek from comment #7) > The correct solution would be as I mentioned above: > ``` > query = > dnf.subject.Subject(name).get_best_query(dbo.sack). > filter(version__glob=version, latest=True) > if not query: > install_errors.append(("%s-%s" % (name, version), "No match")) > continue > sltr=dnf.selector.Selector() > sltr = slts.set(pkg=query) > dbo.goal.install(select=sltr, optional=(not strict)) > ``` > From my point of view this is really not a bug and DNF returns everything > correctly. I'm already using package_install() which contains similar code to that. Your example doesn't work though, Selector() isn't documented, and self.goal seems to not exist, although self._goal does :) The problem, which I've worked around using max(), is that it returns too many results. Even if I change it to use this: dnf.subject.Subject(name).get_best_query(dbo.sack).filter(version__glob=version).latest() which according to https://dnf.readthedocs.io/en/latest/api_queries.html#dnf.query.Query.latest should limit the result to 1 package per name/arch? I suppose this could be a corner case since the current issue seems to be that the *same* package is in 2 places for f29. As for name, version the name is a package name, not a NEVRA or glob. The version is a glob or version number to match. Selector is described here https://dnf.readthedocs.io/en/latest/api_selector.html. Unfortunately not descriptive. The description for latest() says Return a new query that limits the result to ``limit`` highest version of packages per package name and per architecture. In case the limit is negative number, it excludes the number of latest versions according to limit. It means that it can returns multiple packages with latest version, but anyway it is not perfect and we will improve it. Here is some improvement https://github.com/rpm-software-management/dnf/pull/1240 But back to your code. What about: for name, version in projects: # Find the best package matching the name + version glob # dnf can return multiple packages if it is in more than 1 repository query = dbo.sack.query().filterm(name=name) if version: query.filterm(version=version) # decide what is better "latest" filter or "latest_per_arch" query.filterm(latest=1) # query.filterm(latest_per_arch=1) if not query: install_errors.append(("%s-%s" % (name, version), "No match")) continue # If there is a multiple arch in query ask your self what you want to add to create a single install request or request per arch. sltr=dnf.selector.Selector() sltr = slts.set(pkg=query) # in near future there will be a "goal" attribute of Base class dbo._goal.install(select=sltr, optional=(not strict)) Thanks for the suggestions, I have it working with a bit more tweaking (I need to use provides__glob instead of name and version__glob instead of version in the filterm() calls. Note that the .filterm() isn't documented on dnf.readthedocs.org which makes it a bit difficult to figure out how to use it. Brian: do you think we could get the fix on F28 as well as F29? It's impossible to build an installer image for F28 using updates repo atm... Never mind, the F28 bug turned out to be something else. |