Created attachment 1400046 [details]
Reproducer script

Run this to reproduce the problem.

Created attachment 1400047 [details]
Output from reproducer

Note the multiple packages returned by the query+filter, and the multiple versions of firmware, eg. iwl1000-firmware from both repos with different versions.

I think that the first problem is not a bug. The get_best_query returns all packages that were represented by provided string. Because you used a provide therefore multiple package names can return. The correct solution would by to pass the result query to base.goal.install(select=sltr, optional=(not strict)) where:
sltr=dnf.selector.Selector()
sltr = slts.set(pkg=query)


The second part with with filter(latest=True) is fixed in upstream. Please can you check it for our copr repo ("dnf copr enable rpmsoftwaremanagement/dnf-nightly") - fixed in libdnf-0.13.0+

Please if you thing that issue is somewhere else, or upstream version does't work like you expect, don't hesitate to reopen the bug report.

*** Bug 1548635 has been marked as a duplicate of this bug. ***

"The get_best_query returns all packages that were represented by provided string."

In that case, what does "best" mean?

I am not sure, but get_best_query() tries if provided string is NEVRA or its part, then if it is a valid provide or, file provide. I think that best was supposed as as best choice in best order. Like if string return positive result for nevra search and provide it returns only result for nevra search. Or nevra often can be parsed as name or name-version. Then if both provides a result only one will be returned according to given priority (in forms or default priority).

can we get this bug fixed in f28 ? I'm hitting an issue in lorax in f28: https://github.com/weldr/lorax/issues/368

The problem can be fixed in F28 in about next 30 days with dnf-3.0 release.

should we open another bug to track that?

(In reply to Jaroslav Mracek from comment #8)
> The problem can be fixed in F28 in about next 30 days with dnf-3.0 release.

That's not ideal, it's a regression causing rel-eng issues in particular around composing atomic updates.

I ran into this today as well.  I agree with Peter; 30 days isn't ideal or acceptable.

Actually thinking about this at all a regression _AT_ _ALL_ like this for a stable release is actually completely unacceptable!!

Where is the test process to ensure no regression? This is core functionality now affecting (at a minimum):
* Fedora Atomic host
* Fedora Atomic Workstation
* IoT

Sooooo.... we are past 30 days. Any news?

(In reply to Matthew Miller from comment #13)
> Sooooo.... we are past 30 days. Any news?

AFAICT the comment about 30 days was made 3 days ago.

(Oh, sorry, calendar math error. I see that we are not past thirty days, just three. Still. That seems like a long time!)

As I understand it, Atomic Host has a workaround in place. Peter, do you have a similar workaround for IoT, or is this blocking you?

rpm-ostree uses a (currently forked) version of libdnf and some of the behavior around queries is different.  To my knowledge rpm-ostree isn't affected by this.

For a lot of our editions, it's actually just a pungi-imposed constraint that the generated artifacts use an Anaconda version (generated by lorax) from the same package set.  We can trivially unblock ourselves in a lot of these cases by just using a known-good installer image, and updating the "known good" version one periodically, etc.

(In reply to Matthew Miller from comment #16)
> As I understand it, Atomic Host has a workaround in place. 

Yes the workaround is here: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/bodhi2/backend/templates?id=7855fe096908181b5fb272651659a196596083e2

(In reply to Colin Walters from comment #17)
> rpm-ostree uses a (currently forked) version of libdnf and some of the
> behavior around queries is different.  To my knowledge rpm-ostree isn't
> affected by this.

This was an issue in building the ISO. Not in rpm-ostree itself.

(In reply to Colin Walters from comment #18)
> For a lot of our editions, it's actually just a pungi-imposed constraint
> that the generated artifacts use an Anaconda version (generated by lorax)
> from the same package set.  We can trivially unblock ourselves in a lot of
> these cases by just using a known-good installer image, and updating the
> "known good" version one periodically, etc.

yeah I agree. Having an installer that was maintained separately wouldn't be a bad idea. I've mentioned it before: https://pagure.io/pungi-fedora/pull-request/598#comment-50701

Still hitting this in f28 (at least) with libdnf-0.11.1-3.fc28.x86_64 and nothing in updates-testing. Any ETA on a fix?

Please can you try dnf-3.0 from cops repo (dnf copr enable rpmsoftwaremanagement/dnf-nightly)?

(In reply to Jaroslav Mracek from comment #22)
> Please can you try dnf-3.0 from cops repo (dnf copr enable
> rpmsoftwaremanagement/dnf-nightly)?

Yes, libdnf-0.16.0-0.11g230dc638.fc28.x86_64, solve the problem for me.

Note that this prevents a test I'm currently working on from being useful for stable releases. I'd like to have an openQA test run on candidate updates that creates a netinst image using lorax then runs install tests with it; obviously for stable releases it should include all packages from the release repo, packages from the stable updates repo, and packages from the candidate update. But because of this bug, the image build always fails if both the release repo and the stable updates repo are used.

Just to make the above comment more obvious since I'd missed the implication:

We'd really like this bug fixed in the stable F28 release, not just in future releases, because it impedes testing of updates.

The backporting will be difficult due to massive changes in upstream that cannot be even compiled in libdnf-0.11.1. Additionally the new behavior will brake dnf and probably other tools, because in some cases dnf-2.7.5 rely on behavior of libdnf-0.11, therefore it is risky and requires a lot of resources for backporting and testing (postpone other work). Are you ok with that?

If this will work from F29 forwards, that sounds like more effort and risk than would be worthwhile for the purposes of doing update testing on F28 and F27.

Marking private as this is a reply to a private comment, but did it really need to be private? I see nothing in it that is confidential.

I'm okay with that as long as Fedora QA is.

I appreciate there's a lot of work and everything is overwhelming, so if this is something we can reduce in favor of things being better on F29+, that's probably the right call.

Also, yes, I share Adam's wish to make these comments un-private, if that's okay with you.

(In reply to Jaroslav Mracek from comment #26)
> The backporting will be difficult due to massive changes in upstream that
> cannot be even compiled in libdnf-0.11.1. Additionally the new behavior will
> brake dnf and probably other tools, because in some cases dnf-2.7.5 rely on
> behavior of libdnf-0.11, therefore it is risky and requires a lot of
> resources for backporting and testing (postpone other work). Are you ok with
> that?

So does this mean I'm going to need to kludge together my own solution to this in lorax?