DNF has already to options how to resolve your issue.
1. Exclude removed package. In this case dnf will be unable to install it.

2. In general disable installing weak deps. Add "install_weak_deps=false" into /etc/dnf/dnf.conf or use "--setopt=install_weak_deps=False" from the commandline for certain operations.

I know that this is not exactly what you request, but it should help.

Other solution of your issue could result that transaction provided by libsolv and stored with --degugsolver option will differ from set of packages set to install. It is also not transparent logic and in such cases some users will recognize the new behavior as a regression in behavior.

I believe that we cannot do more for you.

1. might help, but is tedious
2. I want new weak deps, I don't want to disable all

That said, I understand your reasons not to accept this. Thanks

See also this upstream issue:
https://github.com/openSUSE/libsolv/issues/168
that was closed with no actual fix.

Neither proposed solution is acceptable. This issue makes weak dependencies essentially useless.

There is a very clear mathematical/logical definition of the expected behavior:
https://github.com/openSUSE/libsolv/issues/168#issuecomment-349738465

I do not see how this would confuse users. It is the current broken behavior that is confusing users. If I deliberately opted out of a weak dependency, DNF should NOT reinstall it behind my back.

The implementation will be very complicate, because each transaction will require scanning the whole history. Also it will require to store whole decision tree created by libsolv. I would like to point out that complicate thinks has a tendency to break and ar untransparet.

Additionally dnf never knows reason why user removed a package or what is an intention in future. We only see the command. Also single computer could have multiple operators therefore multiple reasons behind the action. To test that scenario just send some dnf command to frend and ask him why you did it or what command will be the next.

I am closing it again as wontfix, because it is impossible to implement it with behaviour that was requested and without any side effect. Also we have to focus on more important issues or features. The same opinion have people around libsolv (solver used by dnf), and without support in libsolv we cannot do anything.

(In reply to Jaroslav Mracek from comment #4)
> The implementation will be very complicate, because each transaction will
> require scanning the whole history.

This is absolutely not true. See the simple set arithmetic rule that I posted to the upstream bug:

| When I update foo-1 to foo-2, that should only install Recommends that are
| new in foo-2 (i.e., that are contained in the set difference
| Recommends(foo-2)\Recommends(foo-1)). If the Recommends was already present
| in foo-1, but the package was not installed on the system, it should be
| assumed that that was for a reason.

which is actually used by other depsolvers out there (e.g., as far as I know, apt-get does exactly that). The history is completely irrelevant, you only have to compare the installed and requested version of the package foo.

> I would like to point out that complicate thinks has a tendency to break and
> ar untransparet.

What is intransparent is that updating a package installs the same old unchanged weak dependencies again and again each time!

> Additionally dnf never knows reason why user removed a package or what is an
> intention in future. We only see the command.

Again, this is irrelevant if you use the above simple set arithmetic heuristic. If the Recommends was already there in the existing installed package, but the recommended package is not present, it is safe to assume that it was deliberately removed. If the Recommends is new in the updated package to be installed, it should be pulled in by default.

> The same opinion have people around libsolv (solver used by dnf), and
> without support in libsolv we cannot do anything.

The upstream libsolv bug needs to get reopened, too.

The current broken behavior is making Recommends essentially useless. What use is it to be able to remove a dependency when every single update will pull it back in again and again?

Then I am changing the component to libsolv. Without a support in libsolv we cannot do much. But libsolv upstream already rejected the idea, therefore it requires the new source of contribution. Kevin because you know how to easily deliver the feature, please could you also provide the code that handles it?

DNF can record the reason why some package was removed and tell libsolv not to install it.

I will have a look at implementing my idea in libsolv (probably as yet another optional flag, they have a bunch of those for various distro's special wishes already), but given that I'm not familiar with the libsolv code at all, I cannot promise anything at this time.

As for Igor's idea, recording the reason why some package was removed is not necessarily enough, because it will not cover the:
dnf --exclude=unwanted-soft-dependency install foo
dnf update (with a new version of foo)
case.

(In reply to Kevin Kofler from comment #8)
> I will have a look at implementing my idea in libsolv (probably as yet
> another optional flag, they have a bunch of those for various distro's
> special wishes already), but given that I'm not familiar with the libsolv
> code at all, I cannot promise anything at this time.
> 
> As for Igor's idea, recording the reason why some package was removed is not
> necessarily enough, because it will not cover the:
> dnf --exclude=unwanted-soft-dependency install foo
> dnf update (with a new version of foo)
> case.

If we had a --weak-exclude or equivalent (something like a zypp lock, which are "soft" in that something that Requires it will override it), then we could write that as a *reason* in the database for the exclusion, and treat that accordingly.

> 1. Exclude removed package. In this case dnf will be unable to install it.

A downside to this is that other packages might have a hard requirement on the excluded package, and a user may forget they set that exclude.  For exclusions to be viable, you would need to add the concept of a "weak exclusion" that excludes the package from being installed from a recommends, but not from a requirement.  I'm not necessarily advocating for that idea, just describing what it would take for exclusions to be a good way to handle this.

After submitting my last comment I realize that I basically just restated Neal's point, albeit from a config viewpoint not a flag.  Sorry about that.

I am really sorry but DNF team cannot implement it without a support in libsolv. Libsolv upstream says NO - https://github.com/openSUSE/libsolv/issues/168. Maintainer of libsolv dowstream says that:

(In reply to Igor Gnatenko from comment #7)
> DNF can record the reason why some package was removed and tell libsolv not
> to install it.

It means again no support in libsolv. 

Also the request has limited value and quite complicate or impossible implementation in libsolv. If it is difficult to implement it in solver, it is impossible to implement it in DNF and still provides reliable results.

I am sorry but again I am closing it.

This is just ridiculous. It is obvious that weak dependencies are of very limited use due to this limitation, it can be very clearly defined what the correct behavior (according to the principle of least surprise) should be (see https://github.com/openSUSE/libsolv/issues/168#issuecomment-349738465 ), yet there has been no movement on this serious usability issue for more than 3 years, and DNF and libsolv developers just keep punting it to each other (and those developers who are active in both projects have done nothing to solve the issue either).

There has been much more complex special-case behavior implemented for, e.g., the Modularity feature, yet nobody wants to implement the logical behavior for weak dependencies.

I also do not understand your statement that "the request has limited value". The limitation at hand essentially makes Recommends equivalent to Requires, because even if you explicitly exclude or remove the weak dependency, the next unrelated update of the package that has the Recommends will drag it back in again behind your back (and you might not even notice if you use the GUI updaters). This request is about making Recommends actually useful, which I think is a very clearly defined added value.

(In reply to Jaroslav Mracek from comment #12)
> I am really sorry but DNF team cannot implement it without a support in
> libsolv. Libsolv upstream says NO -
> https://github.com/openSUSE/libsolv/issues/168. Maintainer of libsolv
> dowstream says that:
> 
> (In reply to Igor Gnatenko from comment #7)
> > DNF can record the reason why some package was removed and tell libsolv not
> > to install it.
> 
> It means again no support in libsolv. 
> 
> Also the request has limited value and quite complicate or impossible
> implementation in libsolv. If it is difficult to implement it in solver, it
> is impossible to implement it in DNF and still provides reliable results.
> 
> I am sorry but again I am closing it.

Strictly speaking, libsolv doesn't care. It's the wrong level of abstraction to deal with this. Unless rpm itself gets a "reason" field for packages, there's no reason for libsolv to care. DNF records this information in its own database, and can handle this fairly intelligently. In fact, we already do this for orphan tracking instead of using libsolv's dumber version (which really only exists for zypper and is a pretty bad implementation anyway).

This most likely needs to be paired with soft locks support in DNF (an expanded and more flexible variant of what versionlocks do today) as outlined in comment 9.

What about this workaround?

# sudo dnf update --exclude="<package> [<package>...]"

Or in /etc/dnf.conf:

[main]
<snip>
exclude=<package> [<package>...]

(In reply to Raphael Groner from comment #16)

Oh, it's /etc/dnf/dnf.conf to be fully correct.

Of course I know about these! But they are not suitable solutions for this issue:

Using --exclude each time is highly impractical and does not work with GUI updaters. So it is not a workable workaround at all.

exclude= in dnf.conf might work, if and only if PackageKit also honors it, but it is the kind of configuration that should not be necessary, and it will break if the excluded package becomes a hard dependency of something else.

Both are just poor workarounds and cannot replace properly working soft dependencies.

(PS: Last I checked, PackageKit did not honor dnf.conf at all, and there were also some limitations with excluding/including packages from .repo files. This may or may not have improved. If it has not, then the exclude= workaround also does not work at all.)

(In reply to Kevin Kofler from comment #19)
> (PS: Last I checked, PackageKit did not honor dnf.conf at all, and there
> were also some limitations with excluding/including packages from .repo
> files. This may or may not have improved. If it has not, then the exclude=
> workaround also does not work at all.)

PackageKit should be respecting these settings now, since libdnf should automatically load the config...

FWIW, I agree that being able to opt out of specific Recommends would be super useful.
(I often go through a cycle of 'dnf update', oh I get something pulled in that I don't
want, 'dnf update --exclude foo', then 'dnf update --exclude foo --exclude bar', etc.
It's doable, but not pleasant.)

I think Kevin's approach in comment #c5 above is the right approach. There is no
"history" needed, and the packages that are pulled in are determined entirely by the
installed package set and the requested package set. Nice and easy to understand.

> DNF can record the reason why some package was removed and tell libsolv not to install it.
I think that is less useful than the approach proposed above, and more complicated too.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #21)
> FWIW, I agree that being able to opt out of specific Recommends would be
> super useful.
> (I often go through a cycle of 'dnf update', oh I get something pulled in
> that I don't
> want, 'dnf update --exclude foo', then 'dnf update --exclude foo --exclude
> bar', etc.
> It's doable, but not pleasant.)
> 
> I think Kevin's approach in comment #c5 above is the right approach. There
> is no
> "history" needed, and the packages that are pulled in are determined
> entirely by the
> installed package set and the requested package set. Nice and easy to
> understand.
> 

So, libsolv *already* supports this (by inverting the SOLVER_FLAG_ADD_ALREADY_RECOMMENDED flag setting). The problem is that there's basically no avenue in which you could *revisit* those choices, which is why we don't have this solver flag enabled. But maybe that doesn't matter anymore...

(In reply to Neal Gompa from comment #22)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #21)
> > FWIW, I agree that being able to opt out of specific Recommends would be
> > super useful.
> > (I often go through a cycle of 'dnf update', oh I get something pulled in
> > that I don't
> > want, 'dnf update --exclude foo', then 'dnf update --exclude foo --exclude
> > bar', etc.
> > It's doable, but not pleasant.)
> > 
> > I think Kevin's approach in comment #c5 above is the right approach. There
> > is no
> > "history" needed, and the packages that are pulled in are determined
> > entirely by the
> > installed package set and the requested package set. Nice and easy to
> > understand.
> > 
> 
> So, libsolv *already* supports this (by inverting the
> SOLVER_FLAG_ADD_ALREADY_RECOMMENDED flag setting). The problem is that
> there's basically no avenue in which you could *revisit* those choices,
> which is why we don't have this solver flag enabled. But maybe that doesn't
> matter anymore...

Erk, I meant "which is why we don't have this solver flag *disabled*.

(In reply to Neal Gompa from comment #22)
> 
> So, libsolv *already* supports this (by inverting the
> SOLVER_FLAG_ADD_ALREADY_RECOMMENDED flag setting). The problem is that
> there's basically no avenue in which you could *revisit* those choices,
> which is why we don't have this solver flag enabled. But maybe that doesn't
> matter anymore...

Daniel Mach mentioned that he looked at that flag but it does not seem to do what this bugzilla calls for. Instead, it will merely add all unmet Recommends into the transaction, no matter if the package recommending it is being upgraded or not. In short it's for use-case of "What all Recommends might have I be missing? Pull them all in."

Daniel, am I correct?

I propose to resolve the issue by implementing a new option weakexcludes. It will be resolved by a similar way like excludepkgs configuration option including support of globs. Internally it uses SOLVER_DISFAVOR that will prevent to use described packages to satisfy weakdeps.

SOLVER_DISFAVOR:
Avoid the specified packages if the solver encounters an alternative. This
can also be used to block recommended or supplemented packages from being
installed.


See: https://github.com/rpm-software-management/dnf/pull/1771, https://github.com/rpm-software-management/libdnf/pull/1241

Examples:
Install dnf without `deltarpm`
`dnf install dnf --installroot /tmp/testweak --releasever 34 --setopt=weakexcludes=deltarpm`

To set the rule permanently:
`dnf config-manager --installroot /tmp/testweak --setopt=weakexcludes=deltarpm --save`

Then any operation like reinstall, upgrade, ... will not bring `deltarpm` as a weakdep.

But `dnf install deltarpm` will install `deltarpm`

The proposed solution is better than the status quo, however it is not what was expected when reporting this bugzilla.

Sorry, but I do not see how the proposed solution adds value at all. It is already possible to use the exclude= option for this purpose. The whole point of this RFE is that manually setting an option should NOT be needed in the first place.

(In reply to Kevin Kofler from comment #27)
> Sorry, but I do not see how the proposed solution adds value at all. It is
> already possible to use the exclude= option for this purpose. The whole
> point of this RFE is that manually setting an option should NOT be needed in
> the first place.

The difference is that when `--setopt=weakexcludes=deltarpm` is used then still `deltarpm` can be installed when any package requires it. Also `deltarpm` is listed by `dnf repoquery` or `dnf list`.

But it does not address the expectations of the RFE, which are that DNF should be able to decide automatically to not pull in those weak dependencies (and the algorithm that would allow that is mentioned in comment #5).

I tend to agree with Kevin here. What would it take to populate that SOLVER_DISFAVOR during upgrade with a list of unmet weak (Recommends) dependencies, found in the rpm database at the start of the dnf operation?

(In reply to Jan Pazdziora from comment #30)
> I tend to agree with Kevin here. What would it take to populate that
> SOLVER_DISFAVOR during upgrade with a list of unmet weak (Recommends)
> dependencies, found in the rpm database at the start of the dnf operation?

That shouldn't be difficult, but this feature should be explicitly disabled for system-upgrade/distro-sync case, because otherwise it'll break the expectation that system upgrades bring you as close to fresh installed systems as possible. This will also mean that the workaround that Workstation does to get the OpenH264 codecs installed on Firefox updates will no longer work.

I would assume that this would be covered by some boolean option so that it's easy for admins to change their defaults, and for special cases like system-upgrade to do the same.

But as a data point, if I know that I don't want certain Recommended packages (and their dependencies) cluttering my workstation, I typically want that to be the case after system-upgrade as well. Currently I use https://copr.fedorainfracloud.org/coprs/isimluk/fedora-minimal/ to prevent the weak dependencies from getting pulled back in but being able to have dnf maintain the status quo would certainly be easier.

(In reply to Miro Hrončok from comment #26)
> The proposed solution is better than the status quo, however it is not what
> was expected when reporting this bugzilla.

To clarify: The proposed feature does not solve this bugzilla.

Thanks a lot for clarification. It means that we should close the bugzilla as wontfix (again).

The logic that is requested is complicated, fragile and not transparent.

1. Complicated
A lot of calculation during runtime with unpredictable results. Available and installed packages are from multiple resources and dependencies varies. We never know why the package was not installed on the first place. There are different expectations from maintainers, distribution and end users and even different expectations from the same set of people but for different packages.

2. Fragile
Dependencies are dynamic. In one version the dependency can be there in another one it can disappear or modify or can be satisfied by a different set of packages. There are also multiple reasons why weak deps were not satisfied in the first place.

3. Not transparent
We are going to introduce rules that are not written in rpm, metadata, therefore for many users it will be a puzzle.

We never know what was intended by user, we only see arguments. We do not know whether rules applied to the transaction were intended as permanent rules or temporal. This is the reason why user is supposed to modify dnf configuration files to change behavior of dnf.

I remember only one feature in DNF that has the same set of weaknesses - modularity. Do we really want to go by this way?

You, the dnf developers, have every right to decide whatever you want to implement this (or any other) feature. I am here only as somebody who thinks that the current UX wrt weak dependencies is not ideal and who wants to make the experience better for Fedora users (and quite frankly for myself). I understand you have other priorities, I have no power to force you to do anything.

I disagree that the implementation proposed by Kevin in comment #5 is complicated, fragile and not transparent. Quite the opposite, really. I think it is rather simple, deterministic and can be made absolutely transparent if needed (e.g. by producing "Not installing the following weak dependencies that were already not satisfied" section in the output). So yes, I really want to go this way. I think it would make the UX significantly better than it currently is.

I really don't understand what does this have to do with modularity at all. This RFE, unlike modularity, has a very limited scope.

Let's try to dissect and maybe simplify it a bit:

1. It does not matter why the package is not found installed on the machine. It does not matter if it was excluded from the initial kickstart, or installed and upgraded and only then removed -- all that matters is that it's not there in spite of the Recommends, and should not be there after upgrade either. The history does not play any role here.

2. Yes, it's true that the Recommends weak dependency might be new in the upgraded version of the package. But then in the old version of the package there is no unmet (broken) Recommends and this logic should not kick in for this particular Recommended package at all -- normal install_weak_deps logic applies.

And again, it does not really matter why they were not satisfied -- if they weren't there before, there was nothing to satisfy; and if they were there and were not satisfied -- they shouldn't be satisfied after the upgrade either.

3. Dnf is currently not great in being very informative about the details of the decisions it makes, unless you are willing to dive into the libsolv dumps. I'm all for making the situation better as part of this effort, by (for example) listing what packages will be set to that SOLVER_DISFAVOR, to make the beaviour a clear(er) to the admin.

> We never know what was intended by user, we only see arguments.

And that's fine -- having to use --setopt=install_weak_deps_during_upgrade=0 or --setopt=upgrade_pulls_unmet_recommends=0 (or similar in dnf.conf) is just fine to be explicit about the intent. We do not propose to make unconditional change to the behaviour -- just expose the option.

What will be the default can be distro-specific -- RHEL may chose to use the stricter approach, Fedora the more relaxed "just install everything every time" style.

> We do not know whether rules applied to the transaction were intended as permanent rules or temporal. This is the reason why user is supposed to modify dnf configuration files to change behavior of dnf.

However, you are asking the user to make informed decisions by explicitly listing the packages they don't even know they don't want (because they are currently not installed on the system). What is the way how the admin is supposed to determine the list of unmet Recommends on their system? How should they get the list? Do we even give them an API (command line) to make that configuration change?

Frankly, when I see packages I know I don't want or need on my system, I don't care how they got there, whether I installed them for whatever reason a year ago or they got pulled in via Recommends during last dnf upgrade and I missed it. I just know I want the package removed. And by doing that (and thus breaking the Recommends), that's the decision right there, captured in the rpm database. Isn't it?

> I remember only one feature in DNF that has the same set of weaknesses - modularity. Do we really want to go by this way?

Hmmm, isn't it a bit the other way round? Weak dependencies feature got introduced (like modularity) and with that came unfortunate user experience. That only became apparent when enough maintainers started to be creative with weak dependencies and people started to find out that they are removing the same packages from their systems over and over again. So we actually already went that path, now we are just trying to make it bearable.

Let ask another question - why maintainers add forward weakdeps into package? They know that they will be installed every time when package will be updated.
Why the did not remove such a deps especially when distribution did not like it? I guess they have their own reasons - They see benefits for end users.

Why distribution wants to create deployments without particular packages installed to satisfy weak deps and to keep them out after update? I guess they have their own reasons - They see benefits for end users.
Why people around distribution cannot make an agreement with maintainer? Probably because they see a different sets of benefits or problems.

And what about end users? They have also their own reasons but who knows what they expect?

And the last question: Why anyone ask DNF team to hack content to work differently from data described in rpm metadata?

(In reply to Jaroslav Mracek from comment #37)
> Let ask another question - why maintainers add forward weakdeps into
> package? They know that they will be installed every time when package will
> be updated.

In glibc, we do it because we want these dependencies to be installed by default, but still give users to deinstall them easily (without installing an equivs-style replacement package or using rpm --nodeps). The reinstall on update doesn't work for this use case because the package comes back all the time. But I'm not aware of any alternative that would work better.

(In reply to Jaroslav Mracek from comment #37)
> Let ask another question - why maintainers add forward weakdeps into
> package? They know that they will be installed every time when package will
> be updated.

I bet most of them doesn't know that. Because that behavior is so unexpected.

But those of us who do know that choose to use Recommends anyway because there is no better alternative for "optional dependencies installed by default with an easy opt-out for the users". Additionally, we hope that one day the behavior might get better.


> Why the did not remove such a deps especially when distribution did not like
> it? I guess they have their own reasons - They see benefits for end users.
> 
> Why distribution wants to create deployments without particular packages
> installed to satisfy weak deps and to keep them out after update? I guess
> they have their own reasons - They see benefits for end users.
> Why people around distribution cannot make an agreement with maintainer?
> Probably because they see a different sets of benefits or problems.

I don't follow what do you mean by distribution people, sorry. Is this part of your comment relevant to Fedora?


> And what about end users? They have also their own reasons but who knows
> what they expect?

Technically, you are correct that we don't know what do the users expect. But it seems from the discussion here (and on the mailing lists) that many of the Fedora packagers (who also happen to be the users) expect the behavior as requested in this RFE.


> And the last question: Why anyone ask DNF team to hack content to work
> differently from data described in rpm metadata?

Nobody asked that. The metadata says "x is recommended by y". I've merely asked to be more user-friendly when interpreting that metadata.

(In reply to Jaroslav Mracek from comment #37)
> Let ask another question - why maintainers add forward weakdeps into
> package?

Great question. Here's what I found out when I approached various maintainers about their Recommends.

I feel it's a mixture of being welcoming to users (kind of "other customers also liked these goods with this product" eshop approach), being part of distribution architecture decisions while still only working with their own .spec file (because many maintainers don't really know how comps could be used for affecting installed package sets), and taking the easy route instead of previously needed split into subpackages.

Maintainers know they cannot justify putting Requires there, so they go with Recommends and say "you can remove that weakly-dependent package if you don't like it". And when I point out that it will only stay removed until the next dnf upgrade of the Recommending package, they say dnf should change behaviour.

> They know that they will be installed every time when package will
> be updated.

No, they don't. We'd need a survey to make that claim but from my interactions with maintainers over the past couple of weeks, they are surprised when I point that out.

> Why the did not remove such a deps especially when distribution did not like
> it? I guess they have their own reasons - They see benefits for end users.

Yes, and it's cheap and easy to do that, since they don't really see the negative impacts. And from their package's point of view, *of course* everyone should want that Recommended dependency anyway.

> Why distribution wants to create deployments without particular packages
> installed to satisfy weak deps and to keep them out after update? I guess
> they have their own reasons - They see benefits for end users.

It's not distributions that want to create those deployments, it's admins that want to do that. Limiting space consumption and limiting installation of unneeded software that can only introduce security problems are some of the reasons.

> Why people around distribution cannot make an agreement with maintainer?
> Probably because they see a different sets of benefits or problems.

Exactly.

> And what about end users? They have also their own reasons but who knows
> what they expect?

Let them chose with install_weak_deps_during_upgrade or upgrade_pulls_unmet_recommends or whatever-the-name option.

> And the last question: Why anyone ask DNF team to hack content to work
> differently from data described in rpm metadata?

It's not differently, it's exactly based on the rpm metadata.

Let me ask a question in return -- when I have cura installed but not texlive (and not the other 996 - 222 = 774 packages of other dependencies) as I've already removed texlive before, and I run dnf upgrade -- isn't my expectation that texlive will stay out of my disk at least partially valid? Do you say that that use-case is not justified?

(In reply to Jan Pazdziora from comment #40)
> > And what about end users? They have also their own reasons but who knows
> > what they expect?
> 
> Let them chose with install_weak_deps_during_upgrade or
> upgrade_pulls_unmet_recommends or whatever-the-name option.

We can implement such an option when libsolv will support it. So far libsolv upstream did not like that idea.

> > And the last question: Why anyone ask DNF team to hack content to work
> > differently from data described in rpm metadata?
> 
> It's not differently, it's exactly based on the rpm metadata.

Please take in account that maintainers time to time remove dependencies by the accident and in next release they add it again. When it happens then auto logic described in comment 5 will fail. Also I can change dependencies between release.
`Recommends: deltarpm` to `Recommends: deltarpm(x86_64)` or `Recommends: deltarpm > 5.34`. Each of those dependencies can lead to a different set of solutions. For example `deltarpm(x86_64)` can be provided by multiple packages. This is the reason why I don't like the conditional auto exclusion of weakdeps on upgrade. Also I dislike the proposed change of behaviour between upgrade and system-upgrade (Comment 31). I suggest that users wants to keep out the same set of packages in latest Fedore like in Fedora before. And this is another reason why I prefer mechanism that will set exceptions according to configurations (fully controlled by end the user).
 
> Let me ask a question in return -- when I have cura installed but not
> texlive (and not the other 996 - 222 = 774 packages of other dependencies)
> as I've already removed texlive before, and I run dnf upgrade -- isn't my
> expectation that texlive will stay out of my disk at least partially valid?
> Do you say that that use-case is not justified?

I agree that we need to implement a mechanism how to control weakdeps. Personally I only disagree with auto detection approach that was suggested in bug. From technical point of view I disagree to implement the auto logic in DNF but it must be supported in libsolv. Workarounds in dnf will bring a pain.
I also believe that the issue with installing unwanted weak deps can be resolve by my proposal. Where during deployment the rules can be presets and later on they can be modified by the end user.

> Please take in account that maintainers time to time remove dependencies by the accident and in next release they add it again.
> When it happens then auto logic described in comment 5 will fail. Also I can change dependencies between release.

I think this isn't a practical concern. I surely *happens*, but probably only very rarely.
This would be like dropping or adding another kind of dependency by mistake… It happens,
people open bugs, the issue is resolved. It certainly isn't frequent.
And even if a Recommends is dropped and readded, this will usually happen in rawhide, and
only rarely in an update that reaches stable releases.

- we have 3429 Recommends or Suggests in .spec files [*]
- 54 are (... if ...)
- 196 are '... = %{version}-%{release}'
- 249 are other forms with '='
- 234 with %{_isa}

So maybe 80% are the basic form with just a name.

[*] all done using primitive grep, so it only takes lines that appear literally in
.spec files, doesn't do automatically generated deps, etc. But since those are the
ones that are most likely to be changed by hand, I think this is OK.

(In reply to Jaroslav Mracek from comment #42)
> Please take in account that maintainers time to time remove dependencies by
> the accident and in next release they add it again. When it happens then
> auto logic described in comment 5 will fail. Also I can change dependencies
> between release.
> `Recommends: deltarpm` to `Recommends: deltarpm(x86_64)` or `Recommends:
> deltarpm > 5.34`. Each of those dependencies can lead to a different set of
> solutions.

When we have a deterministic behavior, we can solve this it in the guidelines.

E.g. saying that when the exact form of the Recommends tag changes, it might cause dnf to install the previously removed/excluded package on updates. The guidelines can discourage changing them unless absolutely necessary and discourage using %{version}-%{release} in Recommends, etc. This is a problem that can be solved. Compared with the status quo, it is a minor inconvenience.

(In reply to Jaroslav Mracek from comment #42)
> (In reply to Jan Pazdziora from comment #40)
> > 
> > Let them chose with install_weak_deps_during_upgrade or
> > upgrade_pulls_unmet_recommends or whatever-the-name option.
> 
> We can implement such an option when libsolv will support it. So far libsolv
> upstream did not like that idea.

I'm not sure I follow why any support from libsolv is needed. Why can't the option only affect what dnf explores in the rpm database and what it sets in that SOLVER_DISFAVOR? In the approach your proposed in comment 25, there is no new functionality needed in libsolv -- dnf gets a list from the config file and configures the transaction in certain way. If dnf could get the list dynamically, the exact same libsolv mechanism could be used, couldn't it?

> of weakdeps on upgrade. Also I dislike the proposed change of behaviour
> between upgrade and system-upgrade (Comment 31).

I sort of disagree with that as well. But what the exact defaults should be for various situations (and if they should differ) seems like secondary topic.

> I agree that we need to implement a mechanism how to control weakdeps.
> Personally I only disagree with auto detection approach that was suggested
> in bug. From technical point of view I disagree to implement the auto logic
> in DNF but it must be supported in libsolv. Workarounds in dnf will bring a
> pain.
> I also believe that the issue with installing unwanted weak deps can be
> resolve by my proposal. Where during deployment the rules can be presets and
> later on they can be modified by the end user.

I believe that for many admins, the approach you propose with explicit configuration will turn out to be auto-detection-based in reality anyway. People will learn to run

  dnf --setopt=weakexcludes=$( get-all-unmet-recommends ) ...

or set aliases to do the same. I don't think people spend their days reading and curating lists of Recommends dependencies. They see that a package they don't want got installed or pulled into the transaction, they remove that package (and its dependencies and everything that hard-depends on it) but what was the actual weak dependency is not an interesting topic. And I don't think that it should be something that admins should be asked to spend their time on.

To maybe get a compromise, can't we go with the explicit weakexcludes option, plus an option to autopopulate it with unmet recommends? Or maybe even an explicit external command to run?

(In reply to Jaroslav Mracek from comment #42)
> Also I dislike the proposed change of behaviour between upgrade and
> system-upgrade (Comment 31). I suggest that users wants to keep out the
> same set of packages in latest Fedore like in Fedora before.

I agree with you on this point, but comment #31 is explicitly NOT part of Miro's or my feature request. It was proposed by Neal Gompa. I also disagree with that proposal.

Installing unmet Recommends dependencies should be a dedicated standalone command such as "dnf install-recommends", not a global option, nor an implicit (mis)feature of system-upgrade and/or distro-sync. That way, it is opt in and can be run at any time independently of other operations.

I generally disagree with the premise that "system upgrades" should be expected to "bring you as close to fresh installed systems as possible". I think system upgrades should preserve user customizations to the maximum possible extent. Especially in this case where we are not talking about a change in the distribution defaults that we want to deliver to users on upgrades, because if the Recommends is actually a change, then the algorithm from comment #5 will install it without any special-casing. On the other hand, if the Recommends has always been there unsatisfied, there is no change in distribution defaults, so installing the unmet Recommends on system-upgrade would deliberately revert a user customization.

> And this is another reason why I prefer mechanism that will set exceptions
> according to configurations (fully controlled by end the user).

Configuring this manually is entirely impractical. You cannot expect an end user to track down all the unwanted Recommends and maintain a list in a config file. (Heck, real end users do not even know what a config file *is* to begin with.)

> I agree that we need to implement a mechanism how to control weakdeps.
> Personally I only disagree with auto detection approach that was suggested
> in bug. From technical point of view I disagree to implement the auto logic
> in DNF but it must be supported in libsolv. Workarounds in dnf will bring a
> pain.

Well, if you can implement SOLVER_DISFAVOR, then as comment #30 points out, you can also implement the automatic solution with it.

My point of view (I'm also on the dnf team, hi :)):

A solution is certainly required for the weakdeps situation. As a power user, for myself, I fully welcome Jaroslav's solution as it gives me direct control over which weakdeps I want installed and which I don't.

As a matter of fact, the situation is similar to the list of userinstalled packages, which I also like to micro-manage myself, and I feel current dnf's presentation of the userinstalled concept is lacking and doesn't train users to think that way. Possibly as a result, (mostly unexperienced) users often `dnf install/remove whatever`, disregarding what they should have pulled as a dep and end up with a messy system. (the fact that RPMs installed outside dnf are treated as userinstalled greatly increases the mess factor here, /rant)

Akin to clearly presenting the userinstalled concept, I believe that dnf should in similar way present the weakdeps concept, as it is a mechanism that gives users choice and they need a clear presentation of the choice as well as an interface to conveniently "make" it, i.e. pick which weakdeps they want and which they don't.

As for the solution suggested by Miro, Kevin and Jan, my not entirely informed opinion is it should work in majority of cases, but it will fail in various edge cases when the reldeps change and in complex situations (I assume we agree on that). So for the most part to the end user the weakdeps excludes will appear to "somehow" work, until one of those situations arises and then something unexpected happens. And then you just need to go and `dnf remove that-pesky-weakdep` again, right? If we look at it from a mundane unexperienced user perspective, this is non-deterministic behavior. They have no way to expect this and they will not analyze it. And this is my issue with the approach. While we don't want our users to necessarily know where the configs are and how this works, we also don't want to train them to believe things are magically working most of the time and for the rest, you just run some commands until it looks good again. If there's the weakdeps concept that the user needs to deal with, they should have explicit and unambiguous control over it. IMO and AFAIK attempts to be too clever have ultimately failed historically and we carry a lot of those burdens to this day. I think Jaroslav knows a lot about it and when he says it's not a good idea, it's the best informed opinion you can get around here :)

Of course the user interface needs to be convenient and clear. In Jaroslav's proposal the focus was on the PoC of the business logic, the UI needs to be polished and is up for discussion. I believe it can be made in such a way that users won't need to care about config files and it will be convenient enough for them, but I'll save drafting it for another time...

(In reply to Lukáš Hrázký from comment #47)
> As a matter of fact, the situation is similar to the list of userinstalled
> packages, which I also like to micro-manage myself, and I feel current dnf's
> presentation of the userinstalled concept is lacking and doesn't train users
> to think that way. Possibly as a result, (mostly unexperienced) users often
> `dnf install/remove whatever`, disregarding what they should have pulled as
> a dep and end up with a messy system. (the fact that RPMs installed outside
> dnf are treated as userinstalled greatly increases the mess factor here,
> /rant)

For the record, this whole "userinstalled packages" concept is something I disable completely (clean_requirements_on_remove=false). I do not really care why some package was installed initially. If I remove a package, I expect it to remove only its reverse dependencies, not any of its dependencies. For that one use case of "I installed something yesterday and I want to completely undo that installation including the dependencies", I use "dnf history undo". In all other cases, I do not care about the history.

But please note that the feature proposed here is different because it does NOT depend on the history, ONLY on the current state of the RPM database (i.e., which Recommends are already unsatisfied right now). I also explicitly disagreed with proposed (even as a non-goal, see, e.g., comment #4) solutions that would depend on the history of why something was installed, exactly because those are confusing and too complicated, and not necessary to begin with.

> Akin to clearly presenting the userinstalled concept, I believe that dnf
> should in similar way present the weakdeps concept, as it is a mechanism
> that gives users choice and they need a clear presentation of the choice as
> well as an interface to conveniently "make" it, i.e. pick which weakdeps
> they want and which they don't.

Manually tracking and tweaking the userinstalled packages is exactly something I am NOT willing to do. Please do not introduce more such unhelpful "features" that are a pain for power users, and entirely unusable for ordinary end users.

> As for the solution suggested by Miro, Kevin and Jan, my not entirely
> informed opinion is it should work in majority of cases, but it will fail in
> various edge cases when the reldeps change and in complex situations (I
> assume we agree on that). So for the most part to the end user the weakdeps
> excludes will appear to "somehow" work, until one of those situations arises
> and then something unexpected happens. And then you just need to go and `dnf
> remove that-pesky-weakdep` again, right?

So what? How is that worse than the current status quo where you ALWAYS "need to go and `dnf remove that-pesky-weakdep` again" after every single update of the package carrying the Recommends?

It is much better to have to remove an unwanted package once in a while in a corner case (I expect that to almost never happen, especially in the approach proposed in comment #30 where the unmet Recommends are tracked globally, not per package, so even installing a new package that also Recommends the unwanted package will not drag it in) than to have to do that all the time.

> If we look at it from a mundane unexperienced user perspective, this is
> non-deterministic behavior.

The behavior is as deterministic as it can possibly get. The behavior actually perceived as non-deterministic is the current one, because whether an unwanted dependency gets dragged in or not depends on which packages happened to be updated in the update push.

> They have no way to expect this and they will not analyze it.

So how can you expect them to work with the approach Jaroslav and you are proposing, which requires much more analysis from the end user?

> Of course the user interface needs to be convenient and clear. In Jaroslav's
> proposal the focus was on the PoC of the business logic, the UI needs to be
> polished and is up for discussion. I believe it can be made in such a way
> that users won't need to care about config files and it will be convenient
> enough for them, but I'll save drafting it for another time...

Considering that we have at least 2 different middlewares (dnfdaemon and PackageKit) on which at least 3 different UIs (Dnfdragora, GNOME Software and Plasma Discover) (plus update-only ones such as plasma-pk-updates) are building, I do not see how DNF can provide a UI for such a thing at all, in a way that benefits all users. This needs to just work with no setup needed.

[I wrote this in parallel with Kevin's reply, and it says many same things… I'll post it anyway.]

Hi Lukáš, welcome to the dicussion ;]

> As a power user, for myself

I think that shows a discrepancy in thinking about this: at least for me, this is for power users. It is
power users who have many many installation that they upgrade and are able and willing to remove individual
packages. Newbie users (or people who just don't care) will just install the defaults, including any
Recommeneded packages. So this issue is not really relevant to them, except that the *default* should
do the reasonable thing.

> As for the solution suggested by Miro, Kevin and Jan, my not entirely informed opinion is it should work in
> majority of cases, but it will fail in various edge cases when the reldeps change and in complex situations

And that is not a problem. With the solution working *perfectly*, new unwanted packages will occasionally be
pulled in because new Recommends are added. With the solution working *imperfectly*, this will still happen,
just more often. In both cases, the user will have to remedy the situation, either by restarting the
transaction with --exclude, or uninstalling some packages after the fact. That's expected, and not a big issue.
The difference with status quo, is that with the proposed solution, this will happen much much less
frequently, making the *average* experience much better. We are not asking for a perfect solution, just
something that increases the usability of weak deps.

(This also works the other way: when a weak dep is skipped, the user will sometimes decide to install it.
If dnf would be so nice as to output the list of skipped weak deps as proposed in comment #36 above,
the user would have control and could make an informed decision.)

> especially in the approach proposed in comment #30 where the unmet Recommends are tracked globally, not per package,
> so even installing a new package that also Recommends the unwanted package will not drag it in

That would be my preferred behaviour too.

Alright. I'd very much like for the discussion to not explode, as it is a hard read already...

(In reply to Kevin Kofler from comment #48)
> For the record, this whole "userinstalled packages" concept is something I
> disable completely (clean_requirements_on_remove=false). I do not really
> care why some package was installed initially. If I remove a package, I
> expect it to remove only its reverse dependencies, not any of its
> dependencies. For that one use case of "I installed something yesterday and
> I want to completely undo that installation including the dependencies", I
> use "dnf history undo". In all other cases, I do not care about the history.
> 
> ...
> 
> Manually tracking and tweaking the userinstalled packages is exactly
> something I am NOT willing to do. Please do not introduce more such
> unhelpful "features" that are a pain for power users, and entirely unusable
> for ordinary end users.

Ok. I respectfully, but completely disagree with this approach and I think you're doing it wrong. I also can't really understand why you'd even think your approach is better and I feel we share very little common ground if that's the case... Please take no offense.

It is also one of the main concepts of how dnf works, it's not going away and yes, it will be enhanced in the future. Not saying you have to use it, but again, I just don't understand why you wouldn't. I assume your system is full of packages that you don't need, because something pulled them in and then the dep requirement disappeared...

Lets not diverge to that though, I just wanted to address it. We can have the discussion elsewhere if you want, though it kind of appears it may end up not being very fruitful.
 
> So what? How is that worse than the current status quo where you ALWAYS
> "need to go and `dnf remove that-pesky-weakdep` again" after every single
> update of the package carrying the Recommends?

I'd like to point out I'm not looking for a "better than status quo" solution, I'm looking for the correct solution.

> The behavior is as deterministic as it can possibly get. The behavior
> actually perceived as non-deterministic is the current one, because whether
> an unwanted dependency gets dragged in or not depends on which packages
> happened to be updated in the update push.

Deterministic is perhaps a wrong choice of a word, but you don't understand my point. The situation where a weak dep is added to a package is a simple one and is the same with both yours and Jaroslav's proposal. The "non-deterministic" part from a user PoV is when a previously "removed" weak dep gets pulled in again due to more complex changes in how a possible weak reldep is defined or possibly other packages being installed or removed, I don't really even want to construct a case and dig into it :/ Jaroslav tried to explain it already. It's hard to really fathom if you're not familiar with how reldeps work and how libsolv internally resolves all that, and most of us here aren't really familiar with that. Jaroslav is.

> So how can you expect them to work with the approach Jaroslav and you are
> proposing, which requires much more analysis from the end user?

It doesn't require any analysis. My idea of how it could work for the end user is (simplified dnf output):

# dnf install --ask-weak foo
Dependencies resolved.
Installing:
 foo
Installing dependencies:
 libfoo
Installing weak dependencies:
 you-dont-need-this
 another-useless-package

Do you want to skip installing weak dependencies and mark them to not be installed in the future? (y/n): y

The following weak dependencies were marked to not be installed:
 you-dont-need-this
 another-useless-package


^^^ that's just a quickly thrown together example.

> Considering that we have at least 2 different middlewares (dnfdaemon and
> PackageKit) on which at least 3 different UIs (Dnfdragora, GNOME Software
> and Plasma Discover) (plus update-only ones such as plasma-pk-updates) are
> building, I do not see how DNF can provide a UI for such a thing at all, in
> a way that benefits all users. This needs to just work with no setup needed.

In short, IMO this is a pipe dream. Weak dependencies give users choices to install them or not. Having this working completely transparently just isn't going to happen. You can't give users a choice and then try to hide it from them and assume it's going to work well.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #49)
> I think that shows a discrepancy in thinking about this: at least for me,
> this is for power users. It is
> power users who have many many installation that they upgrade and are able
> and willing to remove individual
> packages. Newbie users (or people who just don't care) will just install the
> defaults, including any
> Recommeneded packages. So this issue is not really relevant to them, except
> that the *default* should
> do the reasonable thing.

I'm not sure you're argumenting against Jaroslav's proposal then. Surely a config option is not a problem and just a more explicit and reliable way to do things for power users, who have no issue with editing a config file?

We would introduce a commandline interface to ease the manipulation of the weakdeps excludes though.

> And that is not a problem. With the solution working *perfectly*, new
> unwanted packages will occasionally be
> pulled in because new Recommends are added. With the solution working
> *imperfectly*, this will still happen,
> just more often. In both cases, the user will have to remedy the situation,
> either by restarting the
> transaction with --exclude, or uninstalling some packages after the fact.
> That's expected, and not a big issue.
> The difference with status quo, is that with the proposed solution, this
> will happen much much less
> frequently, making the *average* experience much better. We are not asking
> for a perfect solution, just
> something that increases the usability of weak deps.

Well, what we in the dnf team are looking for, though, is not a hotfix to make things a bit less painful. We're looking for a solid long-term solution. Whatever we go with we'll need to maintain. If it turns out the proposed solution has issues (which is what Jaroslav is telling you all along ;)), we'll need to deal with it, and replacing the "automatic" way with e.g. the configuration option will be much harder, because it will be perceived as a regression by users and we'll be the "even worse guys" (as I'm assuming we're the "bad guys" already :D).

And we're aiming for a consistently good and consistent experience, not "better on average" (sorry) :)

> And we're aiming for a consistently good and consistent experience, not "better on average" (sorry) :)

I sincerely believe that what I (and Kevin and others) proposed here is consistently a better user experience than both the status quo and the proposed weakexcludes option.

What would make you consider it? A survey on Fedora devel mailing list that shows what the packagers actually expect?

(In reply to Miro Hrončok from comment #52)
> I sincerely believe that what I (and Kevin and others) proposed here is consistently a better user experience than both the status quo and the proposed weakexcludes option.

"Consistently better" != "consistent" :)

And what is "consistently better" is highly subjective, I think Jaroslav's solution is consistently better than yours...

> What would make you consider it?

For myself (I should stray from speaking for the whole team here) I'd say a complete analysis of how this will impact dependency resolving in libsolv together with an enumeration of cases in which the mechanism will fail. That'd do it :D

I'm joking of course, but...

> A survey on Fedora devel mailing list that shows what the packagers actually expect?

... this is certainly not the point here. We are discussing the soundness and reliability of the solution. We know what the users want.

> We know what the users want.

What do they want?

(In reply to Miro Hrončok from comment #54)
> What do they want?

To not have unwanted weak deps installed, with as little fiddling as possible. (from your PoV and from what you're requesting, with no fiddling at all, but IMO that's a bit unrealistic :))

If we all agree that what I requested here is what the users want, I'd say that should be the goal instead of a new option that the users don't want ;)

There are also real-world limitations to what is possible, reasonable and simply a good or a bad idea. I hope I have explained it well enough, so lets please keep the discussion on point, if the solution isn't good, we're not going to implement it.

Lukáš, could you show a command to list all unmet Recommends dependencies on a system?

(In reply to Lukáš Hrázký from comment #53)
> We know what the users want.

As a user, I want that dnf(1) does not pull in all weak dependencies again during "dnf update" (like it is unfortunately happening), if I uninstalled these weak dependencies before. And I don't want to configure that expected behaviour in a configuration file, nor do I want to use --yet-another-fancy-option=nobody-can-remember as being discussed here (and let alone keeping a list of excluded weak dependencies or using a shell alias for arguments etc.).

Personally, I like the proposal from comment #44 which seems to clarify a possible leftover comment #5. And even this might not be perfect, but it is better than the current dnf user experience regarding weak dependencies.

(In reply to Lukáš Hrázký from comment #50)
> I'd like to point out I'm not looking for a "better than status quo"
> solution, I'm looking for the correct solution.

Well, as a matter of fact, that is what I am looking for, too, and the solution I propose happens to be the only one I consider to be correct. :-)

> > The behavior is as deterministic as it can possibly get. The behavior
> > actually perceived as non-deterministic is the current one, because whether
> > an unwanted dependency gets dragged in or not depends on which packages
> > happened to be updated in the update push.
> 
> Deterministic is perhaps a wrong choice of a word, but you don't understand
> my point. The situation where a weak dep is added to a package is a simple
> one and is the same with both yours and Jaroslav's proposal.

Well, the default behavior for a NEW weak dependency is obviously to install it, since that is what the weak dependency is for to begin with. So it is clear that the proposals are not going to differ for that case. Where the behavior differs is when a weak dependency is NOT new.

> The "non-deterministic" part from a user PoV is when a previously "removed"
> weak dep gets pulled in again due to more complex changes in how a possible
> weak reldep is defined or possibly other packages being installed or removed,
> I don't really even want to construct a case and dig into it :/

Well, if the weak dependency changed from, say, Recommends: foo to Recommends: (foo or bar), that is arguably a new weak dependency. Depending on the implementation, it can result in a probably unwanted reinstallation of foo. If it can be avoided (i.e., either install bar instead or install nothing because the Recommends was already unmet), that is of course better, but I would not consider the implementation unusable or incorrect if it ends up reinstalling foo in this corner case.

> Jaroslav tried to explain it already. It's hard to really fathom if you're
> not familiar with how reldeps work and how libsolv internally resolves all
> that, and most of us here aren't really familiar with that. Jaroslav is.

Well, I am not going to claim to be an expert in libsolv internals (I am clearly not), but I do know my way around boolean logic and SAT solving. So if you bring up examples, I can certainly follow.

> It doesn't require any analysis. My idea of how it could work for the end
> user is (simplified dnf output):
> 
> # dnf install --ask-weak foo
> Dependencies resolved.
> Installing:
>  foo
> Installing dependencies:
>  libfoo
> Installing weak dependencies:
>  you-dont-need-this
>  another-useless-package
> 
> Do you want to skip installing weak dependencies and mark them to not be
> installed in the future? (y/n): y
> 
> The following weak dependencies were marked to not be installed:
>  you-dont-need-this
>  another-useless-package

The assumption here is that users use the DNF CLI directly as opposed to a GUI frontend, which is not a realistic assumption for an end user. And I am a power user, but still, I only use the CLI when I have to. Normally, I am going to install packages through Dnfdragora, not the DNF CLI.

> In short, IMO this is a pipe dream. Weak dependencies give users choices to
> install them or not. Having this working completely transparently just isn't
> going to happen. You can't give users a choice and then try to hide it from
> them and assume it's going to work well.

The way to achieve what the user expects has been pointed out several times in this thread now.

(In reply to Lukáš Hrázký from comment #53)
> We know what the users want.

Then why do you insist on implementing something else that you know the users do not want?

(In reply to Kevin Kofler from comment #60)
> The assumption here is that users use the DNF CLI directly as opposed to a
> GUI frontend, which is not a realistic assumption for an end user. And I am
> a power user, but still, I only use the CLI when I have to. Normally, I am
> going to install packages through Dnfdragora, not the DNF CLI.

I don't use those GUIs unfortunately, but one way or another you need a way to "turn off" (or in the way you're looking at things, I assume, to uninstall) the weak deps. I'd imagine a good GUI for that would be, after you click to install something, to give you a list of weak deps which you could uncheck to remove from the transaction. This would need to be added as weakdeps support to the GUI tools, but that's just the way to do it: If users are given an option, the option needs to be properly represented in the UI.

> The way to achieve what the user expects has been pointed out several times
> in this thread now.

I know. I've tried to point out the faults I see in that idea, as did Jaroslav.

> Then why do you insist on implementing something else that you know the
> users do not want?

The intention of all my comments in this discussion was to explain that and I don't have anything else to add, you can re-read my comments and if you still don't understand, I must have failed in conveying the message :(


Guys, we are discussing this extensively within the team, it is a thing we want to resolve as well. However, as of now it is looking like we just don't consider the "automagic" solution to be the way to go. We realize people don't want to micro-manage another non-trivial configuration option and we'll strive to give you a UI that's as friendly and as easy to use as possible. In return I'd like to ask you to believe us if we tell you a fully transparent solution isn't feasible (which hasn't been ultimately decided yet).

Thanks!

I just want to point out that attempting to end a technical discussion with "trust us that we are right and you are wrong" might not work.

Your team proposed a solution to the problem. Here you have several people explaining why that solution does not solve the problem (and no users who would seem satisfied with the proposed solution), yet you appear predetermined that this is a way to go.

I understand that your team has other priorities and a bunch of random Fedora "guys" is probably not going to change that. However, please don't patronize us.

Miro, with all due respect, you haven't explained why our proposition doesn't solve the problem. You just keep repeating that you want it your way and that it's "what the users want", and I don't see any other argument from your side (granted I could easily have missed it in the flood of comments). And this one argument you have is not technical at all.

I'm not really sure what you're expecting, that you'll just pressure us into the solution even if we don't believe it to be a good one? That's what might not work here.

We actually know for a fact that our solution does solve the problem of being able to control which weakdeps get pulled in and which not, just as your solution does, don't you agree? We're just discussing UX on one side and reliability, robustness, and being explicit instead of implicit on the other.

I'll say one more time that a plain config option was used in Jaroslav's PoC for simplicity, he just didn't bother spending the time on a nicer UI for the PoC. The idea is the list of weakdeps_excludes will be stored somewhere (not necessarily in a config file) and that a UI will be added for users to control this. In number of operations required from the user this actually boils down to essentially the same. The only real difference is that yes, you'll need to remember --yet-another-switch and possibly a new command to control it. There's no way around it if we want to be explicit and I do believe we want to be explicit.

Lukáš, this is primarily about the user experience so the fact that Miro focuses on the user experience side of things (and not the technical details) is expected. As a reporter, he described the problematic situation and why it is problematic, the dnf team has agreed that the behaviour is bad, and Miro's point is that with no or minimal changes in steps used, he'd like to get rid of the problematic behaviour.

Personally I'm very worried about your optimism of storing extra information somewhere and UI being added because it would need to be added in a way that is compatible with anaconda and all package managing frontends there are. Therefore I find it critical that the feature could get configured to behave automatically without anybody being forced to remember --yet-another-switch or a new command -- at the risk of the user enabling such an automatic behaviour. Underneath it can store and manage all information it needs if it's really needed and if it's going to work, externally it needs to be seamless.

I'd like to evaluate viability of the proposal it in real-life scenarios fairly. As mentioned in comment 58, I'd need a command to list all unmet Recommends dependencies on a system because that's what the proposal effectively forces every admin to do to start with and frankly I have no idea how to do it effectively. How do you get list of all unmet Recommends on a system?

Actual behavior:

1. I install foo when release 1 is up to date, dnf pulls in bar.
2. I remove bar.
3. dnf update
4. Package foo is updated to release 2, I get bar.

Expected behavior:

1. I install foo when release 1 is up to date, dnf pulls in bar.
2. I remove bar.
3. dnf update
4. Package foo is updated to release 2, I don't get bar.

Ergo, your proposition doesn't solve the problem.

Miro, may I assume that if a step

0. set up something in the dnf configuration, without listing specific package names

was added to the list, it would still be acceptable solution for you? In other words, is it OK if the expected behaviour is not the default out of box but can be achieved with some config change?

I'd argue it would be better than status quo, but if that option existed, I'd lobby for it to be the default for Fedora.

Two notes here:

1. I'm not trying to pressure you into "my" solution, I want you to see that it is a good solution. If I fail to manage that, maybe it isn't. It is a discussion. OTOH others seem to agree it is a good solution.

2. I don't claim to know what the users want. I have a decent idea and offered to do a survey to back it up with some data. You said you know what the users want.

(In reply to Miro Hrončok from comment #65)
> Actual behavior:
> 
> 1. I install foo when release 1 is up to date, dnf pulls in bar.
> 2. I remove bar.
> 3. dnf update
> 4. Package foo is updated to release 2, I get bar.
> 
> Expected behavior:
> 
> 1. I install foo when release 1 is up to date, dnf pulls in bar.
> 2. I remove bar.
> 3. dnf update
> 4. Package foo is updated to release 2, I don't get bar.
> 
> Ergo, your proposition doesn't solve the problem.

No. This is not a description of the problem. This is your interpretation that includes the solution you insist on as expected semantics.

The problem is users need to control which weak deps get or don't get pulled in. Assigning a very generic `dnf remove bar` command the semantics of "I don't want this weak dependency to be automatically installed" is basically forcing a concept on the user and I don't agree it's a good thing at all. Explicit is better than implicit.

This is just repeating the solution you insist on for n-th time :/


(In reply to Jan Pazdziora from comment #64)
> Lukáš, this is primarily about the user experience so the fact that Miro
> focuses on the user experience side of things (and not the technical
> details) is expected. As a reporter, he described the problematic situation
> and why it is problematic, the dnf team has agreed that the behaviour is
> bad, and Miro's point is that with no or minimal changes in steps used, he'd
> like to get rid of the problematic behaviour.

The issue is Miro's minimal solution to the problematic behavior is in itself problematic.

I consider the original bug report to be "Users need control over which weak deps get pulled in".

Given that, we need to come up with a sound technical solution to allow this.

After we have the designed a good mechanism to control this, we can design a good user interface. Surely the technical limitations of how to implement this dictate how the UI is going to be, not the other way around?

> Personally I'm very worried about your optimism of storing extra information
> somewhere and UI being added because it would need to be added in a way that
> is compatible with anaconda and all package managing frontends there are.

Indeed. By introducing the concept of weak deps in the RPM ecosystem you are giving the users a choice. This choice clearly needs to be added to all package management UIs, how else do you expect the users take advantage of having the choice? Yes, it needs to be added to the frontends and yes, it is work. It seems to be a common attitude to pick the path of least resistance and implement features with the minimal amount of effort, as it obviously requires coordinating interfaces across tools etc. This just leads to issues and inconsistent behavior. If you're not willing to do things properly, I say just don't do them, you'll be better off in the long run.

Also, for frontends not adding support for this feature the behavior would simply be the old one - the weak deps would be pulled in for newly installed packages. The previously excluded weak deps would remain in effect. The support for this can then either be added later or the users would need to use a different tool to have control over the weak deps.

> I'd like to evaluate viability of the proposal it in real-life scenarios
> fairly. As mentioned in comment 58, I'd need a command to list all unmet
> Recommends dependencies on a system because that's what the proposal
> effectively forces every admin to do to start with and frankly I have no
> idea how to do it effectively. How do you get list of all unmet Recommends
> on a system?

That's one way of doing it. There isn't such a command now (AFAIK), we may add it later. But this really is practically the same as it would be with Miro's solution, I fail to see where's the actual difference between the two.

scenario 1:

1. `dnf install foo`
   The transaction is listed and you see a list of weak deps, some of which you don't like

2. `dnf install foo --exclude-weak <LIST-UNWANTED-DEPS>`
   Instead of using --exclude or removing the weak deps post-transaction

scenario 2:

1. `dnf install foo --exclude-all-weakdeps`
   dnf installs foo without pulling in any weak dependencies; that'd be a new shortcut to exclude them all

Also, for the use case of removing the weak dep after installing it, not installing a package at all in the first place is a better way to do it, as packages may have scriptlets or other side-effects that change the system just by being installed, which may not be reverted fully when uninstalled.

We still need to figure out some details of what I outlined and the interface may end up being a bit different.

(In reply to Lukáš Hrázký from comment #69)
> 
> No. This is not a description of the problem. This is your interpretation
> that includes the solution you insist on as expected semantics.

It however captures pretty well the implicit expectations that even people who know what weak dependencies are (package maintainers) have told me about dnf behaviour. People tell me "just remove that Recommended package" and are surprised when I tell them it will get pulled back in.

> The problem is users need to control which weak deps get or don't get pulled
> in. Assigning a very generic `dnf remove bar` command the semantics of "I
> don't want this weak dependency to be automatically installed" is basically
> forcing a concept on the user and I don't agree it's a good thing at all.
> Explicit is better than implicit.
> 
> This is just repeating the solution you insist on for n-th time :/

You rephrasing it as "users need to control which weak deps get or don't get pulled in" is just repeating the problem statement to match the solution that you propose. ;-)

So ... is it time for a user survey, to ask people what their expectation about dnf remove is?

> I consider the original bug report to be "Users need control over which weak
> deps get pulled in".

And I read it as "if I removed that package twice already, maybe dnf should get the hint". :-)

> After we have the designed a good mechanism to control this, we can design a
> good user interface. Surely the technical limitations of how to implement
> this dictate how the UI is going to be, not the other way around?

My assumption is that if it's not possible to configure dnf to have the Expected behaviour as described by Miro, and that includes for dnf remove + dnf upgrade without any other options or commands to work that way, it will be a hard sell. Note, we are dnf users, we don't care about the technical limitations. We care about usability experience because that's what bites us.

So no -- the technical limitations don't dictate the UI. If the approach does not allow for "no extra steps needed" user experience, it will be a hard sell.

But as mentioned above -- survey time?

> Indeed. By introducing the concept of weak deps in the RPM ecosystem you are
> giving the users a choice. This choice clearly needs to be added to all
> package management UIs, how else do you expect the users take advantage of
> having the choice?

vi /etc/dnf/dnf.conf or similar file?

> Yes, it needs to be added to the frontends and yes, it is
> work. It seems to be a common attitude to pick the path of least resistance
> and implement features with the minimal amount of effort, as it obviously
> requires coordinating interfaces across tools etc. This just leads to issues
> and inconsistent behavior. If you're not willing to do things properly, I
> say just don't do them, you'll be better off in the long run.

I don't understand. Repeatedly I've offered to test the proposed approach and to see how hard it would be to turn it into something that I'd consider usable from the UX point of view.

> Also, for frontends not adding support for this feature the behavior would
> simply be the old one - the weak deps would be pulled in for newly installed
> packages.

Obviously not for example if libsolv would suddenly decide to change the behaviour WRT Recommends upon upgrades.

> That's one way of doing it. There isn't such a command now (AFAIK), we may
> add it later. But this really is practically the same as it would be with
> Miro's solution, I fail to see where's the actual difference between the two.
> 
> scenario 1:
> 
> 1. `dnf install foo`
>    The transaction is listed and you see a list of weak deps, some of which
> you don't like
> 
> 2. `dnf install foo --exclude-weak <LIST-UNWANTED-DEPS>`
>    Instead of using --exclude or removing the weak deps post-transaction
> 
> scenario 2:
> 
> 1. `dnf install foo --exclude-all-weakdeps`
>    dnf installs foo without pulling in any weak dependencies; that'd be a
> new shortcut to exclude them all

I start with my workstation that I've kept in a working state across numerous Fedora upgrades. I have no idea which of the packages I've removed in the past were weak dependencies and which got installed via other means. There is no "dnf install" in my case -- those were done years ago. I have "dnf upgrade" to run but I don't know what packages to exclude. How do you suggest I even start testing the feature?

> I consider the original bug report to be "Users need control over which weak deps get pulled in".

I think this is the baseline of our misunderstanding. The the original bug report is "Users should not care and the week deps should not be reinstalled over and over again on updates".

I understand that this might be hard. It might not be even possible with the available resources. That is sad, but completely understandable.

However, from the discussion here I assume you actually have resources to solve "a problem", yet you decided to solve a different one.

Jaroslav Mracek and I are working on a proposal we'll share with you shortly.

Internal implementation seems to be clear:
* Jaroslav proposed using SOLVER_DISFAVOR - it seems to be working as expected - see comment#25
* SOLVER_FLAG_ADD_ALREADY_RECOMMENDED does the opposite than what Neal expected in comment#22 - it makes it unusable for this use case

We're currently designing user interactions with this feature because it needs to integrate seamlessly with the rest of DNF.
Once we have it, we'll share it here for review.

I have an alternative prove of concept: https://github.com/rpm-software-management/dnf/pull/1774. It implements auto detection of unmet weak dependencies and disfavor relevant available packages. The concept is supposed to discover weak and strong points of each approach.

Hello folks, I'd like to point out that the proposal in https://github.com/rpm-software-management/dnf/pull/1774 seems to work out of box very well, so I'd recommend you check it out.

When testing it, I came across a couple of weird corner cases that we are discussing with Jaroslav there and your opinions would be valuable. For example, what to do with

   Recommends: exiv2 = 0.27.3-5.fc34

which becomes

   Recommends: exiv2 = 0.27.3-7.fc34

so obviously not the exact same weak dependency.

(In reply to Jan Pazdziora from comment #74)
> Hello folks, I'd like to point out that the proposal in
> https://github.com/rpm-software-management/dnf/pull/1774 seems to work out
> of box very well, so I'd recommend you check it out.
> 
> When testing it, I came across a couple of weird corner cases that we are
> discussing with Jaroslav there and your opinions would be valuable. For
> example, what to do with
> 
>    Recommends: exiv2 = 0.27.3-5.fc34
> 
> which becomes
> 
>    Recommends: exiv2 = 0.27.3-7.fc34
> 
> so obviously not the exact same weak dependency.

That was expected and we can make sure our packaging guidelines discourage Recommedns with full NERV. As said in comment #44

s/NERV/NEVR/

For testing of https://github.com/rpm-software-management/dnf/pull/1774 I created builds for Fedora: https://copr.fedorainfracloud.org/coprs/jmracek/disfavor/

Using Disfavor approach has some weak points. Disfavors are added to every transaction - for install, upgrade, downgrade, ... . It has a reason because even install request can result in upgrade operation.

With the latest changes in PR I am not sure whether the logic disfavors too many packages.

I am not sure whether disfavors can trigger some problems with very weak dependencies. They are used by langpacks and I even don't know how to test it.

Thank you very much for the testing copr!

After the Python 3.10 rebuild is done, I'll give it a try.

It looks like that disfavors brake handling of debug-source packages. The behaviour is covered by test plugins-core/debuginfo-install.feature in our CI.

@bz1532378
Scenario: debugsource is installed even if debuginfo is present on the system
Given I successfully execute dnf with args "install --setopt=install_weak_deps=False foo-debuginfo"
 When I execute dnf with args "debuginfo-install foo"
 Then the exit code is 0
 Then Transaction is following
      | Action        | Package                                   |
      | install       | foo-debugsource-0:2.0-1.x86_64            |

I fixed the problem reported in comment 81. Also all repositories with builds were updated.

(In reply to Jan Pazdziora from comment #74)
> Hello folks, I'd like to point out that the proposal in
> https://github.com/rpm-software-management/dnf/pull/1774 seems to work out
> of box very well, so I'd recommend you check it out.
> 
> When testing it, I came across a couple of weird corner cases that we are
> discussing with Jaroslav there and your opinions would be valuable. For
> example, what to do with
> 
>    Recommends: exiv2 = 0.27.3-5.fc34
> 
> which becomes
> 
>    Recommends: exiv2 = 0.27.3-7.fc34
> 
> so obviously not the exact same weak dependency.

As Miro wrote, Recommends: with exact EVR does not make sense, and is probably not even what is actually meant here. This probably ought to be:
Recommends: exiv2
Requires: (exiv2 = 0.27.3-7.fc34 if exiv2)

That is a good point irrespectively of the dnf behaviour change in question. In case of exiv2, there already is reverse

  Requires: %{name}-libs%{?_isa} = %{version}-%{release}

there in https://src.fedoraproject.org/rpms/exiv2/blob/rawhide/f/exiv2.spec#_45 so the

  Recommends: %{name} = %{version}-%{release}

in package libs in https://src.fedoraproject.org/rpms/exiv2/blob/rawhide/f/exiv2.spec#_74 doesn't really buy anything new.

And if the version weak dependency is not met, in case there isn't the reverse strong dependency present, the transaction would presumably proceed anyway.

Would you like to propose edit to https://docs.fedoraproject.org/en-US/packaging-guidelines/ to talk about this? Or should we start with a discussion at the devel@ mailing list?

> Requires: (exiv2 = 0.27.3-7.fc34 if exiv2)

That is not even necessary, because of the reverse fully-versioned dependency. But serves as  good example that it can be achieved in case the reverse dependency does not exits.


> Would you like to propose edit to https://docs.fedoraproject.org/en-US/packaging-guidelines/ to talk about this? Or should we start with a discussion at the devel@ mailing list?

That makes sense once we have the proposed behavior confirmed. I can work with the packaging committee to propose exact wording.

I create another solution (https://github.com/rpm-software-management/dnf/pull/1780, https://github.com/rpm-software-management/libdnf/pull/1279, https://github.com/openSUSE/libsolv/pull/454, tests: https://github.com/rpm-software-management/ci-dnf-stack/pull/1005). Patches requires a new feature in libsolv.

I also created a new repository with builds (https://copr.fedorainfracloud.org/coprs/jmracek/weak_excludes) for testing. There is still a question how to handle Supplements (See https://github.com/rpm-software-management/dnf/pull/1780#issuecomment-869480718).

After
# dnf copr enable jmracek/weak_excludes
# dnf upgrade dnf

to 4.8.1-1.git.9433.3410cb9.fc34

the subsequent dnf invocation fails with

# dnf upgrade
Last metadata expiration check: 0:00:15 ago on Mon 05 Jul 2021 11:13:20 AM CEST.
Traceback (most recent call last):
  File "/usr/bin/dnf", line 58, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 201, in user_main
    errcode = main(args)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 166, in resolving
    base.resolve(cli.demands.allow_erasing)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 856, in resolve
    self._add_weak_excludes_to_goal()
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 821, in _add_weak_excludes_to_goal
    self._goal.reset_weak_excludes()
AttributeError: 'Goal' object has no attribute 'reset_weak_excludes'

It was necessary to manually pull libdnf, python3-libdnf, and python3-hawkey from https://download.copr.fedorainfracloud.org/results/jmracek/weak_excludes/fedora-34-x86_64/02306005-libdnf/.

Jaroslav, thank you! This seems to be work great.

(I assume the 'excluded|not excluded' messages are for debugging and not intended for the final version.)

A short summary:
$ sudo dnf reinstall exiv2-libs
=====================================================================================================================================================
 Package                             Architecture                    Version                                  Repository                        Size
=====================================================================================================================================================
Reinstalling:
 exiv2-libs                          x86_64                          0.27.4-1.fc35                            rawhide                          778 k
Installing weak dependencies:
 exiv2                               x86_64                          0.27.4-1.fc35                            rawhide                          976 k

With the upgraded dnf:
$ sudo dnf reinstall exiv2-libs
 Package                             Architecture                    Version                                  Repository                        Size
=====================================================================================================================================================
Reinstalling:
 exiv2-libs                          x86_64                          0.27.4-1.fc35                            rawhide                          778 k

One unexpected thing is that I don't see exiv2 listed anywhere in the debug output.
I would expect it to say ''excluded exiv2-0.27.4-1.fc35.x86_64 somehwere.