I have uploaded the output of `dnf --debugsolver distro-sync` to firefox send: https://send.firefox.com/download/96d5bbc3ccf9cfaf/#e8v7kSttItMK8xPIb87Oeg (unfortunately that link will expire in 7 days or 100 downloads, so please grab a copy of it soon-ish; I can of course re-upload it).

Thanks for reporting and debugsolver data.

Please can you provide a few additional pieces of information?

- versions of other parts of dnf stack (`rpm -q dnf libdnf libsolv`)
- version of the running kernel (`uname -a`)
- dnf configuration file (`cat /etc/dnf/dnf.conf`)
- limit the distro-sync only to kernel-default package and try to find out minimal set of repositories for which the bug occurs. Please provide debugsolver data for this command. I suppose something like `dnf distro-sync --debugsolver --repoid=download.opensuse.org-oss,openSUSE-20190717-0 kernel-default` could do the job.

(In reply to Marek Blaha from comment #2)
> Thanks for reporting and debugsolver data.
> 
> Please can you provide a few additional pieces of information?
> 
> - versions of other parts of dnf stack (`rpm -q dnf libdnf libsolv`)

dnf-4.2.19-1.2.noarch
libdnf2-0.45.0-3.1.x86_64
libsolv-devel-0.7.11-1.2.x86_64

> - version of the running kernel (`uname -a`)

Linux Boreas 5.6.4-1-default #1 SMP Mon Apr 13 18:13:21 UTC 2020 (e42da4f) x86_64 x86_64 x86_64 GNU/Linux

> - dnf configuration file (`cat /etc/dnf/dnf.conf`)

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True

> - limit the distro-sync only to kernel-default package and try to find out
> minimal set of repositories for which the bug occurs. Please provide
> debugsolver data for this command. I suppose something like `dnf distro-sync
> --debugsolver --repoid=download.opensuse.org-oss,openSUSE-20190717-0
> kernel-default` could do the job.

I've uploaded it here: https://send.firefox.com/download/7c323cab824076c2/#JWT7_I9V1e6K4WX1pqbCrQ

This is my theory what is going on (it still needs further investigation):

From here on I will write kernel-1 instead of kernel-default-5.6.4-1.1.x86_64 and kernel-2 instead of kernel-default-5.6.4-1.2.x86_64 for the sake of readability.

- you have kernel-1 installed and running
- this package is not in repository any more. It was replaced by the version kernel-2
- the newer version kernel-2 obsoletes the kernel-1
- distrosync tries to install kernel-2 (the highest version available in the repos) and the `obsoletes` rule forces removal of kernel-1
- but kernel-1 cannot be removed as dnf protects currently running kernel version (this is the error message you are getting)

I see two basic approaches how to deal with this (both of them are kind of a workaround):
- either exclude kernel-default package from distro-sync
- or reboot with kernel other then 5.6.4 (according to debug data you should have kernel-default-5.6.2-1.3 installed) and then do the distrosync

In fact I'm not sure what is dnf supposed to do in situation that one kernel version obsoletes another and so they cannot be installed in parallel. I'll need to prepare test case for this and play with it a bit.

(In reply to Marek Blaha from comment #4)
> This is my theory what is going on (it still needs further investigation):
> 
> From here on I will write kernel-1 instead of
> kernel-default-5.6.4-1.1.x86_64 and kernel-2 instead of
> kernel-default-5.6.4-1.2.x86_64 for the sake of readability.
> 
> - you have kernel-1 installed and running
> - this package is not in repository any more. It was replaced by the version
> kernel-2
> - the newer version kernel-2 obsoletes the kernel-1
> - distrosync tries to install kernel-2 (the highest version available in the
> repos) and the `obsoletes` rule forces removal of kernel-1
> - but kernel-1 cannot be removed as dnf protects currently running kernel
> version (this is the error message you are getting)

That makes sense. It also explains why this issue only appears sometimes and not every time.

> 
> I see two basic approaches how to deal with this (both of them are kind of a
> workaround):
> - either exclude kernel-default package from distro-sync
> - or reboot with kernel other then 5.6.4 (according to debug data you should
> have kernel-default-5.6.2-1.3 installed) and then do the distrosync

I can work around this by upgrading the kernel with zypper and the rest with dnf, although I would prefer to use dnf for everything ;-).

> 
> In fact I'm not sure what is dnf supposed to do in situation that one kernel
> version obsoletes another and so they cannot be installed in parallel. I'll
> need to prepare test case for this and play with it a bit.

From a user's perspective it would be preferable if the new kernel just gets installed. But I really cannot judge how this could negatively affect distributions that don't behave like Tumbleweed.

(In reply to dan.cermak from comment #5)
> (In reply to Marek Blaha from comment #4)
> > This is my theory what is going on (it still needs further investigation):
> > 
> > From here on I will write kernel-1 instead of
> > kernel-default-5.6.4-1.1.x86_64 and kernel-2 instead of
> > kernel-default-5.6.4-1.2.x86_64 for the sake of readability.
> > 
> > - you have kernel-1 installed and running
> > - this package is not in repository any more. It was replaced by the version
> > kernel-2
> > - the newer version kernel-2 obsoletes the kernel-1
> > - distrosync tries to install kernel-2 (the highest version available in the
> > repos) and the `obsoletes` rule forces removal of kernel-1
> > - but kernel-1 cannot be removed as dnf protects currently running kernel
> > version (this is the error message you are getting)
> 
> That makes sense. It also explains why this issue only appears sometimes and
> not every time.
> 
> > 
> > I see two basic approaches how to deal with this (both of them are kind of a
> > workaround):
> > - either exclude kernel-default package from distro-sync
> > - or reboot with kernel other then 5.6.4 (according to debug data you should
> > have kernel-default-5.6.2-1.3 installed) and then do the distrosync
> 
> I can work around this by upgrading the kernel with zypper and the rest with
> dnf, although I would prefer to use dnf for everything ;-).
> 
> > 
> > In fact I'm not sure what is dnf supposed to do in situation that one kernel
> > version obsoletes another and so they cannot be installed in parallel. I'll
> > need to prepare test case for this and play with it a bit.
> 
> From a user's perspective it would be preferable if the new kernel just gets
> installed. But I really cannot judge how this could negatively affect
> distributions that don't behave like Tumbleweed.

So this is a little weird, because I thought that when we tell libsolv to do a dist-upgrade job like this, multiversioned packages like the kernel should be handled correctly as an install job without triggering a removal of the running kernel. We don't see the same issue when system-upgrading to a new Fedora release, and that has all the makings of the same problem, so how does this happen now?

The problem here is with the 'Obsoletes: kernel-default = <older version>` directive in newer kernel version, which is quite unusual and I do not think Fedora use it.
This prevents dnf from installing both newer and older kernel side by side.

btw, how exactly does zypper deal with upgrading the kernel?
Does it install both kernels in parallel (and ignores the obsolete). It might not be possible due to the file conflicts because both kernels are in the same version.
Or does the zypper do "classic" update of the kernel if the version of updated kernel is the same?

(In reply to Marek Blaha from comment #8)
> btw, how exactly does zypper deal with upgrading the kernel?
> Does it install both kernels in parallel (and ignores the obsolete). It
> might not be possible due to the file conflicts because both kernels are in
> the same version.
> Or does the zypper do "classic" update of the kernel if the version of
> updated kernel is the same?

I believe that's what it does. I'm not sure why the Obsoletes is there, and it probably shouldn't be present...

This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.