Updated the additional details with `dmesg` output, `cat /proc/sys/kernel/tainted` output, and forgot to mention my KVM switch.

I tried running kernel 6.3.6 with the additional parameter `module_blacklist=ucsi_acpi` but still go no video output. I also tried unplugging my KVM and plugging the Display Port directly from the monitor into the video card and I got display (e.g. I could see the Plymouth screen where I tyoe in  my LUKS password) when booting kernel 6.3.6, so this looks like it is related to the KVM.

I tried running kernel 6.3.7 with the additional parameter `amdgpu.dcdebugmask=0x10` but still go no video output. This bug could possibly be related to https://gitlab.freedesktop.org/drm/amd/-/issues/2610

For those of you where adding module_blacklist=ucsi_acpi to the kernel commandline helps, I believe that this is an issue which was already fixed recently and for which a patch is currently pending upstream:

https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git/commit/?h=usb-linus&id=c4a8bfabefed706bb9150867db528ceefd5cb5fe

I have submitted a merge-request to get this fix added to the Fedora kernels as a downstream patch for now:

https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2511

So that we can hopefully get this resolved for Fedora users ASAP.

FEDORA-2023-0d0eb153c9 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-0d0eb153c9

FEDORA-2023-0ea5b492c7 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-0ea5b492c7

FEDORA-2023-0ea5b492c7 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-0ea5b492c7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-0ea5b492c7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-0d0eb153c9 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-0d0eb153c9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-0d0eb153c9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-0d0eb153c9 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-0ea5b492c7 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Still no video output with any 6.3.x kernel, I've tried 6.3.4 up to 6.3.8 and they all have the same problem with no video display via the KVM switch. Still using 6.2.15 to get video output through the KVM.

Can you please re-open this ticket and change the version to F38 @hdegoede ?

This ticket seems to have been closed accidentally as CLOSED ERRATA: this issue was never solved by the module_blacklist=ucsi_acpi blacklisting but it was marked to be solved by the module_blacklist=ucsi_acpi-related update. The issue that was solved by blacklisting ucsi_acpi was the other one that was mentioned in the mailing list discussion (https://lists.fedoraproject.org/archives/search?mlist=devel%40lists.fedoraproject.org&q=module_blacklist%3Ducsi_acpi).

The user reports in ask.Fedora that the issue remains also on F38 and with all kernels up to 6.4.11. I asked the user to also test F39 and let us know. 

The user also created a ticket upstream, which is tackled by someone from AMD, who already assigned "6000 dGPU Series" label: https://gitlab.freedesktop.org/drm/amd/-/issues/2798 -> they found some indication about the bug.

The user will maybe switch to kwizard's long term supported kernels to get rid of 6.2.15 -> these are not officially supported and not officially provided by Fedora, but maybe a better compromise at this point compared to sticking with 6.2.15. If 6.1.46 does not work, it might be worth to check which patches had been added to 6.1 after it entered the LTS period. 

However, if AMD solves the issue upstream, it might be helpful for the user if the related patch gets backported to the Fedora kernel soon in order to get back to a "supported Fedora".

> Can you please re-open this ticket and change the version to F38 @hdegoede ?

Done.

Thanks for reopening this. For the past 2 months I've been trying all new F37 kernels when they came out and all of them had the same problem. I've been stuck on 6.2.15 as the last working kernel for me whilst using the KVM switch. Today I tried:
* Upgrading from F37 to F38 and kernel 6.4.11 didn't work.
* I did a fresh install of F38 and it comes with 6.2.9 and that works as expected.
* Upgraded the F38 system to the latest kernel of 6.4.11 and it didn't work again.
* Manually installed 6.2.15 into F38 and that works as expected.
* Manually installed 6.3.3 into F38 and that is broken as expected.
* Installed the F38 6.1 LTS kernel and that is working, which I did expect but was unsure if it would.
* I am trying to do a git bisect with the upstream AMD developer between kernel tag 6.2.15 and 6.3.3. I chose 6.3.3 as that was the next released F37/F38 kernel.
* I followed https://kparal.wordpress.com/2023/08/15/bisecting-fedora-kernel/ and did a build from the Linux kernel repo for 6.2.16 using Fedora Linux kernel repo and I was able to boot it, but the Plymouth service failed to load and the system couldn't boot. This is my first time doing it so maybe I am doing it wrong or maybe that kernel is broken? I did notice that kernel is deleted from Koji ( https://koji.fedoraproject.org/koji/buildinfo?buildID=2202061 )

Looking at https://koji.fedoraproject.org/koji/packageinfo?buildStart=150&packageID=8&buildOrder=-completion_time&tagOrder=name&tagStart=0#buildlist gives me

kernel-6.3.3-200.fc38 	jforbes 	2023-05-17 15:46:02 OK
kernel-6.2.16-200.fc37 	jforbes 	2023-05-17 15:07:08 DELETED
kernel-6.2.15-300.fc38 	jforbes 	2023-05-11 18:27:14 OK
kernel-6.3.2-200.fc38 	jforbes 	2023-05-11 17:38:54 DELETED
kernel-6.3.1-200.fc38 	jforbes 	2023-05-05 17:34:34 DELETED

All kernels before that for 6.3 don't seem to be for F37/F38, happy to be corrected. Basically I am trying to work out if the git bisect between 6.2.15 and 6.3.3 is good or maybe I can narrow it down.

kernel builds being deleted means that they never made it to a stable Fedora release so they are then deleted to save diskspace.

Note you can happily install e.g. a f36 or f39 kernel build on a f37 or f38 machine.

For bisects, what I usually do is first do a rpm level bisect by installing kernels from koji until I've found the first broken one, as you have already done.

You say 6.3.3 is the first bad kernel, Have you tried 6.3.0 :

https://koji.fedoraproject.org/koji/buildinfo?buildID=2191871

? Again this should work fine on f37/f38 even though it is for f39.

If we are really lucky 6.3.0 works and then we only need to bisect between 6.3.0 and 6.3.3 (were we can first pin it down to a single 6.3.y step by trying more rpms).

If 6.3.0 is confirmed to be broken, I would advice you to also try 6.2.0 hopefully that one is good and then you can do a bisect between 6.2.0 and 6.3.0.  Usually I try to limit bisects to between .0 releases for a bit quicker / cleaner bisects (or do a bisect within a stable series, e,g, between 2 6.3.y releases).

p.s.

Have you tried a 6.5-rc? release, e.g. :

https://koji.fedoraproject.org/koji/buildinfo?buildID=2277075

With a good dose of luck this might already be fixed there, saving you from having to do the bisect.

Oh thank you for the tip, I thought I was locked to kernels from that release only. e.g. So I was only trying F38 kernels with F38. I will try https://koji.fedoraproject.org/koji/buildinfo?buildID=2191871 and https://koji.fedoraproject.org/koji/buildinfo?buildID=2277075 and report back. If that doesn't go to play I'll try your bisect tips next.

So both 6.3.0 and 6.5.0-rc had the same issue. You suggested to try bisecting between 6.2.0 and 6.3.0, but isn't 6.2.0 before 6.2.15 (which is working)? I am new to all of this so it is a learning curve for me.

Yes 6.2.0 is before 6.2.15, the reason to test 6.2.0 is also working fine and then use that as the good revision for bisecting is that 6.2.0 and 6.3.0 are both part of the same source-code lineage.

Basically once 6.2.0 is released by Linusthen  development branches into 2 different directions 6.2.y is maintained by Greg KH using 6.2.0 as a base and adding only fixes. At the same time Linus starts merging new code intended for 6.3.0. So the (much simplified) history looks like this:

         6.2.0
         /   \
        /     \
      6.3.0   6.2.y
      /  \
     /    \
   6.4.0  6.3.y

Notice how 6.2.0 and 6.3.0 are only 1 step removed from each other, so git bisect only needs to check changes done on that branch (and any subsystem branches merged in of which there are a lot).  Where as 6.2.y and 6.3.y are 3 steps removed from each other making git bisect do more work.

Yeah cool that makes sense, thanks for explaining that. Okay I will install 6.2.0 and test it works and if so (it should) then I will do a bisect between 6.2.0 and 6.3.0. I will leave that for tomorrow, I'm nearly up to 10 kernels installed and tested for today :)

A couple of more questions. So when I did the first git bisect between v6.2 and v6.3 and booted the kernel that I built I wasn't able to actually boot my system entirely. Basically the flow that I saw with that kernel was BIOS -> GRUB -> Fedora logo (at a really low resolution, like 320 or 640). I am expecting BIOS -> GRUB -> Plymouth password screen -> Fedora logo -> GDM. If I press escape on the Fedora logo I can see that the Plymouth service is trying to be called but never loads and eventually times out and the system cannot boot. So my questions are:

* Should I care that I can't load the Plymouth service? My thinking is that the bug when using the KVM switch I never get any video output after GRUB. But with this git bisected kernel I can see video output although I can't actually log into the system. So should I:
  * Try to fix Plymouth and test end to end? If so, how do I get Plymouth to actual load so I can enter my password and boot?
  * Ignore Plymouth and move onto the next commit in the git bisect and keep doing that until I get a no video output after GRUB?

The steps I am doing to build the kernel are basically from https://kparal.wordpress.com/2023/08/15/bisecting-fedora-kernel/ - what I actually did is:

```
# Clone the repos
git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git # Renamed to linux-kernel
git clone https://src.fedoraproject.org/rpms/kernel # Renamed to fedora-kernel

# Start the git bisect
cd linux-kernel
git bisect start
status: waiting for both good and bad commits
git bisect bad v6.3
status: waiting for good commit(s), bad commit known
git bisect good v6.2
Bisecting: 7399 revisions left to test after this (roughly 13 steps)
[a5c95ca18a98d742d0a4a04063c32556b5b66378] Merge tag 'drm-next-2023-02-23' of git://anongit.freedesktop.org/drm/drm

# Get the current commit hash
git show HEAD

# Create a tar of the current kernel code
git archive --prefix=linux-6.2-bisect/ -o linux-6.2-bisect.tar HEAD

# Move the kernel tar into the Fedora kernel code base
mv linux-6.2-bisect.tar ../fedora-kernel/
cd ../fedora-kernel/

# Update the kernel.spec so we can build the kernel easily
vi kernel.spec
```

```
# Diff output between the Fedora kernel.spec file and my updates
diff --git a/kernel.spec b/kernel.spec
index 409879222..05c395faf 100644
--- a/kernel.spec
+++ b/kernel.spec
@@ -171,12 +171,12 @@ Summary: The Linux kernel
 #  the --with-release option overrides this setting.)
 %define debugbuildsenabled 1
 # define buildid .local
-%define specrpmversion 6.4.11
-%define specversion 6.4.11
+%define specrpmversion 6.2.1
+%define specversion 6.2.1
 %define patchversion 6.4
 %define pkgrelease 200
 %define kversion 6
-%define tarfile_release 6.4.11
+%define tarfile_release 6.2-bisect
 # This is needed to do merge window version magic
 %define patchlevel 4
 # This allows pkg_release to have configurable %%{?dist} tag
@@ -697,7 +697,7 @@ Name: %{package_name}
 License: GPLv2 and Redistributable, no modification permitted
 URL: https://www.kernel.org/
 Version: %{specrpmversion}
-Release: %{pkg_release}
+Release: %{pkg_release}.gita5c95c
 # DO NOT CHANGE THE 'ExclusiveArch' LINE TO TEMPORARILY EXCLUDE AN ARCHITECTURE BUILD.
 # SET %%nobuildarches (ABOVE) INSTEAD
 %if 0%{?fedora}
@@ -860,7 +860,7 @@ BuildRequires: tpm2-tools
 # exact git commit you can run
 #
 # xzcat -qq ${TARBALL} | git get-tar-commit-id
-Source0: linux-%{tarfile_release}.tar.xz
+Source0: linux-%{tarfile_release}.tar
```

```
# Build the kernel
fedpkg mockbuild --with baseonly --with vanilla --without debuginfo

# Install the kernel and test
cd results_kernel/6.2.1/200.fc38.gita5c95c/
dnf install ./kernel{,-core,-modules,-modules-core,-modules-extra}-6.2.1*x86_64.rpm
dracut -f
```

Again, when I chose this kernel from GRUB I do get video output but the Plymouth service is failing to load so I can't test end to end. Is there something that I am missing from those steps or doing wrong that would break Plymouth?

Can someone please have a look at my previous comment. I haven't done anymore testing since I am not sure what I am doing is good enough.

It might be worth to test if the new kernel 6.5 solves your issue.

We currently do testing days of 6.5 and so far it looks good.

I assume you have F38 at the moment with x86_64 (`uname -r` will tell :). So you can test the currently-tested 6.5.2 kernel in F38 with the following command:

`dnf update https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-core-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-core-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-extra-6.5.2-300.fc38.x86_64.rpm`

If it works, you can save a lot of time.

Thanks for the reply, unfortunately I've already tried 6.5 and it doesn't solve my issue. I tried 6.5.2 like you suggested and that didn't help either. I noted above that the last working kernel was the last 6.2 kernel and the first broken kernel was the first 6.3 kernel. I am happy to do the bisect between the 2 but as I said I can't properly test end to end. It seems that the way I built the kernel isn't successfully loading the Plymouth password screen for LUKS. On the broken kernels I never see the Plymouth password screen for LUKS so to test properly I need to find a way to build the kernel to do that. Can someone please review these steps and let me know what I am doing wrong. From the previous post:


---


The steps I am doing to build the kernel are basically from https://kparal.wordpress.com/2023/08/15/bisecting-fedora-kernel/ - what I actually did is:

```
# Clone the repos
git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git # Renamed to linux-kernel
git clone https://src.fedoraproject.org/rpms/kernel # Renamed to fedora-kernel

# Start the git bisect
cd linux-kernel
git bisect start
status: waiting for both good and bad commits
git bisect bad v6.3
status: waiting for good commit(s), bad commit known
git bisect good v6.2
Bisecting: 7399 revisions left to test after this (roughly 13 steps)
[a5c95ca18a98d742d0a4a04063c32556b5b66378] Merge tag 'drm-next-2023-02-23' of git://anongit.freedesktop.org/drm/drm

# Get the current commit hash
git show HEAD

# Create a tar of the current kernel code
git archive --prefix=linux-6.2-bisect/ -o linux-6.2-bisect.tar HEAD

# Move the kernel tar into the Fedora kernel code base
mv linux-6.2-bisect.tar ../fedora-kernel/
cd ../fedora-kernel/

# Update the kernel.spec so we can build the kernel easily
vi kernel.spec
```

```
# Diff output between the Fedora kernel.spec file and my updates
diff --git a/kernel.spec b/kernel.spec
index 409879222..05c395faf 100644
--- a/kernel.spec
+++ b/kernel.spec
@@ -171,12 +171,12 @@ Summary: The Linux kernel
 #  the --with-release option overrides this setting.)
 %define debugbuildsenabled 1
 # define buildid .local
-%define specrpmversion 6.4.11
-%define specversion 6.4.11
+%define specrpmversion 6.2.1
+%define specversion 6.2.1
 %define patchversion 6.4
 %define pkgrelease 200
 %define kversion 6
-%define tarfile_release 6.4.11
+%define tarfile_release 6.2-bisect
 # This is needed to do merge window version magic
 %define patchlevel 4
 # This allows pkg_release to have configurable %%{?dist} tag
@@ -697,7 +697,7 @@ Name: %{package_name}
 License: GPLv2 and Redistributable, no modification permitted
 URL: https://www.kernel.org/
 Version: %{specrpmversion}
-Release: %{pkg_release}
+Release: %{pkg_release}.gita5c95c
 # DO NOT CHANGE THE 'ExclusiveArch' LINE TO TEMPORARILY EXCLUDE AN ARCHITECTURE BUILD.
 # SET %%nobuildarches (ABOVE) INSTEAD
 %if 0%{?fedora}
@@ -860,7 +860,7 @@ BuildRequires: tpm2-tools
 # exact git commit you can run
 #
 # xzcat -qq ${TARBALL} | git get-tar-commit-id
-Source0: linux-%{tarfile_release}.tar.xz
+Source0: linux-%{tarfile_release}.tar
```

```
# Build the kernel
fedpkg mockbuild --with baseonly --with vanilla --without debuginfo

# Install the kernel and test
cd results_kernel/6.2.1/200.fc38.gita5c95c/
dnf install ./kernel{,-core,-modules,-modules-core,-modules-extra}-6.2.1*x86_64.rpm
dracut -f
```

Again, when I chose this kernel from GRUB I do get video output but the Plymouth service is failing to load so I can't test end to end. Is there something that I am missing from those steps or doing wrong that would break Plymouth?

There shouldn't be anything missing here, but you are testing the drm merge in that instance, it is entirely possible that a different issue is present. If it gets far enough to validate that your original issue is not reproducing, mark it good. 
Of note, It looked like you are running this system with a KVM. If you bypass the KVM and connect directly, does the current kernel (or even a 6.5.3 kernel) work?  Bisect is still the best method to find the problem, but I am curious because most AMD users are no longer reporting problems, so this system seems a bit of an oddity.

@Justin Concerning KVM: The issue of the KVM switch is just little elaborated above [0] but more deeply elaborated in the related ask.fp topic: the most relevant data from the user is in the posts [1], [2], [3], [4], [5]. 

The user tested this a few times (with official kernels - including 6.3.X and 6.4.X - but also with the unofficially-provided LTS kernels in copr from kwizard): the bug occurs when the KVM is attached, but it does not occur when the KVM is not attached. So KVM seems related to the bug. The problematic patch is from post-6.2.15 (up to and including 6.2.15 everything is fine with the KVM), whereas the problematic patch seems to be one that was not backported to 6.1 LTS (kwizard's then-current 6.1 LTS worked properly with the KVM when the user tested it in August 2023; see [5] -> the test in [5] was with 6.1.45 or 6.1.46).

@Dallas you could repeat the test (with and without KVM attached) for 6.5 just to ensure nothing has changed...

[0] https://bugzilla.redhat.com/show_bug.cgi?id=2213179#c2
[1] https://discussion.fedoraproject.org/t/black-screen-after-grub-on-amd-desktop-with-all-kernels-later-than-6-2-15-on-f37-and-f38-no-nvidia-kvm-switch-in-use/83863/25
[2] https://discussion.fedoraproject.org/t/black-screen-after-grub-on-amd-desktop-with-all-kernels-later-than-6-2-15-on-f37-and-f38-no-nvidia-kvm-switch-in-use/83863/28
[3] https://discussion.fedoraproject.org/t/black-screen-after-grub-on-amd-desktop-with-all-kernels-later-than-6-2-15-on-f37-and-f38-no-nvidia-kvm-switch-in-use/83863/31
[4] https://discussion.fedoraproject.org/t/black-screen-after-grub-on-amd-desktop-with-all-kernels-later-than-6-2-15-on-f37-and-f38-no-nvidia-kvm-switch-in-use/83863/35
[5] https://discussion.fedoraproject.org/t/black-screen-after-grub-on-amd-desktop-with-all-kernels-later-than-6-2-15-on-f37-and-f38-no-nvidia-kvm-switch-in-use/83863/38

@Justin thanks for suggestion I will keep testing the bisects and as long as I can see display where Plymouth should start I'll consider that a pass. And yes if I unplug the KVM everything works as expected, for any kernel I have tested.

@Christopher I just installed 6.5.2 via the command you gave me earlier

`dnf update https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-core-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-core-6.5.2-300.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/kernel/6.5.2/300.fc38/x86_64/kernel-modules-extra-6.5.2-300.fc38.x86_64.rpm`

and I unplugged the KVM and it worked as expected. I rebooted with the KVM and it didn't work as expected. I will continue using the 6.1 LTS kwizard kernels and go back to doing the bisects between 6.2 and 6.3.

Just an oddity that might be worth looking into.  The 6800XT is a pretty new card, it supports Displayport 1.4a and HDMI 2.1 VRR and FRL.  I am wondering if some support was added, and the monitor and video card are agreeing to a standard that the KVM can't support.  Can you check the specs on the KVM (and both cables) and see if they match?  Just trying to figure it out, as it is clearly the kvm that triggers the issue, and ideally, the kernel shouldn't even know that the KVM is there.

Okay so life got in the way of me progressing this until recently. I was going through the kernel bisects when I had time but hadn’t found the commit causing the issue yet. And then I came across https://www.youtube.com/watch?v=akxU62laPMk which led me to read https://forum.level1techs.com/t/official-l1techs-kvm-faq-ultimate-guide-help/186196 and learn a heap about KVMs. I already knew that my problem was related to the KVM since my computer worked find without it and had problems with it, so I ended up buying one from L1Tech and the problem is resolved. I am currently using 6.6.4-100.fc38.x86_64 and it is working as expected with the KVM. Thanks all for your time and help.