Created attachment 1755136 [details]
Example of the glitches I'm seeing

Created attachment 1755137 [details]
Another glitch example

Menu contents looks displaced

5.10.9 and later have some changes to security mitigations for CVE-2019-14615:
https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00314.html

Of note there is:" Platforms based on Ivy Bridge, Bay Trail and Haswell do not have full mitigations at this time for the Windows* OS." This also applies / applied to Linux, on these so called Gen 7 graphics, implementing a workaround for the CVE is tricky.

5.10.9 is the first kernel to enable the mitigations on all Ivy Bridge, Bay Trail and Haswell models, so that might very well be what is causing this.

You can disable the mitigations by adding the following to the kernel commandline:

i915.mitigations=off

You can do this by running the following command:

sudo grubby --update-kernel=ALL --args="i915.mitigations=off"

Please let us know if this works-around the problem. If it does then I will report this to the upstream i915 developers and ask them to look into this.

I am facing similar issue, but with occasional black screens.
I am on the same kernel versions as op.

sudo grubby --update-kernel=ALL --args="i915.mitigations=off" works as a workaround.

(In reply to dykow from comment #4)
> I am facing similar issue, but with occasional black screens.
> I am on the same kernel versions as op.
> 
> sudo grubby --update-kernel=ALL --args="i915.mitigations=off" works as a
> workaround.

Thank you for testing, can you please file a bug with the upstream i915 devs for this ? :

https://gitlab.freedesktop.org/drm/intel/-/wikis/How-to-file-i915-bugs

Ignore the bit about testing with the drm-tip and put something like "5.10.9 regression" in the subject of the bug.

Jesús, if you can confirm that i915.mitigations=off also fixes things for you then please also file a bug for this upstream.

dykow, I just noticed that the 5.10.13 kernel has a fix for the new mitigation stuff. The 5.10.13 kernel is in updates-testnig now. Can you please give the 5.10.13 kernel a try:

sudo dnf --enablerepo updates-testing 'kernel*'
sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"

And then boot into the new kernel and do:

cat /proc/cmdline

And check that i915.mitigations=off is NOT there and then test things.


Note if the latest kernel does not fix things you can re-enable the workaround by running:

sudo grubby --update-kernel=ALL --args="i915.mitigations=off"

Again.

Jesús, can you also please test with 5.10.13 both with and without the workaround ?

Yes, disabling the mitigations worked fine for me. I haven't seen screen glitches anymore.

Now I'm going to install kernel 5.10.13 to test it with and without the workaround.

I've been seeing screen glitches as well on my Thinkpad T430s [ Intel HD Graphics 4000 + NVIDIA Corporation GF117M]. This issue comes with newer kernels up to 5.10.14-200.fc33

The kernel that works [that I fall back to] is: 5.10.8-200.fc33

And I see this primarily with firefox [have seen this ocassionally with gnome3 top bar aswell]

I haven't yet tried disabling the mitigations.

Kernel 5.10.13-200 seems to work properly on my T430

(In reply to Hans de Goede from comment #6)

> sudo dnf --enablerepo updates-testing 'kernel*'
> sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"
> 

I changed the dnf line because it had an error. (lacked an equal sign)

I installed the 5.10.13 kernel and disabled the workaround:


sudo dnf update --enablerepo=updates-testing 'kernel*'
sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"


Now, if I boot into the 5.10.13 kernel without the workaround, screen glitches take more time before they appear, they look different than before, but they still appear after some hours.

If I apply the workaround, disabling the mitigations:


sudo grubby --update-kernel=ALL --args="i915.mitigations=off"


And then I boot into 5.10.13 kernel, I don't see glitches anymore, even after hours of using the computer.

(In reply to dykow from comment #9)
> Kernel 5.10.13-200 seems to work properly on my T430

EDIT screen glitches are still there, but different and they appear later.

Hi All,

Thank you all very much for all the testing you are doing. Now that it has been confirmed that this is not fixed in 5.10.13, I've reported this problem to the upstream developers. I'll get back to you when I hear back from them.

In the mean time you can keep using:

sudo grubby --update-kernel=ALL --args="i915.mitigations=off"

As a workaround.

Regards,

Hans

ATM the plan to fix this for future 5.10.y releases is to revert the three patches which are likely causing this problem.

I've created a scratch/test kernel build with the three patches reverted:

https://koji.fedoraproject.org/koji/taskinfo?taskID=61666542

Here are some generic instructions for installing a kernel directly from koji:

https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

After following the generic instructions to install the test kernel-build, please run:

sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"

And then boot into the new kernel and do:

cat /proc/cmdline

And check that i915.mitigations=off is NOT there and then test things.

Once it is sure that reverting the 3 patches help I will let the Greg KH (the stable kernel series maintainer) now about this and then we will see from there.

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.14-200.bz1925346.fc33.x86_64 root=UUID=a7c35b40-18d5-477e-8047-16b9b78b6111 ro resume=UUID=dc581e65-3727-4724-aae5-182856f6bed6 rhgb quiet

Looks good [with my limited testing]. will report if the issue reappears.

(In reply to Hans de Goede from comment #13)
> ATM the plan to fix this for future 5.10.y releases is to revert the three
> patches which are likely causing this problem.
> 
> I've created a scratch/test kernel build with the three patches reverted:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=61666542
> 

I installed your 5.10.14-200.bz1925346.fc33.x86_64 kernel, and disbled the workaround:


sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"


This is the output of cat /proc/cmdline:


BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.14-200.bz1925346.fc33.x86_64 root=/dev/mapper/fedora_sol--invictus-root ro rd.lvm.lv=fedora_sol-invictus/root rd.lvm.lv=fedora_sol-invictus/swap rhgb quiet


I haven't seen glitches anymore. Everything is working fine since I installed this kernel yesterday.

Thank you all for testing the test-build with the reverts.

Chris Wilson, one of the upstream Intel i915 driver developers has been working on a set of fixes for this, because as you can hopefully understand disabling the mitigation for the security issue again (which the reverts do on some models) is undesirable.

Chris has created a set of 3 patches which hopefully fixes the glitches, while leaving the security mitigations in place.

I've started a new kernel test-build of the .5.10.15 kernel with the 3 patches from Chris added:

https://koji.fedoraproject.org/koji/taskinfo?taskID=61751956

If you can give this a try, that would be great.

As usual here are some generic instructions for installing a kernel directly from koji:

https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

After following the generic instructions to install the test kernel-build, please run:

sudo grubby --update-kernel=ALL --remove-args="i915.mitigations=off"

And then boot into the new kernel and do:

cat /proc/cmdline

And check that i915.mitigations=off is NOT there and then test things.

Sorry for asking for yet more tests, but resolving this issue properly is a bit tricky and we are unable to reproduce this locally.

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.15-200.bz1925346.fc33.x86_64 root=UUID=a7c35b40-18d5-477e-8047-16b9b78b6111 ro resume=UUID=dc581e65-3727-4724-aae5-182856f6bed6 rhgb quiet

Looks good so far.

I have been using the kernel with fixes (5.10.15-200.bz1925346.fc33.x86_64) for two days, and haven't seen glitches anymore. Everything is working fine for me.

This is the output for cat /proc/cmdline:

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.15-200.bz1925346.fc33.x86_64 root=/dev/mapper/fedora_sol--invictus-root ro rd.lvm.lv=fedora_sol-invictus/root rd.lvm.lv=fedora_sol-invictus/swap rhgb quiet

I'm glad to help fixing an issue that affects myself and others with similar hardware, so this is no problem for me.

Satish, Jesús,

Thank you for testing the 5.10.15 build with the extra patches.

I've reported to the upstream i915 developers, that the 3 extra patches in that build seem to fix the issue. So hopefully they will be included in a future 5.10.y build soon.

Hello, I also have this issue which started to happen after the 5.10.9 (including 5.10.9) update. The workaround "echo off | sudo tee /sys/module/i915/parameters/mitigations" hugely fixes the heavy glitches but still have some minor black squares popping up while scrolling on Firefox. I wanted to test the 5.10.15-200.bz1925346.fc33.x86_64 edition with patches but unfortunately it has the same black squares without me turning off the mitigations.

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.15-200.bz1925346.fc33.x86_64 root=UUID=29ee4323-9de8-4771-9acc-c122e93a24fd ro rootflags=subvol=root rd.luks.uuid=luks-d1d630a7-0fb2-45a1-b871-2e148cb38a44 rhgb quiet

Here is a video of it:
https://streamable.com/k0u715
(see top right of the screen under the extensions the white area) 

I use a thinkpad x230 with i5-3320M (HD Graphics 4000)

(In reply to doubtee from comment #20)
> Hello, I also have this issue which started to happen after the 5.10.9
> (including 5.10.9) update. The workaround "echo off | sudo tee
> /sys/module/i915/parameters/mitigations" hugely fixes the heavy glitches but
> still have some minor black squares popping up while scrolling on Firefox. I
> wanted to test the 5.10.15-200.bz1925346.fc33.x86_64 edition with patches
> but unfortunately it has the same black squares without me turning off the
> mitigations.

Thank you for you report. I also have a 5.10.14 kernel build which reverts the 3 patches which went into 5.10.9 / 5.10.10 which cause this. Can you try if that build does fix the black squares thing? You can find this here:

https://koji.fedoraproject.org/koji/taskinfo?taskID=61666542

Thanks Hans, using it for a few hours now and I don't see those black squares anymore on 5.10.14-200.bz1925346.fc33.x86_64. Will keep using it and report back if I see something weird but I can say that reverting the patches worked.

(In reply to doubtee from comment #22)
> Thanks Hans, using it for a few hours now and I don't see those black
> squares anymore on 5.10.14-200.bz1925346.fc33.x86_64. Will keep using it and
> report back if I see something weird but I can say that reverting the
> patches worked.

Ok, thank you for testing and reporting this. I've forwarded the info about this problem to the upstream i915 devs.

Been using 5.10.14-200.bz1925346.fc33.x86_64 since the morning and I can say that the black squares and glitches are fixed. However something awkward happened but not sure if this is even related to this kernel but since I was running this kernel it would be great to note it down here if it ever happens again. So basically when I was browsing the web a sudden reboot happened out of nowhere. I tried to check journalctl but there wasn't anything useful and now a few hours has passed since that happened. Was weird but as I said this may not be related to this issue. 

Here is a little snippet (from bottom to top):
Feb 18 22:22:38 localhost.localdomain kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Feb 18 22:22:38 localhost.localdomain kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Feb 18 22:22:38 localhost.localdomain kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Feb 18 22:22:38 localhost.localdomain kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Feb 18 22:22:38 localhost.localdomain kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Feb 18 22:22:38 localhost.localdomain kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.14-200.bz1925346.fc33.x86_64 root=UUID=29ee4323-9de8-47>
Feb 18 22:22:38 localhost.localdomain kernel: Linux version 5.10.14-200.bz1925346.fc33.x86_64 (mockbuild.fedoraproject.org) (gcc (>
-- Reboot --
Feb 18 22:22:16 localhost.localdomain gnome-shell[1946]: meta_window_set_stack_position_no_sync: assertion 'window->stack_position >= 0' failed
Feb 18 22:22:09 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sy>
Feb 18 22:22:09 localhost.localdomain systemd[1]: Started Hostname Service.
Feb 18 22:22:09 localhost.localdomain systemd[1]: Starting Hostname Service...
Feb 18 22:22:09 localhost.localdomain audit: BPF prog-id=102 op=LOAD
Feb 18 22:22:09 localhost.localdomain audit: BPF prog-id=101 op=LOAD
Feb 18 22:22:08 localhost.localdomain systemd[1803]: Started dbus-:1.2-org.gnome.Nautilus.
Feb 18 22:22:02 localhost.localdomain audit: BPF prog-id=99 op=UNLOAD
Feb 18 22:22:02 localhost.localdomain audit: BPF prog-id=100 op=UNLOAD
Feb 18 22:22:02 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sys>
Feb 18 22:22:02 localhost.localdomain systemd[1]: systemd-hostnamed.service: Succeeded.
Feb 18 22:21:54 localhost.localdomain rtkit-daemon[967]: Supervising 7 threads of 4 processes of 1 users.
Feb 18 22:21:54 localhost.localdomain rtkit-daemon[967]: Supervising 7 threads of 4 processes of 1 users.
Feb 18 22:21:32 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sy>
Feb 18 22:21:32 localhost.localdomain systemd[1]: Started Hostname Service.
Feb 18 22:21:32 localhost.localdomain systemd[1]: Starting Hostname Service...
Feb 18 22:21:32 localhost.localdomain audit: BPF prog-id=100 op=LOAD
Feb 18 22:21:32 localhost.localdomain audit: BPF prog-id=99 op=LOAD

I can confirm that Kernel 5.10.15-200.bz1925346.fc33.x86_64 fixed the issue for me.  No more glitches.  Thanks!

I am on 5.10.16 and the issue is still there. Noticed it while using Firefox (xorg).

(In reply to Hans de Goede from comment #13)

Instructions from comment 13 fixes this issue for me:

# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.14-200.bz1925346.fc33.x86_64 root=/dev/mapper/sys-root ro rd.luks.uuid=luks-4b84d320-78be-4813-ba9d-69311e6a3df3 rd.lvm.lv=sys/root rhgb quiet


Graphics card detail:
# lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller (rev 09)
Intel® HD Graphics 4000 (IVB GT2) (From Gnome About Dialog)

Dear all, Hans de Goede,

I have a setup for kernel compiling and would like to try the three patches, but cannot really find where to get the right three.
Could you be so kind to point me to the right place where to download the three patches I need?

(My kernel should be signed for my BIOS, so I figure precompiled koji kernels do not help me here.)

Greetings,

Arjen

*** Bug 1922511 has been marked as a duplicate of this bug. ***

*** Bug 1925903 has been marked as a duplicate of this bug. ***

*** Bug 1931065 has been marked as a duplicate of this bug. ***

(In reply to Arjen P. de Vries from comment #28)
> I have a setup for kernel compiling and would like to try the three patches,
> but cannot really find where to get the right three.
> Could you be so kind to point me to the right place where to download the
> three patches I need?

This should be fixed by adding the following 3 commits from the upstream drm-intel-next branch:

e627d5923cae ("drm/i915/gt: One more flush for Baytrail clear residuals")
d30bbd62b1bf ("drm/i915/gt: Flush before changing register state")
1914911f4aa0 ("drm/i915/gt: Correct surface base address for renderclear")

I'll also attach the patch which I'm using for the test-kernels here, that combines the 3 commits into a single patch file.

Created attachment 1759255 [details]
[PATCH] 3 i915 fixes from -next combined in a single patch

Do you have any idea when this will get into production?  I accidentally installed 5.10.17-200.fc33.x86_64 and found that it seems to work a little better than the last production kernel, 5.10.16-200.fc33.x86_64, but is still not right.

Hans, I also followed the instructions at Comment 13 and all my reported problems are solved. Many thanks!

Hans:

Installed test candidate per comment 16 on one of my machines which have been exhibiting the problem using 5.10.9 through 5.10.17. Have not seen any issues with test candidate over reasonable time frame.

$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt6)/vmlinuz-5.10.15-200.bz1925346.fc33.x86_64 root=/dev/mapper/fedora_sam1-root ro rd.md=0 rd.dm=0 vconsole.keymap=us rd.lvm.lv=fedora_sam1/swap rd.luks=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora_sam1/root rhgb quiet resume=/dev/mapper/fedora_sam1-swap

excerpts from lshw:
product: Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz
product: 3rd Gen Core processor Graphics Controller
         version: 09
         configuration: driver=i915

test method after installation:

1. start konsole window and run clamscan as root to push system load above 1.5
2. view security camera video via chrome
3. use firefox to view BBC REEL videos continuously in a partial sized screen window, not full screen. Leave this window visible to stress GPU.
4. in another konsole, watch system load is between 2.5 and 3

No black blinking of KDE plasma tool bar, no distortion of icons and background with mouse movement. No display issues with any of the windows.

I believe this fixes the issues I've seen.

thanks!

FEDORA-2021-79396b21b2 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-79396b21b2

(In reply to Hans de Goede from comment #33)
> Created attachment 1759255 [details]
> [PATCH] 3 i915 fixes from -next combined in a single patch

Thanks Hans, I managed to apply the patch and problems are gone!

A brief write-up of the process of patching the FC33 kernel for those who cannot wait for the update to be processed:
https://arjenp.dev/2021/02/26/kernel-patch.html

I have got the latest kernel from the Koji build (kernel-5.10.19-200.fc33) and the patches work fine for the heavy glitches etc. (which makes everything usable again) but I still encounter the little black glitches on Firefox. Here is a new video about this: https://streamable.com/6y41it (top right of the video the glitches can be noticed right under the extensions) 

5.10.14-200.bz1925346.fc33.x86_64 worked fine but that one entirely reverted the newly introduced mitigation commits where kernel-5.10.19-200.fc33 patched the issues with the new mitigation work. At this point I am not sure if this a kernel issue anymore maybe its now something that Firefox should fix to match up with these new mitigation commits. 

The reason I am writing here is that I am curious if anyone else notices or have these little black glitches or some minor weird glitches on Firefox other than me (feeling kinda alone :P). I am sure there are lots of people with similar/same hardware I would be really happy if I can get some feedback here before I am going to open an issue on Firefox Bugzilla. I use an ThinkPad x230. (Intel® HD Graphics 4000 (IVB GT2))

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.10.19-200.fc33.x86_64 root=UUID=a7c35b40-18d5-477e-8047-16b9b78b6111 ro resume=UUID=dc581e65-3727-4724-aae5-182856f6bed6 rhgb quiet

This works fine for me!

Wrt "little black glitches" - I haven't noticed this on my ThinkpadT430s [with i7-3520M - i.e the same  Intel® HD Graphics 4000, along with NVIDIA GF117M - (i915,nouveau)]

FEDORA-2021-79396b21b2 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-79396b21b2`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-79396b21b2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-cf0236f45f has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cf0236f45f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-cf0236f45f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

I have no account on Bodhi, so am reporting here.  An attempt to install the kernel update produced this message:
> sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cf0236f45f
> [sudo] password for jonrysh: 
> Fedora 33 - x86_64 - Test Updates                                                                                                                            > 1.6 MB/s |  12 MB     00:07    
> Last metadata expiration check: 0:00:06 ago on Sat 27 Feb 2021 09:53:13 AM PST.
> No security updates needed, but 64 updates available
> Dependencies resolved.
> Nothing to do.
> Complete!
It looks like either the update failed or is already in the mainstream.  I can't find anything that looks like a this update in the latest kernel log at https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.11.2, so the situation remains a mystery.  Please enlighten me.

I have installed the latest kernel, /boot/vmlinuz-5.10.18-200.fc33.x86_64, which still has the bug.  How do install the kernel from testing?

(In reply to Jonathan Ryshpan from comment #43)
> ...
> It looks like either the update failed or is already in the mainstream. ...

(In reply to Jonathan Ryshpan from comment #43)
> I have no account on Bodhi, so am reporting here.  An attempt to install the
> kernel update produced this message:
> > sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cf0236f45f
> > [sudo] password for jonrysh: 
> > Fedora 33 - x86_64 - Test Updates                                                                                                                            > 1.6 MB/s |  12 MB     00:07    
> > Last metadata expiration check: 0:00:06 ago on Sat 27 Feb 2021 09:53:13 AM PST.
> > No security updates needed, but 64 updates available
> > Dependencies resolved.
> > Nothing to do.
> > Complete!
> It looks like either the update failed or is already in the mainstream.  I
> can't find anything that looks like a this update in the latest kernel log
> at https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.11.2, so the
> situation remains a mystery.  Please enlighten me.

Looks like you use the FC32 advisory on a FC33 system.
You need `--advisory=FEDORA-2021-79396b21b2` I think.

FEDORA-2021-79396b21b2 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2021-cf0236f45f has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

No problems with vmlinuz-5.10.19-200.fc33.x86_64 after a full day of use.  Congrats.