Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1752961

Summary: After upgrading to Fedora 31 got a kernel panic loading 5.3.0 kernel
Product: [Fedora] Fedora Reporter: Brandon Bennett <bbennett>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 31CC: abrahm.scully, airlied, alberto.patino.limon, awilliam, bskeggs, gaul, gmarr, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, jsnitsel, kernel-maint, kparal, labbott, leonid, linville, malucious81, masami256, mchehab, mjg59, mzink, pasik, pbrobinson, robatino, steved, thetaeridanus, tomek, vrutkovs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedFreezeException
Fixed In Version: kernel-5.3.2-300.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-04 20:05:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1269538, 1644939, 1644940    
Attachments:
Description Flags
Capture of the panic from a camera
none
Picture of panic w/ rhgb and quiet removed none

Description Brandon Bennett 2019-09-17 16:32:56 UTC
Created attachment 1615921 [details]
Capture of the panic from a camera

1. Please describe the problem:
After upgrading and rebooting into new Fedora 31 I received a kernel panic

On a Lenovo T480s laptop.


2. What is the Version-Release number of the kernel:

5.3.0.rc6.git0.1.fc31



3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

I was able to boot into the previous 5.2.14-200 kernel without any issues


4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

For me I just need to reboot into the 5.3.0-git kernel


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Will add in a subsequent comment once tested.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.  Panic is at the start of the boot (see screenshot)

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Panic occurs before logging.

Comment 1 Laura Abbott 2019-09-17 19:26:29 UTC
part of the panic is cut off so it's hard to tell what's going on. I suspect this is an issue with not being able to open the root file system. Can you try booting without rhgb and quiet on the command line?

Comment 2 Brandon Bennett 2019-09-17 23:00:08 UTC
Created attachment 1616025 [details]
Picture of panic w/ rhgb and quiet removed

Comment 3 Brandon Bennett 2019-09-17 23:00:29 UTC
Attached.  It seems to be before mounting the root file system.

Comment 4 Brandon Bennett 2019-09-17 23:01:49 UTC
This seems to be related: https://www.spinics.net/lists/linux-integrity/msg08544.html

Comment 5 Abrahm Scully 2019-09-18 15:48:57 UTC
This affects an XPS 13 9370 with 5.3.0-1.fc31.x86_64 and 5.3.0-0.rc6.git0.1.fc31.x86_64. Booting 5.2.14-200.fc30.x86_64 doesn't exhibit the problem.

When it panics, if I force power off and turn the laptop right back on the boot has been successful.

Comment 6 Brandon Bennett 2019-09-18 20:55:55 UTC
Upgrading to 5.3.0-1.fc31 seems to have fixed the issue for me.

Comment 7 Milan Zink 2019-09-19 10:32:34 UTC
*** Bug 1753521 has been marked as a duplicate of this bug. ***

Comment 8 Milan Zink 2019-09-19 10:35:44 UTC
I just discovered this bug. Closing mine as duplicate - https://bugzilla.redhat.com/show_bug.cgi?id=1753521.


System Information
	Manufacturer: LENOVO
	Product Name: 20L8S2N80D
	Version: ThinkPad T480s
	Wake-up Type: Power Switch
	SKU Number: LENOVO_MT_20L8_BU_Think_FM_ThinkPad T480s
	Family: ThinkPad T480s

I've tried
kernel-5.4.0-0.rc0.git2.1.fc32
kernel-5.3.0-1.fc31
kernel-5.3.0-0.rc6.git0.1.fc31

And all are the same -> kernel panic

Exactly the same as https://www.spinics.net/lists/linux-integrity/msg08544.html

5.2.15-200.fc30.x86_64 - works perfect

Comment 9 Leonid Podolny 2019-09-21 00:57:16 UTC
Same problem on T470s with 5.3.0-0.rc6.git0.1 and 5.3.0-1.


Interestingly, I managed to boot into F31 for a few times after the upgrade. This started happening a few "dnf update"s later.

Comment 10 Andrew Gaul 2019-09-21 02:12:48 UTC
Disabling Secure Boot allows my NUC8i7BEH to boot with 5.3.0-1.fc31.x86_64.

Comment 11 Bill 2019-09-22 11:11:21 UTC
I get the same problem after upgrading to fedora 31 on a Dell laptop with an Intel® Core™ i7-8650U CPU. I have noticed that it doesn't happen on every boot. Maybe 1 out of three. Happening on 5.3.0-1.fc31.x86_64 and also the previous RC kernel

Comment 12 Milan Zink 2019-09-23 10:50:14 UTC
I have a workaround. The message speaks about "TPM address not". I've disabled TPM module in BIOS and system is booting normally.

There must be some kind of regression in tpm module I assume.

Comment 13 Milan Zink 2019-09-23 10:54:18 UTC
(In reply to Milan Zink from comment #12)
> I have a workaround. The message speaks about "TPM address not". I've
> disabled TPM module in BIOS and system is booting normally.
> 
> There must be some kind of regression in tpm module I assume.

BTW: F31 Live Beta Workstation is booting fine.. Maybe it's not loading tpm module.. or not trying to decrypt LUKS device..

Comment 14 Laura Abbott 2019-09-23 14:38:15 UTC
*** Bug 1754141 has been marked as a duplicate of this bug. ***

Comment 15 Laura Abbott 2019-09-23 14:56:40 UTC
There's a proposed fix at https://lore.kernel.org/linux-efi/20190918191626.5741-1-jsnitsel@redhat.com/ . I'm waiting to see if it gets a review or ack but if not I may just bring this in by the end of the week.

Comment 16 Jerry Snitselaar 2019-09-23 17:16:01 UTC
I'll try to get a response out of matthew and jarkko again today. Maybe I should resubmit with a more dire summary message.

Comment 17 Jerry Snitselaar 2019-09-25 20:20:10 UTC
The following commits should resolve issues:

(from git://git.kernel.org/pub/scm/linux/kernel/git/efi/efi.git urgent branch)

c0e71ec75e07 | 2019-09-25 | efi/tpm: only set efi_tpm_final_log_size after successful event log parsing (Jerry Snitselaar)
1f112c0544b1 | 2019-09-25 | efi/tpm: don't traverse an event log with no events (Peter Jones)
512fb49c9e54 | 2019-09-25 | efi/tpm: Don't access event->count when it isn't mapped. (Peter Jones)

Comment 18 Bill 2019-09-27 04:09:53 UTC
The problem seems to have gone away since I installed kernel 5.3.1-300.fc31.x86_64, so I guess the above fix is included in that.

Comment 19 Milan Zink 2019-09-27 08:23:01 UTC
5.3.1-300.fc31.x86_64 is still causing the same kernel panic on Lenovo t480s with TPM enabled. No progress here.

Comment 20 Bill 2019-09-27 09:06:31 UTC
Well I posted too soon. Just had the problem again on the Dell Latitude 7490 with 5.3.1-300.fc31.x86_64. It happened on the forth reboot with that kernel.

Comment 21 Adam Williamson 2019-09-30 15:44:23 UTC
Let's propose this as a Final blocker, criterion would be "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility...A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles." That's obviously not happening on...what, one boot out of four or so?...on affected systems.

Comment 22 Geoffrey Marr 2019-10-01 00:15:21 UTC
Discussed during the 2019-09-30 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug and accept is as an AcceptedFreezeException was made as this is clearly worthy of an FE, we're not sure if it will affect enough systems to qualify as a blocker yet.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-09-30/f31-blocker-review.2019-09-30-16.00.txt

Comment 23 Kamil Páral 2019-10-01 10:41:31 UTC
I can confirm I also get kernel panic immediately during boot on Thinkpad T480s. Kernel panic in 3 out of 3 attempts. TPM is active by default on this laptop, which is also my case.

Comment 24 Alberto Patino 2019-10-02 20:12:14 UTC
Same issue here after upgrade from fedora 30 to 31 with following system (Thinkpad P51):

           /:-------------:\          albertop@polamalu
        :-------------------::        OS: Fedora 31 ThirtyOne
      :-----------/shhOHbmp---:\      Kernel: x86_64 Linux 5.2.15-200.fc30.x86_64
    /-----------omMMMNNNMMD  ---:     Uptime: 15h 33m
   :-----------sMMMMNMNMP.    ---:    Packages: 2284
  :-----------:MMMdP-------    ---\   Shell: bash 5.0.7
 ,------------:MMMd--------    ---:   Resolution: 6400x2160
 :------------:MMMd-------    .---:   DE: GNOME 
 :----    oNMMMMMMMMMNho     .----:   WM: GNOME Shell
 :--     .+shhhMMMmhhy++   .------/   WM Theme: 
 :-    -------:MMMd--------------:    GTK Theme: Adwaita-dark [GTK2/3]
 :-   --------/MMMd-------------;     Icon Theme: Adwaita
 :-    ------/hMMMy------------:      Font: Cantarell 11
 :-- :dMNdhhdNMMNo------------;       CPU: Intel Xeon E-2176M @ 12x 4.4GHz [50.0°C]
 :---:sdNMMMMNds:------------:        GPU: Quadro P2000
 :------:://:-------------::          RAM: 2532MiB / 128739MiB
 :---------------------:// 


However, same machine executed correctly yesterday when I tested kernel-5.3.2-300.fc30 from Test Day:2019-09-30 Kernel 5.3 Test Week (https://fedoramagazine.org/contribute-at-the-kernel-and-iot-edition-fedora-test-days/).

This it means that issue is fixed in 5.3.2.300? 

Thanks

Comment 25 Jerry Snitselaar 2019-10-02 20:19:38 UTC
Yes, 5.3.2-300 has the commits:

de0ec6491fe6 | 2019-10-02 | kernel-5.3.2-300.fc30 configs (Josh Boyer)
a9a469be6888 | 2019-10-02 | tpm: only set efi_tpm_final_log_size after successful event log parsing (Jerry Snitselaar)
db46d179ec97 | 2019-10-02 | efi+tpm: don't traverse an event log with no events (Peter Jones)
d896420b9798 | 2019-10-02 | efi+tpm: Don't access event->count when it isn't mapped. (Peter Jones)

Comment 26 Alberto Patino 2019-10-02 20:54:27 UTC
(In reply to Jerry Snitselaar from comment #25)
> Yes, 5.3.2-300 has the commits:
> 
> de0ec6491fe6 | 2019-10-02 | kernel-5.3.2-300.fc30 configs (Josh Boyer)
> a9a469be6888 | 2019-10-02 | tpm: only set efi_tpm_final_log_size after
> successful event log parsing (Jerry Snitselaar)
> db46d179ec97 | 2019-10-02 | efi+tpm: don't traverse an event log with no
> events (Peter Jones)
> d896420b9798 | 2019-10-02 | efi+tpm: Don't access event->count when it isn't
> mapped. (Peter Jones)

Great!. Thanks for positive feedback.

Comment 27 Fedora Update System 2019-10-02 22:09:45 UTC
FEDORA-2019-fbcb04143f has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-fbcb04143f

Comment 28 Fedora Update System 2019-10-03 03:20:07 UTC
kernel-5.3.2-300.fc31, kernel-headers-5.3.2-300.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-fbcb04143f

Comment 29 Kamil Páral 2019-10-03 08:05:09 UTC
I tested kernel-5.3.2-300.fc30 (not fc31) with my T480s and it works fine. I can confirm the fix.

Comment 30 Milan Zink 2019-10-03 09:46:56 UTC
5.3.2-300.fc31.x86_64 on t480s work fine.

Comment 31 Bill 2019-10-03 11:06:05 UTC
The 5.3.2-300.fc31.x86_64 version has fixed the problem for the Dell Latitude 7490. Ten consecutive reboots with no problem.

Comment 32 Andrew Gaul 2019-10-03 19:13:07 UTC
(In reply to Andrew Gaul from comment #10)
> Disabling Secure Boot allows my NUC8i7BEH to boot with 5.3.0-1.fc31.x86_64.

I can successfully boot with Secure Boot enabled with 5.3.2-300.fc31.x86_64.

Comment 33 Fedora Update System 2019-10-04 20:05:53 UTC
kernel-5.3.2-300.fc31, kernel-headers-5.3.2-300.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 34 Adam Williamson 2019-10-29 01:21:15 UTC
Clearing CommonBugs as we fixed this ahead of Final.