Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1752961 - After upgrading to Fedora 31 got a kernel panic loading 5.3.0 kernel
Summary: After upgrading to Fedora 31 got a kernel panic loading 5.3.0 kernel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 31
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
: 1753521 1754141 (view as bug list)
Depends On:
Blocks: IoT F31FinalBlocker F31FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2019-09-17 16:32 UTC by Brandon Bennett
Modified: 2019-10-29 01:21 UTC (History)
34 users (show)

Fixed In Version: kernel-5.3.2-300.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-04 20:05:53 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)
Capture of the panic from a camera (82.36 KB, image/jpeg)
2019-09-17 16:32 UTC, Brandon Bennett 		no flags Details
Picture of panic w/ rhgb and quiet removed (258.23 KB, image/jpeg)
2019-09-17 23:00 UTC, Brandon Bennett 		no flags Details
View All

Description Brandon Bennett 2019-09-17 16:32:56 UTC 
Created attachment 1615921 [details]
Capture of the panic from a camera

1. Please describe the problem:
After upgrading and rebooting into new Fedora 31 I received a kernel panic

On a Lenovo T480s laptop.


2. What is the Version-Release number of the kernel:

5.3.0.rc6.git0.1.fc31



3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

I was able to boot into the previous 5.2.14-200 kernel without any issues


4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

For me I just need to reboot into the 5.3.0-git kernel


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Will add in a subsequent comment once tested.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.  Panic is at the start of the boot (see screenshot)

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Panic occurs before logging.
Comment 1 Laura Abbott 2019-09-17 19:26:29 UTC 
part of the panic is cut off so it's hard to tell what's going on. I suspect this is an issue with not being able to open the root file system. Can you try booting without rhgb and quiet on the command line?
Comment 2 Brandon Bennett 2019-09-17 23:00:08 UTC 
Created attachment 1616025 [details]
Picture of panic w/ rhgb and quiet removed
Comment 3 Brandon Bennett 2019-09-17 23:00:29 UTC 
Attached.  It seems to be before mounting the root file system.
Comment 4 Brandon Bennett 2019-09-17 23:01:49 UTC 
This seems to be related: https://www.spinics.net/lists/linux-integrity/msg08544.html
Comment 5 Abrahm Scully 2019-09-18 15:48:57 UTC 
This affects an XPS 13 9370 with 5.3.0-1.fc31.x86_64 and 5.3.0-0.rc6.git0.1.fc31.x86_64. Booting 5.2.14-200.fc30.x86_64 doesn't exhibit the problem.

When it panics, if I force power off and turn the laptop right back on the boot has been successful.
Comment 6 Brandon Bennett 2019-09-18 20:55:55 UTC 
Upgrading to 5.3.0-1.fc31 seems to have fixed the issue for me.
Comment 7 Milan Zink 2019-09-19 10:32:34 UTC 
*** Bug 1753521 has been marked as a duplicate of this bug. ***
Comment 8 Milan Zink 2019-09-19 10:35:44 UTC 
I just discovered this bug. Closing mine as duplicate - https://bugzilla.redhat.com/show_bug.cgi?id=1753521.


System Information
	Manufacturer: LENOVO
	Product Name: 20L8S2N80D
	Version: ThinkPad T480s
	Wake-up Type: Power Switch
	SKU Number: LENOVO_MT_20L8_BU_Think_FM_ThinkPad T480s
	Family: ThinkPad T480s

I've tried
kernel-5.4.0-0.rc0.git2.1.fc32
kernel-5.3.0-1.fc31
kernel-5.3.0-0.rc6.git0.1.fc31

And all are the same -> kernel panic

Exactly the same as https://www.spinics.net/lists/linux-integrity/msg08544.html

5.2.15-200.fc30.x86_64 - works perfect
Comment 9 Leonid Podolny 2019-09-21 00:57:16 UTC 
Same problem on T470s with 5.3.0-0.rc6.git0.1 and 5.3.0-1.


Interestingly, I managed to boot into F31 for a few times after the upgrade. This started happening a few "dnf update"s later.
Comment 10 Andrew Gaul 2019-09-21 02:12:48 UTC 
Disabling Secure Boot allows my NUC8i7BEH to boot with 5.3.0-1.fc31.x86_64.
Comment 11 Bill 2019-09-22 11:11:21 UTC 
I get the same problem after upgrading to fedora 31 on a Dell laptop with an Intel® Core™ i7-8650U CPU. I have noticed that it doesn't happen on every boot. Maybe 1 out of three. Happening on 5.3.0-1.fc31.x86_64 and also the previous RC kernel
Comment 12 Milan Zink 2019-09-23 10:50:14 UTC 
I have a workaround. The message speaks about "TPM address not". I've disabled TPM module in BIOS and system is booting normally.

There must be some kind of regression in tpm module I assume.
Comment 13 Milan Zink 2019-09-23 10:54:18 UTC 
(In reply to Milan Zink from comment #12)
> I have a workaround. The message speaks about "TPM address not". I've
> disabled TPM module in BIOS and system is booting normally.
> 
> There must be some kind of regression in tpm module I assume.

BTW: F31 Live Beta Workstation is booting fine.. Maybe it's not loading tpm module.. or not trying to decrypt LUKS device..
Comment 14 Laura Abbott 2019-09-23 14:38:15 UTC 
*** Bug 1754141 has been marked as a duplicate of this bug. ***
Comment 15 Laura Abbott 2019-09-23 14:56:40 UTC 
There's a proposed fix at https://lore.kernel.org/linux-efi/20190918191626.5741-1-jsnitsel@redhat.com/ . I'm waiting to see if it gets a review or ack but if not I may just bring this in by the end of the week.
Comment 16 Jerry Snitselaar 2019-09-23 17:16:01 UTC 
I'll try to get a response out of matthew and jarkko again today. Maybe I should resubmit with a more dire summary message.
Comment 17 Jerry Snitselaar 2019-09-25 20:20:10 UTC 
The following commits should resolve issues:

(from git://git.kernel.org/pub/scm/linux/kernel/git/efi/efi.git urgent branch)

c0e71ec75e07 | 2019-09-25 | efi/tpm: only set efi_tpm_final_log_size after successful event log parsing (Jerry Snitselaar)
1f112c0544b1 | 2019-09-25 | efi/tpm: don't traverse an event log with no events (Peter Jones)
512fb49c9e54 | 2019-09-25 | efi/tpm: Don't access event->count when it isn't mapped. (Peter Jones)
Comment 18 Bill 2019-09-27 04:09:53 UTC 
The problem seems to have gone away since I installed kernel 5.3.1-300.fc31.x86_64, so I guess the above fix is included in that.
Comment 19 Milan Zink 2019-09-27 08:23:01 UTC 
5.3.1-300.fc31.x86_64 is still causing the same kernel panic on Lenovo t480s with TPM enabled. No progress here.
Comment 20 Bill 2019-09-27 09:06:31 UTC 
Well I posted too soon. Just had the problem again on the Dell Latitude 7490 with 5.3.1-300.fc31.x86_64. It happened on the forth reboot with that kernel.
Comment 21 Adam Williamson 2019-09-30 15:44:23 UTC 
Let's propose this as a Final blocker, criterion would be "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility...A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles." That's obviously not happening on...what, one boot out of four or so?...on affected systems.
Comment 22 Geoffrey Marr 2019-10-01 00:15:21 UTC 
Discussed during the 2019-09-30 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug and accept is as an AcceptedFreezeException was made as this is clearly worthy of an FE, we're not sure if it will affect enough systems to qualify as a blocker yet.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-09-30/f31-blocker-review.2019-09-30-16.00.txt
Comment 23 Kamil Páral 2019-10-01 10:41:31 UTC 
I can confirm I also get kernel panic immediately during boot on Thinkpad T480s. Kernel panic in 3 out of 3 attempts. TPM is active by default on this laptop, which is also my case.
Comment 24 Alberto Patino 2019-10-02 20:12:14 UTC 
Same issue here after upgrade from fedora 30 to 31 with following system (Thinkpad P51):

           /:-------------:\          albertop@polamalu
        :-------------------::        OS: Fedora 31 ThirtyOne
      :-----------/shhOHbmp---:\      Kernel: x86_64 Linux 5.2.15-200.fc30.x86_64
    /-----------omMMMNNNMMD  ---:     Uptime: 15h 33m
   :-----------sMMMMNMNMP.    ---:    Packages: 2284
  :-----------:MMMdP-------    ---\   Shell: bash 5.0.7
 ,------------:MMMd--------    ---:   Resolution: 6400x2160
 :------------:MMMd-------    .---:   DE: GNOME 
 :----    oNMMMMMMMMMNho     .----:   WM: GNOME Shell
 :--     .+shhhMMMmhhy++   .------/   WM Theme: 
 :-    -------:MMMd--------------:    GTK Theme: Adwaita-dark [GTK2/3]
 :-   --------/MMMd-------------;     Icon Theme: Adwaita
 :-    ------/hMMMy------------:      Font: Cantarell 11
 :-- :dMNdhhdNMMNo------------;       CPU: Intel Xeon E-2176M @ 12x 4.4GHz [50.0°C]
 :---:sdNMMMMNds:------------:        GPU: Quadro P2000
 :------:://:-------------::          RAM: 2532MiB / 128739MiB
 :---------------------:// 


However, same machine executed correctly yesterday when I tested kernel-5.3.2-300.fc30 from Test Day:2019-09-30 Kernel 5.3 Test Week (https://fedoramagazine.org/contribute-at-the-kernel-and-iot-edition-fedora-test-days/).

This it means that issue is fixed in 5.3.2.300? 

Thanks
Comment 25 Jerry Snitselaar 2019-10-02 20:19:38 UTC 
Yes, 5.3.2-300 has the commits:

de0ec6491fe6 | 2019-10-02 | kernel-5.3.2-300.fc30 configs (Josh Boyer)
a9a469be6888 | 2019-10-02 | tpm: only set efi_tpm_final_log_size after successful event log parsing (Jerry Snitselaar)
db46d179ec97 | 2019-10-02 | efi+tpm: don't traverse an event log with no events (Peter Jones)
d896420b9798 | 2019-10-02 | efi+tpm: Don't access event->count when it isn't mapped. (Peter Jones)
Comment 26 Alberto Patino 2019-10-02 20:54:27 UTC 
(In reply to Jerry Snitselaar from comment #25)
> Yes, 5.3.2-300 has the commits:
> 
> de0ec6491fe6 | 2019-10-02 | kernel-5.3.2-300.fc30 configs (Josh Boyer)
> a9a469be6888 | 2019-10-02 | tpm: only set efi_tpm_final_log_size after
> successful event log parsing (Jerry Snitselaar)
> db46d179ec97 | 2019-10-02 | efi+tpm: don't traverse an event log with no
> events (Peter Jones)
> d896420b9798 | 2019-10-02 | efi+tpm: Don't access event->count when it isn't
> mapped. (Peter Jones)

Great!. Thanks for positive feedback.
Comment 27 Fedora Update System 2019-10-02 22:09:45 UTC 
FEDORA-2019-fbcb04143f has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-fbcb04143f
Comment 28 Fedora Update System 2019-10-03 03:20:07 UTC 
kernel-5.3.2-300.fc31, kernel-headers-5.3.2-300.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-fbcb04143f
Comment 29 Kamil Páral 2019-10-03 08:05:09 UTC 
I tested kernel-5.3.2-300.fc30 (not fc31) with my T480s and it works fine. I can confirm the fix.
Comment 30 Milan Zink 2019-10-03 09:46:56 UTC 
5.3.2-300.fc31.x86_64 on t480s work fine.
Comment 31 Bill 2019-10-03 11:06:05 UTC 
The 5.3.2-300.fc31.x86_64 version has fixed the problem for the Dell Latitude 7490. Ten consecutive reboots with no problem.
Comment 32 Andrew Gaul 2019-10-03 19:13:07 UTC 
(In reply to Andrew Gaul from comment #10)
> Disabling Secure Boot allows my NUC8i7BEH to boot with 5.3.0-1.fc31.x86_64.

I can successfully boot with Secure Boot enabled with 5.3.2-300.fc31.x86_64.
Comment 33 Fedora Update System 2019-10-04 20:05:53 UTC 
kernel-5.3.2-300.fc31, kernel-headers-5.3.2-300.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.
Comment 34 Adam Williamson 2019-10-29 01:21:15 UTC 
Clearing CommonBugs as we fixed this ahead of Final.
Note You need to log in before you can comment on or make changes to this bug.