Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1365917 - kernel panic at boot - x2apic_cluster_probe+0x33/0x70
Summary: kernel panic at boot - x2apic_cluster_probe+0x33/0x70
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeException
: 1367396 (view as bug list)
Depends On:
Blocks: F25AlphaBlocker F25AlphaFreezeException
TreeView+ depends on / blocked
 
Reported: 2016-08-10 13:48 UTC by Peter Gervase
Modified: 2016-08-26 11:33 UTC (History)
15 users (show)

Fixed In Version: kernel-4.8.0-0.rc2.git3.1.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-22 22:07:58 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
screen shot showing the panic (3.62 MB, image/jpeg)
2016-08-10 13:48 UTC, Peter Gervase
no flags Details
screen shot showing the new panic message two minutes after the first one (4.03 MB, image/jpeg)
2016-08-10 13:49 UTC, Peter Gervase
no flags Details

Description Peter Gervase 2016-08-10 13:48:34 UTC
Created attachment 1189631 [details]
screen shot showing the panic

Description of problem:
Using either of 
kernel-4.8.0-0.rc0.git3.1.fc26.x86_64
kernel-4.8.0-0.rc0.git5.1.fc26.x86_64
I get the kernel panic in the attached screen shot IMAG0874 when I boot up.

At boot, I'll select the kernel and then in just a few seconds, it'll panic. I let it go for two minutes and it then gave the panic in the 0875 picture. I am able to boot to 
kernel-4.8.0-0.rc1.git0.1.fc26.x86_64

I tried booting to init 1, but that didn't work any better. 
No core file gets created.

Version-Release number of selected component (if applicable):
kernel-4.8.0-0.rc0.git3.1.fc26.x86_64
kernel-4.8.0-0.rc0.git5.1.fc26.x86_64

How reproducible:
100%

Steps to Reproduce:
1. reboot to one of those kernels
2.
3.

Actual results:
panic as shown

Expected results:
no panic

Additional info:
https://bugzilla.kernel.org/show_bug.cgi?id=151311 looks close

Comment 1 Peter Gervase 2016-08-10 13:49:58 UTC
Created attachment 1189632 [details]
screen shot showing the new panic message two minutes after the first one

Comment 2 Laura Abbott 2016-08-11 16:18:45 UTC
Can you test the following scratch build? It contains a probable fix from the upstream developers http://koji.fedoraproject.org/koji/taskinfo?taskID=15217136

Comment 3 Petr Lautrbach 2016-08-16 10:55:54 UTC
I saw the same/similar kernel panic problem on my Lenovo x240 with kernel-4.8.0-0.rc1.git3.1.fc25.x86_64

http://koji.fedoraproject.org/koji/taskinfo?taskID=15217136 build fixes it. Thanks!

Comment 4 Laura Abbott 2016-08-16 14:35:54 UTC
*** Bug 1367396 has been marked as a duplicate of this bug. ***

Comment 5 Petr Schindler 2016-08-17 10:15:11 UTC
kernel from koji build linked in comment 3 works for me. System boots normally with it.

I tested it with Fedora 24 with kernel-4.8.0-0.rc0.git3.1.fc25.x86_64 installed and it didn't boot (kernel panic). Then I installed kernel from koji and it booted normally.

Comment 6 Adam Williamson 2016-08-17 19:07:15 UTC
Can you confirm that kernel-4.8.0-0.rc1.git0.1.fc25 does not display this behaviour?

Comment 7 Adam Williamson 2016-08-17 19:48:28 UTC
for blocker / release engineering purposes: labbott states she's certain that kernel-4.8.0-0.rc1.git0.1.fc25 - which is the current 'stable' f25 kernel build, i.e. the one in the 'fedora' repo and which is included in composes - *would* be affected by this bug. That means that if we decide the bug is a blocker, we must find a fix for it before we can ship Alpha. But, she and jforbes also believe this is fixed in upstream kernel by commit d52c0569bab4edc888832df44dc7ac28517134f6 , and that furthermore that means the bug should be fixed by these Fedora builds:

f25: http://koji.fedoraproject.org/koji/buildinfo?buildID=792279 (kernel-4.8.0-0.rc2.git1.1.fc25)
Rawhide: http://koji.fedoraproject.org/koji/buildinfo?buildID=792280 (kernel-4.8.0-0.rc2.git1.1.fc26)

that build is not currently submitted as an update for F25. It would be good if reporters could confirm the fix.

Comment 8 Adam Williamson 2016-08-17 19:54:50 UTC
labbott also states she'd vote -1 blocker / +1 FE for this bug, given the range of hardware affected. jforbes says "1365917 could theoretically impact any modern intel machine", the upstream commit can be seen at https://lkml.org/lkml/2016/8/11/516 , describing the issue, if anyone feels up to evaluating its impact themselves. "any modern intel machine" is quite scary to me, I might be more inclined to go +1 blocker for this one, I'm definitely +1 FE.

Comment 9 Justin M. Forbes 2016-08-17 20:33:10 UTC
To clarify the "Any modern intel machine" x2apic was introduced with nehalem, so about 6 years ago. It can also be "opted out" of by firmware, and frequently is. I don't know the percentages of machines that do or don't opt out, I know by a quick look at 3 machines here, 2 have it turned off, 1 has it turned on. You can check by looking at a dmesg after boot, you will either see "x2apic enabled" or "DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit" with instructions on how to override the opt out. A quick google search shows that several people have seen this bug, but it is still hard to determine because no one shipped a kernel to masses of users with the bug.

Comment 10 Adam Williamson 2016-08-17 20:34:00 UTC
Bit more discussion about the range of hardware likely affected by this:

<jwb> jforbes: eh... i won't disagree but that might be stretching it
<jforbes> jwb: theoretically. x2apic came in with nahalem, and it is basically a race condition with CPU state change
 realistically it is probably a smaller subset, but a quick google search says it is non trivial
<jwb> jforbes: yeah, but i thought there was a firmware component to x2apic support too
 i might be thinking of something else
<jforbes> jwb: there is, thus the theoretical part
<jwb> right.  so the stretch is that most laptop class hardware doesn't have the firmware bits for x2apic.  at least not that i've seen
 but desktop/larger servers are certainly a possibility
 now if we only could tell for certainty what most Fedora users have for machines.  IN A WORLD
<jforbes> Well, that would certainly be nice
 only 1 out of 3 machines here has it enabled
 I could power on and check others I suppose
 But even in the ones that disable by default, it can be overridden

Comment 11 Peter Gervase 2016-08-17 20:56:59 UTC
For those that have dep issues installing:
$ sudo rpm -ivh kernel-4.8.0-0.rc2.git1.1.fc26.x86_64.rpm                                  
error: Failed dependencies:
        kernel-core-uname-r = 4.8.0-0.rc2.git1.1.fc26.x86_64 is needed by kernel-4.8.0-0.rc2.git1.1.fc26.x86_64
        kernel-modules-uname-r = 4.8.0-0.rc2.git1.1.fc26.x86_64 is needed by kernel-4.8.0-0.rc2.git1.1.fc26.x86_64

I made
https://bugzilla.redhat.com/show_bug.cgi?id=1367929
to clean up the dep checking - "uname -r" not getting parsed.

I'll test booting to that rc2 kernel...

Comment 12 Adam Williamson 2016-08-17 21:03:16 UTC
er...you're reading that wrong. you have to install at least the kernel, kernel-core and kernel-modules packages when manually installing a kernel build. The package called 'kernel' is basically just a metapackage and doesn't contain anything. The actual kernel is in 'kernel-core', the modules are in 'kernel-modules'. You may also need 'kernel-modules-extra' depending on your hardware.

Comment 13 Peter Gervase 2016-08-17 21:08:09 UTC
Right, you need all three, but the error shouldn't say "uname-r" in the failed deps. kernel-core-4.8.0-0.rc2.git1.1.fc26.x86_64 and kernel-modules-4.8.0-0.rc2.git1.1.fc26.x86_64 are what should be specified, not "kernel-core-uname-r" or "kernel-modules-uname-r".

$ sudo rpm -ivh kernel-4.8.0-0.rc2.git1.1.fc26.x86_64.rpm kernel-core-4.8.0-0.rc2.git1.1.fc26.x86_64.rpm kernel-modules-4.8.0-0.rc2.git1.1.fc26.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-core-4.8.0-0.rc2.git1.1.fc################################# [ 33%]
   2:kernel-modules-4.8.0-0.rc2.git1.1################################# [ 67%]
   3:kernel-4.8.0-0.rc2.git1.1.fc26   ################################# [100%]

Comment 14 Adam Williamson 2016-08-17 21:11:46 UTC
nah, the Provides: are explicitly named that way in the spec, the spec clearly doesn't expect the 'uname-r' to be interpreted as a command:

http://pkgs.fedoraproject.org/cgit/rpms/kernel.git/tree/kernel.spec#n633
http://pkgs.fedoraproject.org/cgit/rpms/kernel.git/tree/kernel.spec#n824
http://pkgs.fedoraproject.org/cgit/rpms/kernel.git/tree/kernel.spec#n847

etc. I dunno why the kernel team decided to use those names, but it's a conscious choice.

Comment 15 Stephen Gallagher 2016-08-17 21:17:32 UTC
Per Paul Whalen: "adding 'nox2apic' (on Fedora-25-20160807.n.0) got the installer booting on an x220 laptop".

Given that there's a relatively straightforward workaround on the kernel boot command line, I'm inclined to say -1 blocker, +1 FE here.

Comment 16 Adam Williamson 2016-08-18 18:56:53 UTC
Discussed at 2016-08-18 go/no-go meeting, functioning as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting/2016-08-18/f25-alpha-go_no_go-meeting.2016-08-18-17.00.html . Given our best estimate as to the range of hardware affected, and on the basis there's a simple documentable workaround, we decided to reject it as an Alpha blocker, but accept it as a freeze exception issue.

Comment 17 Fedora Update System 2016-08-19 02:09:40 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 18 Petr Schindler 2016-08-19 07:32:36 UTC
I can confirm that with 'nox2apic' I can boot (installer and installed system).

Comment 19 Fedora Update System 2016-08-19 16:50:38 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 20 Fedora Update System 2016-08-19 21:53:31 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 21 Fedora Update System 2016-08-19 22:08:21 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 22 Fedora Update System 2016-08-19 22:11:14 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 23 Fedora Update System 2016-08-19 22:15:48 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 24 Fedora Update System 2016-08-19 22:19:25 UTC
kernel-4.8.0-0.rc2.git2.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 25 Fedora Update System 2016-08-20 18:50:38 UTC
kernel-4.8.0-0.rc2.git3.1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0dd1a509c8

Comment 26 Fedora Update System 2016-08-22 22:07:45 UTC
kernel-4.8.0-0.rc2.git3.1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Petr Schindler 2016-08-23 08:36:36 UTC
kernel-4.8.0-0.rc2.git3.1.fc25 really solves problem for me.


Note You need to log in before you can comment on or make changes to this bug.