Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1570983

Summary: Kernel improvements for the Raspberry Pi 3 and 3+
Product: [Fedora] Fedora Reporter: Peter Robinson <pbrobinson>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 28CC: airlied, alciregi, awilliam, bskeggs, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, kevin, linville, m.a.young, mboddu, mchehab, mjg59, pbrobinson, pwhalen, steved
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: kernel-4.16.3-301.fc28 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-25 00:04:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1469207    

Description Peter Robinson 2018-04-23 22:09:32 UTC
Commit ce55d365ba258b7a83bd86a79344da935518c975 has a number of improvements for the Raspberry Pi 3 and 3+. In particular the 3+ patches are the newer reviewed upstream patches, and also includes a number of improvements in the lan78xx wired ethernet driver that is the newer 3+ which has had some stability issues.

This is useful as the ARM images aren't regenerated post release and the RPi is a common arm device so the more stable it as at GA the less support issues we have over the cycle once the release it out.

Comment 1 Fedora Blocker Bugs Application 2018-04-23 22:11:41 UTC
Proposed as a Freeze Exception for 28-final by Fedora user pbrobinson using the blocker tracking app because:

 Improvement to the Raspberry Pi 3 series, in particular the new 3+ which has been stabilising it's upstream support still and upstream patches have gone through reviews and improvements.

Comment 2 Adam Williamson 2018-04-23 22:24:22 UTC
So we have two kernel builds running ATM:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1074979 (4.16.3-301)
https://koji.fedoraproject.org/koji/buildinfo?buildID=1074986 (4.16.3-302)

4.16.3-301 *INCLUDES* these changes. 4.16.3-302 *DOES NOT INCLUDE* these changes. (It'd be a bit more intuitive if this was the other way around, but never mind). The idea here is to give us a choice of whether to pull in these fixes or not - if we want these fixes we can go with -301, if we decide it's too risky we can go with -302.

I'm slightly worried that there's a lot of change here and it's showing up very late; clearly rpi3 support is nowhere near perfect ATM, but it worked well enough for us to ship the beta at least, so it's at least theoretically possible that some overlooked issue in these changes could make things worse in some important configuration or other. That doesn't seem like a *huge* risk, though, and the changes are quite well isolated to the rpi3 code, so shouldn't be able to break anything else.

Given that I'm +1 FE for this if we can get at least one or two other folks with Pi 3 hardware (hi, pwhalen) to try out both builds and confirm that 4.16.3-301 works better (or at least as well) for them. If we get at least that much testing it seems worth the risk to go with -301, to me.

Comment 3 Alessio 2018-04-24 15:15:38 UTC
I've installed -301 and -302, and used the RPi3 for a while with each build.
FWIW, I did not experienced any issue, until now.

Comment 4 Kevin Fenzi 2018-04-24 19:13:12 UTC
I'd be +1 FE with some Pi 3 testing also.

Comment 5 Mohan Boddu 2018-04-24 19:17:32 UTC
+1 FE with more RPi3 testing

Comment 6 Adam Williamson 2018-04-24 19:20:01 UTC
That's enough to accept as an FE for now. It'd be great if folks who have Pi 3s can test with the -301 kernel and report if it works for them.

Comment 7 Peter Robinson 2018-04-24 19:40:17 UTC
I've tested on the Raspberry Pi3 and 3+ on aarch 64, the 3 and 2 on ARMv7

Comment 8 Jared Smith 2018-04-24 19:54:46 UTC
+1 FE for me.

Comment 9 Michael Young 2018-04-24 20:21:34 UTC
It works for me on a Pi3 B (not +) aarch64. I did notice a warning in dmesg

[   34.100866] WARNING: CPU: 3 PID: 5 at drivers/gpio/gpiolib.c:2986 gpiod_set_value+0x60/0x68
[   34.100872] Modules linked in: mac80211(+) hci_uart(+) snd_pcm btbcm snd_timer btintel brcmfmac btqca snd soundcore bluetooth brcmutil drm_kms_helper cfg80211 drm ecdh_generic gpio_raspberrypi_exp fb_sys_fops syscopyarea rfkill sysfillrect sysimgblt nvmem_core bcm2835_thermal
[   34.117365] Bluetooth: HCI UART protocol AG6XX registered
[   34.117378] Bluetooth: HCI UART protocol Marvell registered
[   34.158950]  crc32_ce bcm2835_wdt bcm2835_rng leds_gpio xfs libcrc32c smsc95xx usbnet mii mmc_block dwc2 sdhci_iproc sdhci_pltfm udc_core sdhci pwm_bcm2835 bcm2835 i2c_bcm2835 bcm2835_dma phy_generic
[   34.159019] CPU: 3 PID: 5 Comm: kworker/u8:0 Not tainted 4.16.3-301.fc28.aarch64 #1
[   34.159030] Hardware name: raspberrypi rpi/rpi, BIOS 2018.03 04/15/2018
[   34.237430] Workqueue: events_unbound async_run_entry_fn
[   34.247267] pstate: 20400005 (nzCv daif +PAN -UAO)
[   34.256502] pc : gpiod_set_value+0x60/0x68
[   34.256517] lr : gpiod_set_value+0x30/0x68
[   34.273391] sp : ffff00000804bbc0
[   34.273396] x29: ffff00000804bbc0 x28: 0000000000000000 
[   34.273404] x27: ffff800037073cb8 x26: ffff000008c7da08 
[   34.273412] x25: 0000000000000000 x24: 000000000000001d 
[   34.273420] x23: ffff0000013aa0b8 x22: 0000000000000000 
[   34.273427] x21: 0000000000000000 x20: 0000000000000000 
[   34.273435] x19: ffff800033c62b00 x18: 0000000000000003 
[   34.273442] x17: 0000000000000000 x16: 0000000000000000 
[   34.273449] x15: 0000000000000000 x14: 000000000000000c 
[   34.273456] x13: 0000000000000001 x12: 0000000000000020 
[   34.273464] x11: 0000000000000003 x10: 0101010101010101 
[   34.273471] x9 : 02fefeff01fefeff x8 : ffff80002e5a9200 
[   34.273479] x7 : ffff800008007c80 x6 : ffff7e0000b96a40 
[   34.273486] x5 : 00000000000b795c x4 : ffff80003e5628e0 
[   34.273494] x3 : 00000000000b795d x2 : ffff800036032428 
[   34.273501] x1 : ffff80002e29a800 x0 : 0000000000000001 
[   34.273509] Call trace:
[   34.273526]  gpiod_set_value+0x60/0x68
[   34.273646]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
[   34.273749]  bcm_gpio_set_power+0xc0/0x168 [hci_uart]
[   34.273856]  bcm_serdev_probe+0x80/0xe8 [hci_uart]
[   34.273867]  serdev_drv_probe+0x28/0x38
[   34.273875]  really_probe+0x204/0x3c8
[   34.273883]  driver_probe_device+0x54/0xd8
[   34.273889]  __driver_attach+0x124/0x128
[   34.273895]  bus_for_each_dev+0x70/0xa8
[   34.273901]  driver_attach+0x30/0x40
[   34.273907]  driver_attach_async+0x20/0x60
[   34.273915]  async_run_entry_fn+0x4c/0x188
[   34.273925]  process_one_work+0x19c/0x3a8
[   34.273930]  worker_thread+0x4c/0x418
[   34.273938]  kthread+0x130/0x138
[   34.273946]  ret_from_fork+0x10/0x18
[   34.273950] ---[ end trace 8d70df395a36f988 ]---

Comment 10 Adam Williamson 2018-04-24 20:25:18 UTC
Michael: when you say "it" - which kernel did you test, -301 or -302? Or both? Did you also try the bcm283x-firmware update from https://bugzilla.redhat.com/show_bug.cgi?id=1570993 ? Thanks!

Comment 11 Paul Whalen 2018-04-24 20:39:59 UTC
No issues with 301, tested on rpi2, rpi3, rpi3+. Testing 302 on the rpi3 the gui did not come up on first boot, rebooted and it started normally. No other issues with 302 on rpi3+ or rpi2.

Comment 12 Michael Young 2018-04-24 20:44:28 UTC
That was with -301 and the bcm283x-firmware update(you could actually tell from the backtrace). I will now try -302

Comment 13 Adam Williamson 2018-04-24 20:49:48 UTC
Michael: did you run `rpi-firmware-update` and reboot after installing the firmware package? peter's just let us know you actually have to do that, just updating the package doesn't actually load the updated firmware.

Comment 14 Michael Young 2018-04-24 21:03:58 UTC
Yes (you can tell from the backtrace). -302 works as well though I get a similar backtrace
[   21.699472] WARNING: CPU: 0 PID: 434 at drivers/gpio/gpiolib.c:2986 gpiod_set_value+0x60/0x68
[   21.699476] Modules linked in: rc_core mac80211 hci_uart brcmfmac snd_pcm btbcm btintel snd_timer btqca brcmutil snd bluetooth cfg80211 soundcore drm_kms_helper drm crc32_ce fb_sys_fops syscopyarea sysfillrect ecdh_generic gpio_raspberrypi_exp sysimgblt rfkill nvmem_core bcm2835_thermal bcm2835_wdt bcm2835_rng leds_gpio xfs smsc95xx usbnet mii libcrc32c mmc_block sdhci_iproc dwc2 sdhci_pltfm sdhci udc_core i2c_bcm2835 pwm_bcm2835 bcm2835 bcm2835_dma phy_generic
[   21.699599] CPU: 0 PID: 434 Comm: kworker/u8:3 Tainted: G        W        4.16.3-302.fc28.aarch64 #1
[   21.699602] Hardware name: raspberrypi rpi/rpi, BIOS 2018.03 04/15/2018
[   21.699619] Workqueue: events_unbound async_run_entry_fn
[   21.699628] pstate: 20400005 (nzCv daif +PAN -UAO)
[   21.699636] pc : gpiod_set_value+0x60/0x68
[   21.699644] lr : gpiod_set_value+0x30/0x68
[   21.699648] sp : ffff00000a3a3b50
[   21.699651] x29: ffff00000a3a3b50 x28: 0000000000000000 
[   21.699659] x27: ffff8000339864b8 x26: ffff000008c8df08 
[   21.699667] x25: 0000000000000000 x24: 000000000000001d 
[   21.699674] x23: ffff0000013b90b8 x22: fffffffffffffffe 
[   21.699681] x21: 0000000000000001 x20: 0000000000000001 
[   21.699689] x19: ffff80002e6b2700 x18: 00000000fffffffe 
[   21.699696] x17: 0000000000000000 x16: 0000000000000000 
[   21.699703] x15: 0000000000000000 x14: ffff0000093a9000 
[   21.699710] x13: 0000000000000000 x12: ffff00000953a000 
[   21.699718] x11: ffff000009385000 x10: 0000000000000020 
[   21.699725] x9 : 0000000000000000 x8 : ffff00000c4ea260 
[   21.699732] x7 : 0000000000000000 x6 : 000000000000003f 
[   21.699740] x5 : ffff00000a3a3a58 x4 : 0000000000000000 
[   21.699747] x3 : 0000000000000002 x2 : ffff8000373d3628 
[   21.699754] x1 : ffff800037e7f400 x0 : 0000000000000001 
[   21.699761] Call trace:
[   21.699771]  gpiod_set_value+0x60/0x68
[   21.699892]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
[   21.699998]  bcm_gpio_set_power+0x40/0x168 [hci_uart]
[   21.700101]  bcm_open+0xe8/0x1c8 [hci_uart]
[   21.700202]  hci_uart_register_device+0x3c/0x2a0 [hci_uart]
[   21.700306]  bcm_serdev_probe+0x94/0xe8 [hci_uart]
[   21.700318]  serdev_drv_probe+0x28/0x38
[   21.700326]  really_probe+0x204/0x3c8
[   21.700334]  driver_probe_device+0x54/0xd8
[   21.700340]  __driver_attach+0x124/0x128
[   21.700346]  bus_for_each_dev+0x70/0xa8
[   21.700352]  driver_attach+0x30/0x40
[   21.700358]  driver_attach_async+0x20/0x60
[   21.700365]  async_run_entry_fn+0x4c/0x188
[   21.700371]  process_one_work+0x19c/0x3a8
[   21.700376]  worker_thread+0x4c/0x418
[   21.700384]  kthread+0x130/0x138
[   21.700392]  ret_from_fork+0x10/0x18
[   21.700396] ---[ end trace f7eca6931e810844 ]---

Comment 15 Adam Williamson 2018-04-24 21:09:20 UTC
"you can tell from the backtrace"

maybe you can, I can't :P thanks. Do you get the backtrace with -300 too?

Comment 16 Peter Robinson 2018-04-24 21:13:42 UTC
(In reply to Michael Young from comment #9)
> It works for me on a Pi3 B (not +) aarch64. I did notice a warning in dmesg

What was your previous kernel, that warning was also seen in the previous kernel-4.16.2-300.fc28 kernel (and kernel-4.16.3-300.fc28) stable too. 

It's a problem in the btbcm driver which wasn't touched in any of the recent kernel updates. The gpio part of it is about rfkill (basically "Airplane mode") because it's GPIO that deals with the power to the unit and doesn't affect the current bluetooth status.

Comment 17 Peter Robinson 2018-04-24 21:17:37 UTC
(In reply to Michael Young from comment #12)
> That was with -301 and the bcm283x-firmware update(you could actually tell
> from the backtrace). I will now try -302

Can tell the kernel version but not the boot firmware version, not that it will make any difference as this bit is not related to and unaffected by the firmware, both will have the same effect.

Comment 18 Peter Robinson 2018-04-24 21:26:47 UTC
> [   21.699771]  gpiod_set_value+0x60/0x68
> [   21.699892]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
> [   21.699998]  bcm_gpio_set_power+0x40/0x168 [hci_uart]
> [   21.700101]  bcm_open+0xe8/0x1c8 [hci_uart]
> [   21.700202]  hci_uart_register_device+0x3c/0x2a0 [hci_uart]
> [   21.700306]  bcm_serdev_probe+0x94/0xe8 [hci_uart]
> [   21.700318]  serdev_drv_probe+0x28/0x38

For reference the above bits show it's bluetooth, serdev is Serial Device (new framework from 4.14):
https://elixir.bootlin.com/linux/latest/source/drivers/bluetooth/hci_bcm.c#L1155

There have been updates to btbcm in the 4.16 stable cycle but none that have affected this problem in a good or bad manner, bluetooth still mostly sort of works, as good or not so good, as it previously did any time in the 4.16 cycle

Comment 19 Peter Robinson 2018-04-24 21:48:15 UTC
(In reply to Adam Williamson from comment #15)
> "you can tell from the backtrace"
> 
> maybe you can, I can't :P thanks. Do you get the backtrace with -300 too?

[   21.699599] CPU: 0 PID: 434 Comm: kworker/u8:3 Tainted: G        W        4.16.3-302.fc28.aarch64 #1

;-P

Comment 20 Adam Williamson 2018-04-24 21:51:13 UTC
yeah, I meant the firmware version. I guess that's "BIOS 2018.03 04/15/2018", but that's not enough for at least *me* to be sure. :P

Comment 21 Peter Robinson 2018-04-24 21:55:30 UTC
(In reply to Adam Williamson from comment #20)
> yeah, I meant the firmware version. I guess that's "BIOS 2018.03
> 04/15/2018", but that's not enough for at least *me* to be sure. :P

that's the u-boot version "2018.03" and build date "04/15/2018"

the raspberry pi firmware version in dmesg reports looks like:
"raspberrypi-firmware soc:firmware: Attached to firmware from 2018-04-16 18:18"

Comment 22 Fedora Update System 2018-04-24 22:27:18 UTC
kernel-4.16.3-301.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-b921ff4b1e

Comment 23 Adam Williamson 2018-04-24 22:29:51 UTC
Can folks who tested -301 and found it at least no worse than -300 or -302 please file karma at https://bodhi.fedoraproject.org/updates/kernel-4.16.3-301.fc28 ? Thanks! (And of course if you tested it and it ate your kittens, file *negative* karma).

Comment 24 Peter Robinson 2018-04-24 23:04:22 UTC
(In reply to Adam Williamson from comment #23)
> Can folks who tested -301 and found it at least no worse than -300 or -302
> please file karma at
> https://bodhi.fedoraproject.org/updates/kernel-4.16.3-301.fc28 ? Thanks!
> (And of course if you tested it and it ate your kittens, file *negative*
> karma).

For my own (unverified) sanity, this is the rebase patch version (ie everything I had pushed) variation of the options?

Comment 25 Adam Williamson 2018-04-24 23:30:18 UTC
Peter: yep. -301 has the rebase patch, -302 doesn't, as mentioned in comment 2.

Comment 26 Fedora Update System 2018-04-25 00:04:39 UTC
kernel-4.16.3-301.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.