Bug 1570983

Proposed as a Freeze Exception for 28-final by Fedora user pbrobinson using the blocker tracking app because:

 Improvement to the Raspberry Pi 3 series, in particular the new 3+ which has been stabilising it's upstream support still and upstream patches have gone through reviews and improvements.

So we have two kernel builds running ATM:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1074979 (4.16.3-301)
https://koji.fedoraproject.org/koji/buildinfo?buildID=1074986 (4.16.3-302)

4.16.3-301 *INCLUDES* these changes. 4.16.3-302 *DOES NOT INCLUDE* these changes. (It'd be a bit more intuitive if this was the other way around, but never mind). The idea here is to give us a choice of whether to pull in these fixes or not - if we want these fixes we can go with -301, if we decide it's too risky we can go with -302.

I'm slightly worried that there's a lot of change here and it's showing up very late; clearly rpi3 support is nowhere near perfect ATM, but it worked well enough for us to ship the beta at least, so it's at least theoretically possible that some overlooked issue in these changes could make things worse in some important configuration or other. That doesn't seem like a *huge* risk, though, and the changes are quite well isolated to the rpi3 code, so shouldn't be able to break anything else.

Given that I'm +1 FE for this if we can get at least one or two other folks with Pi 3 hardware (hi, pwhalen) to try out both builds and confirm that 4.16.3-301 works better (or at least as well) for them. If we get at least that much testing it seems worth the risk to go with -301, to me.

I've installed -301 and -302, and used the RPi3 for a while with each build.
FWIW, I did not experienced any issue, until now.

I'd be +1 FE with some Pi 3 testing also.

+1 FE with more RPi3 testing

That's enough to accept as an FE for now. It'd be great if folks who have Pi 3s can test with the -301 kernel and report if it works for them.

I've tested on the Raspberry Pi3 and 3+ on aarch 64, the 3 and 2 on ARMv7

+1 FE for me.

It works for me on a Pi3 B (not +) aarch64. I did notice a warning in dmesg

[   34.100866] WARNING: CPU: 3 PID: 5 at drivers/gpio/gpiolib.c:2986 gpiod_set_value+0x60/0x68
[   34.100872] Modules linked in: mac80211(+) hci_uart(+) snd_pcm btbcm snd_timer btintel brcmfmac btqca snd soundcore bluetooth brcmutil drm_kms_helper cfg80211 drm ecdh_generic gpio_raspberrypi_exp fb_sys_fops syscopyarea rfkill sysfillrect sysimgblt nvmem_core bcm2835_thermal
[   34.117365] Bluetooth: HCI UART protocol AG6XX registered
[   34.117378] Bluetooth: HCI UART protocol Marvell registered
[   34.158950]  crc32_ce bcm2835_wdt bcm2835_rng leds_gpio xfs libcrc32c smsc95xx usbnet mii mmc_block dwc2 sdhci_iproc sdhci_pltfm udc_core sdhci pwm_bcm2835 bcm2835 i2c_bcm2835 bcm2835_dma phy_generic
[   34.159019] CPU: 3 PID: 5 Comm: kworker/u8:0 Not tainted 4.16.3-301.fc28.aarch64 #1
[   34.159030] Hardware name: raspberrypi rpi/rpi, BIOS 2018.03 04/15/2018
[   34.237430] Workqueue: events_unbound async_run_entry_fn
[   34.247267] pstate: 20400005 (nzCv daif +PAN -UAO)
[   34.256502] pc : gpiod_set_value+0x60/0x68
[   34.256517] lr : gpiod_set_value+0x30/0x68
[   34.273391] sp : ffff00000804bbc0
[   34.273396] x29: ffff00000804bbc0 x28: 0000000000000000 
[   34.273404] x27: ffff800037073cb8 x26: ffff000008c7da08 
[   34.273412] x25: 0000000000000000 x24: 000000000000001d 
[   34.273420] x23: ffff0000013aa0b8 x22: 0000000000000000 
[   34.273427] x21: 0000000000000000 x20: 0000000000000000 
[   34.273435] x19: ffff800033c62b00 x18: 0000000000000003 
[   34.273442] x17: 0000000000000000 x16: 0000000000000000 
[   34.273449] x15: 0000000000000000 x14: 000000000000000c 
[   34.273456] x13: 0000000000000001 x12: 0000000000000020 
[   34.273464] x11: 0000000000000003 x10: 0101010101010101 
[   34.273471] x9 : 02fefeff01fefeff x8 : ffff80002e5a9200 
[   34.273479] x7 : ffff800008007c80 x6 : ffff7e0000b96a40 
[   34.273486] x5 : 00000000000b795c x4 : ffff80003e5628e0 
[   34.273494] x3 : 00000000000b795d x2 : ffff800036032428 
[   34.273501] x1 : ffff80002e29a800 x0 : 0000000000000001 
[   34.273509] Call trace:
[   34.273526]  gpiod_set_value+0x60/0x68
[   34.273646]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
[   34.273749]  bcm_gpio_set_power+0xc0/0x168 [hci_uart]
[   34.273856]  bcm_serdev_probe+0x80/0xe8 [hci_uart]
[   34.273867]  serdev_drv_probe+0x28/0x38
[   34.273875]  really_probe+0x204/0x3c8
[   34.273883]  driver_probe_device+0x54/0xd8
[   34.273889]  __driver_attach+0x124/0x128
[   34.273895]  bus_for_each_dev+0x70/0xa8
[   34.273901]  driver_attach+0x30/0x40
[   34.273907]  driver_attach_async+0x20/0x60
[   34.273915]  async_run_entry_fn+0x4c/0x188
[   34.273925]  process_one_work+0x19c/0x3a8
[   34.273930]  worker_thread+0x4c/0x418
[   34.273938]  kthread+0x130/0x138
[   34.273946]  ret_from_fork+0x10/0x18
[   34.273950] ---[ end trace 8d70df395a36f988 ]---

Michael: when you say "it" - which kernel did you test, -301 or -302? Or both? Did you also try the bcm283x-firmware update from https://bugzilla.redhat.com/show_bug.cgi?id=1570993 ? Thanks!

No issues with 301, tested on rpi2, rpi3, rpi3+. Testing 302 on the rpi3 the gui did not come up on first boot, rebooted and it started normally. No other issues with 302 on rpi3+ or rpi2.

That was with -301 and the bcm283x-firmware update(you could actually tell from the backtrace). I will now try -302

Michael: did you run `rpi-firmware-update` and reboot after installing the firmware package? peter's just let us know you actually have to do that, just updating the package doesn't actually load the updated firmware.

Yes (you can tell from the backtrace). -302 works as well though I get a similar backtrace
[   21.699472] WARNING: CPU: 0 PID: 434 at drivers/gpio/gpiolib.c:2986 gpiod_set_value+0x60/0x68
[   21.699476] Modules linked in: rc_core mac80211 hci_uart brcmfmac snd_pcm btbcm btintel snd_timer btqca brcmutil snd bluetooth cfg80211 soundcore drm_kms_helper drm crc32_ce fb_sys_fops syscopyarea sysfillrect ecdh_generic gpio_raspberrypi_exp sysimgblt rfkill nvmem_core bcm2835_thermal bcm2835_wdt bcm2835_rng leds_gpio xfs smsc95xx usbnet mii libcrc32c mmc_block sdhci_iproc dwc2 sdhci_pltfm sdhci udc_core i2c_bcm2835 pwm_bcm2835 bcm2835 bcm2835_dma phy_generic
[   21.699599] CPU: 0 PID: 434 Comm: kworker/u8:3 Tainted: G        W        4.16.3-302.fc28.aarch64 #1
[   21.699602] Hardware name: raspberrypi rpi/rpi, BIOS 2018.03 04/15/2018
[   21.699619] Workqueue: events_unbound async_run_entry_fn
[   21.699628] pstate: 20400005 (nzCv daif +PAN -UAO)
[   21.699636] pc : gpiod_set_value+0x60/0x68
[   21.699644] lr : gpiod_set_value+0x30/0x68
[   21.699648] sp : ffff00000a3a3b50
[   21.699651] x29: ffff00000a3a3b50 x28: 0000000000000000 
[   21.699659] x27: ffff8000339864b8 x26: ffff000008c8df08 
[   21.699667] x25: 0000000000000000 x24: 000000000000001d 
[   21.699674] x23: ffff0000013b90b8 x22: fffffffffffffffe 
[   21.699681] x21: 0000000000000001 x20: 0000000000000001 
[   21.699689] x19: ffff80002e6b2700 x18: 00000000fffffffe 
[   21.699696] x17: 0000000000000000 x16: 0000000000000000 
[   21.699703] x15: 0000000000000000 x14: ffff0000093a9000 
[   21.699710] x13: 0000000000000000 x12: ffff00000953a000 
[   21.699718] x11: ffff000009385000 x10: 0000000000000020 
[   21.699725] x9 : 0000000000000000 x8 : ffff00000c4ea260 
[   21.699732] x7 : 0000000000000000 x6 : 000000000000003f 
[   21.699740] x5 : ffff00000a3a3a58 x4 : 0000000000000000 
[   21.699747] x3 : 0000000000000002 x2 : ffff8000373d3628 
[   21.699754] x1 : ffff800037e7f400 x0 : 0000000000000001 
[   21.699761] Call trace:
[   21.699771]  gpiod_set_value+0x60/0x68
[   21.699892]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
[   21.699998]  bcm_gpio_set_power+0x40/0x168 [hci_uart]
[   21.700101]  bcm_open+0xe8/0x1c8 [hci_uart]
[   21.700202]  hci_uart_register_device+0x3c/0x2a0 [hci_uart]
[   21.700306]  bcm_serdev_probe+0x94/0xe8 [hci_uart]
[   21.700318]  serdev_drv_probe+0x28/0x38
[   21.700326]  really_probe+0x204/0x3c8
[   21.700334]  driver_probe_device+0x54/0xd8
[   21.700340]  __driver_attach+0x124/0x128
[   21.700346]  bus_for_each_dev+0x70/0xa8
[   21.700352]  driver_attach+0x30/0x40
[   21.700358]  driver_attach_async+0x20/0x60
[   21.700365]  async_run_entry_fn+0x4c/0x188
[   21.700371]  process_one_work+0x19c/0x3a8
[   21.700376]  worker_thread+0x4c/0x418
[   21.700384]  kthread+0x130/0x138
[   21.700392]  ret_from_fork+0x10/0x18
[   21.700396] ---[ end trace f7eca6931e810844 ]---

"you can tell from the backtrace"

maybe you can, I can't :P thanks. Do you get the backtrace with -300 too?

(In reply to Michael Young from comment #9)
> It works for me on a Pi3 B (not +) aarch64. I did notice a warning in dmesg

What was your previous kernel, that warning was also seen in the previous kernel-4.16.2-300.fc28 kernel (and kernel-4.16.3-300.fc28) stable too. 

It's a problem in the btbcm driver which wasn't touched in any of the recent kernel updates. The gpio part of it is about rfkill (basically "Airplane mode") because it's GPIO that deals with the power to the unit and doesn't affect the current bluetooth status.

(In reply to Michael Young from comment #12)
> That was with -301 and the bcm283x-firmware update(you could actually tell
> from the backtrace). I will now try -302

Can tell the kernel version but not the boot firmware version, not that it will make any difference as this bit is not related to and unaffected by the firmware, both will have the same effect.

> [   21.699771]  gpiod_set_value+0x60/0x68
> [   21.699892]  bcm_gpio_set_shutdown+0x28/0x38 [hci_uart]
> [   21.699998]  bcm_gpio_set_power+0x40/0x168 [hci_uart]
> [   21.700101]  bcm_open+0xe8/0x1c8 [hci_uart]
> [   21.700202]  hci_uart_register_device+0x3c/0x2a0 [hci_uart]
> [   21.700306]  bcm_serdev_probe+0x94/0xe8 [hci_uart]
> [   21.700318]  serdev_drv_probe+0x28/0x38

For reference the above bits show it's bluetooth, serdev is Serial Device (new framework from 4.14):
https://elixir.bootlin.com/linux/latest/source/drivers/bluetooth/hci_bcm.c#L1155

There have been updates to btbcm in the 4.16 stable cycle but none that have affected this problem in a good or bad manner, bluetooth still mostly sort of works, as good or not so good, as it previously did any time in the 4.16 cycle

(In reply to Adam Williamson from comment #15)
> "you can tell from the backtrace"
> 
> maybe you can, I can't :P thanks. Do you get the backtrace with -300 too?

[   21.699599] CPU: 0 PID: 434 Comm: kworker/u8:3 Tainted: G        W        4.16.3-302.fc28.aarch64 #1

;-P

yeah, I meant the firmware version. I guess that's "BIOS 2018.03 04/15/2018", but that's not enough for at least *me* to be sure. :P

(In reply to Adam Williamson from comment #20)
> yeah, I meant the firmware version. I guess that's "BIOS 2018.03
> 04/15/2018", but that's not enough for at least *me* to be sure. :P

that's the u-boot version "2018.03" and build date "04/15/2018"

the raspberry pi firmware version in dmesg reports looks like:
"raspberrypi-firmware soc:firmware: Attached to firmware from 2018-04-16 18:18"

kernel-4.16.3-301.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-b921ff4b1e

Can folks who tested -301 and found it at least no worse than -300 or -302 please file karma at https://bodhi.fedoraproject.org/updates/kernel-4.16.3-301.fc28 ? Thanks! (And of course if you tested it and it ate your kittens, file *negative* karma).

(In reply to Adam Williamson from comment #23)
> Can folks who tested -301 and found it at least no worse than -300 or -302
> please file karma at
> https://bodhi.fedoraproject.org/updates/kernel-4.16.3-301.fc28 ? Thanks!
> (And of course if you tested it and it ate your kittens, file *negative*
> karma).

For my own (unverified) sanity, this is the rebase patch version (ie everything I had pushed) variation of the options?

Peter: yep. -301 has the rebase patch, -302 doesn't, as mentioned in comment 2.

kernel-4.16.3-301.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.