Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1661961
Summary: | Raspberry Pi 3B+: Bluetooth adapter changed from hci0 to hci1 with kernel 4.19.10-300.fc29.armv7hl | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Roel <vruchtengaard> |
Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | low | Docs Contact: | |
Priority: | unspecified | ||
Version: | 29 | CC: | airlied, bskeggs, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, mchehab, mjg59, steved, y9t7sypezp |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | armv7hl | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | kernel-4.19.14-200.fc28 kernel-4.19.14-300.fc29 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-01-15 01:53:29 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Attachments: |
Description
Roel
2018-12-24 22:51:12 UTC
Could you attach the output from this command?
$ journalctl -b --no-hostname > journalctl-1.log
(Use the "Add an attachment" link at the top of the page.)
Could you confirm that your system is fully updated?
> When looking at: /sys/class/bluetooth/hci0 there was a simlink to 'mmc_...'
What do these show?
$ udevadm info /sys/class/bluetooth/hci0
$ udevadm info /sys/class/bluetooth/hci1
> Also hcitool -i hci1 lescan stopped working.
hcitool is in the bluez package, but that problem could be related. Could you post the exact output from these commands?
$ hcitool dev
$ hcitool -i hci1 lescan
Created attachment 1517113 [details]
journalctl-1.log
udevadm info /sys/class/bluetooth/hci0 P: /devices/platform/soc/3f201000.serial/serial0/serial0-0/bluetooth/hci0 E: DEVPATH=/devices/platform/soc/3f201000.serial/serial0/serial0-0/bluetooth/hci0 E: DEVTYPE=host E: SUBSYSTEM=bluetooth E: SYSTEMD_ALIAS=/sys/subsystem/bluetooth/devices/hci0 E: SYSTEMD_WANTS=bluetooth.target E: TAGS=:systemd: E: USEC_INITIALIZED=29685599 udevadm info /sys/class/bluetooth/hci1 P: /devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:3/bluetooth/hci1 E: DEVPATH=/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:3/bluetooth/hci1 E: DEVTYPE=host E: SUBSYSTEM=bluetooth E: SYSTEMD_ALIAS=/sys/subsystem/bluetooth/devices/hci1 E: SYSTEMD_WANTS=bluetooth.target E: TAGS=:systemd: E: USEC_INITIALIZED=29730732 After a reboot with 4.19.10-300.fc29.armv7hl the bluetooth adapter is bond again to "hci0" hcitool dev Devices: hci0 43:45:C0:00:1F:AC And sudo hcitool -i hci1 lescan Invalid device: Network is down I will reboot back to 4.19.9-300.fc29.armv7hl Yours kindly Thanks for attaching the log and for your report.
> After a reboot with 4.19.10-300.fc29.armv7hl the bluetooth adapter is bond
> again to "hci0"
OK. There should be journalctl log for when the bluetooth device was on hci1. Finding it may take some experimenting, but you can start with this:
$ journalctl -g 'Linux version|hci0|hci1' # That will grep all journalctl logs for any of the three strings.
Note the timestamp for the earliest appearance of hci1 after a 4.19.10 boot.
Find the corresponding boot id:
$ journalctl --list-boots # This lists all boots with boot ids and timestamps.
Copy the 32 hex-digit boot id for the timestamp and paste it as an argument to:
$ journalctl -b nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn > journalctl-2.log
Attach that log.
NB: "last | less" is a good way to find timestamps and kernel versions.
Documentation: "man journalctl".
Created attachment 1517249 [details]
output dmesg with Bluetooth connected to hci1
I think this was the first time the bluetooth adapter was assigned to: hci1
(In reply to Roel from comment #7) > Created attachment 1517249 [details] > output dmesg with Bluetooth connected to hci1 > > I think this was the first time the bluetooth adapter was assigned to: hci1 Thanks, Roel. I don't know if this is evidence of a race, but the order differs here: $ egrep 'HCI UART driver|Generic Bluetooth SDIO driver' journalctl-bluetooth-hci0.log Dec 18 00:32:37 kernel: Bluetooth: HCI UART driver ver 2.3 Dec 18 00:32:37 kernel: Bluetooth: Generic Bluetooth SDIO driver ver 0.1 $ egrep 'HCI UART driver|Generic Bluetooth SDIO driver' journalctl-bluetooth-hci1.log Dec 18 00:32:36 kernel: Bluetooth: Generic Bluetooth SDIO driver ver 0.1 Dec 18 00:32:36 kernel: Bluetooth: HCI UART driver ver 2.3 (In reply to Steve from comment #8) > I don't know if this is evidence of a race, ... We might be able to test that by manually loading the drivers. Could you attach the output from: $ lsmod > lsmod-1.txt Created attachment 1517350 [details]
output of lsmod with 4.19.10-300.fc29.armv7hl
(In reply to Roel from comment #10) > Created attachment 1517350 [details] > output of lsmod with 4.19.10-300.fc29.armv7hl Thanks for attaching the lsmod output. After looking into it, I think it would be better to leave any experiments to someone with more expertise than I have. However, I would suggest putting something like this at the beginning of the bug summary, so it is clear what the platform is: "Raspberry Pi 3B+: ..." (click "edit" next to the bug summary at the top of the page) (In reply to Steve from comment #6) ... > Note the timestamp ... Apparently, the RPi doesn't have a realtime clock: Dec 18 00:32:32 kernel: hctosys: unable to open rtc device (rtc0) That would explain why both logs start with nearly the same timestamp: $ grep 'Booting Linux' *.log journalctl-bluetooth-hci0.log:Dec 18 00:32:32 kernel: Booting Linux on physical CPU 0x0 journalctl-bluetooth-hci1.log:Dec 18 00:32:31 kernel: Booting Linux on physical CPU 0x0 Thank you for looking in to this, and maybe it could be some form of a "race" problem where the question is who gets HCI0 first. For now i will stick to the previous kernel version: 4.19.9-300.fc29.armv7hl which uses only one HCI (HCI0 no extra HCI1) $ ls /sys/class/bluetooth/ hci0 The problem is that with the 4.19.10-300.fc29.armv7hl there is a extra HCI (HCI0 and HCI1). But why does the kernel 4.19.10-300.fc29.armv7hl add a extra HCI? Maybe something is changed the way it loads the modules? Yours kindly Roel (In reply to Roel from comment #13) > Thank you for looking in to this, and maybe it could be some form of a > "race" problem where the question is who gets HCI0 first. > For now i will stick to the previous kernel version: 4.19.9-300.fc29.armv7hl > which uses only one HCI (HCI0 no extra HCI1) > $ ls /sys/class/bluetooth/ > hci0 Thanks for your reply and for updating the bug summary. Could you attach the output for lsmod when booted from 4.19.9-300.fc29.armv7hl: $ lsmod > lsmod-2.txt > The problem is that with the 4.19.10-300.fc29.armv7hl there is a extra HCI > (HCI0 and HCI1). > > But why does the kernel 4.19.10-300.fc29.armv7hl add a extra HCI? > Maybe something is changed the way it loads the modules? > > Yours kindly > Roel Those are good questions. It might be possible to blacklist a kernel module, if we can figure out which one ... If you want to experiment, see the man pages for "modinfo", "modprobe", "rmmod". And see this directory: "/etc/modprobe.d/". This is what I blacklist on my system: $ grep 'blacklist ' /etc/modprobe.d/local.conf blacklist eeepc_wmi blacklist wmi_bmof blacklist wmi A web search for "blacklist kernel module" will find more info. Caution: Removing or blacklisting the wrong module could cause the system to malfunction. Created attachment 1517519 [details]
output of lsmod with 4.19.9-300.fc29.armv7hl
I don't need wifi so I also have disabled added this file
/etc/modprobe.d/raspi-blacklist.conf
with:
blacklist brcmfmac
blacklist brcmutil
(In reply to Roel from comment #15) > Created attachment 1517519 [details] > output of lsmod with 4.19.9-300.fc29.armv7hl > > I don't need wifi so I also have disabled added this file > /etc/modprobe.d/raspi-blacklist.conf > with: > blacklist brcmfmac > blacklist brcmutil Thanks. There are some differences. Here is my procedure: Rename the lsmod files and sort: $ sort lsmod-4.19.9-1.txt > lsmod-4.19.9-1-sort.txt $ sort lsmod-4.19.10-1.txt > lsmod-4.19.10-1-sort.txt Look for diffs (details below). btsdio appears to be new in 4.19.10: $ modinfo btsdio | grep description description: Generic Bluetooth SDIO driver ver 0.1 == $ diff -u0 lsmod-4.19.9-1-sort.txt lsmod-4.19.10-1-sort.txt | grep -v '@' --- lsmod-4.19.9-1-sort.txt 2018-12-30 12:07:39.087703894 -0800 +++ lsmod-4.19.10-1-sort.txt 2018-12-30 12:07:22.159229902 -0800 -bcm2835_dma 20480 1 +bcm2835_dma 20480 2 -bluetooth 491520 26 hci_uart,btintel,btqca,bnep,btbcm +bluetooth 491520 28 btsdio,hci_uart,btintel,btqca,bnep,btbcm -bridge 159744 1 br_netfilter -br_netfilter 24576 0 +brcmfmac 253952 0 +brcmutil 16384 1 brcmfmac +btsdio 16384 0 +cfg80211 626688 1 brcmfmac -inet_diag 20480 2 tcp_diag,udp_diag -ipt_MASQUERADE 16384 1 -llc 16384 2 bridge,stp -nf_conntrack 143360 5 ipt_MASQUERADE,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat +nf_conntrack 143360 4 nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat -nf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat +nf_nat_ipv4 16384 1 iptable_nat -overlay 98304 1 -rfkill 28672 3 bluetooth +rfkill 32768 6 bluetooth,cfg80211 -stp 16384 1 bridge -tcp_diag 16384 0 -udp_diag 16384 0 -xt_addrtype 16384 2 -xt_conntrack 16384 18 +xt_conntrack 16384 15 So the race could be between hci_uart and btsdio, as suggested by the different ordering in Comment 8. Summarizing: $ modinfo hci_uart | grep description description: Bluetooth HCI UART driver ver 2.3 $ modinfo btsdio | grep description description: Generic Bluetooth SDIO driver ver 0.1 $ egrep 'hci_uart|btsdio' lsmod-4.19.10-1.txt btsdio 16384 0 hci_uart 106496 0 btqca 16384 1 hci_uart btbcm 16384 1 hci_uart btintel 16384 1 hci_uart bluetooth 491520 28 btsdio,hci_uart,btintel,btqca,bnep,btbcm The module "btsdio" seems to available for previous kernel versions (for example 4.19.9-300.fc29.armv7hl). But with 4.19.10-300.fc29.armv7hl the module "btsdio" get loaded. Is there a change in the "decision routine" 4.19.10-300.fc29.armv7hl? For now I blacklisted the module "btsdio" and rebooted with kernel 4.19.10-300.fc29.armv7hl. Things are working and there is only one entry in /sys/class/bluetooth/ "hci0". And my ruuvitag sensors are doing fine:-) Thank you for all the research and support this was fun. All the best for 2019! Yours kindly Roel (In reply to Roel from comment #18) > The module "btsdio" seems to available for previous kernel versions (for > example 4.19.9-300.fc29.armv7hl). > But with 4.19.10-300.fc29.armv7hl the module "btsdio" get loaded. Thanks for pointing that out. > Is there a change in the "decision routine" 4.19.10-300.fc29.armv7hl? There is a lookup table that maps various device identifiers to modules. Something like this will show some examples: $ grep btsdio /lib/modules/4.19.10-300.fc29.armv7hl/modules.alias $ grep hci_uart /lib/modules/4.19.10-300.fc29.armv7hl/modules.alias However, modules can be loaded in startup scripts or in .conf files in /etc/modprobe.d/ (see "man modprobe.d"). > For now I blacklisted the module "btsdio" and rebooted with kernel > 4.19.10-300.fc29.armv7hl. > Things are working and there is only one entry in /sys/class/bluetooth/ > "hci0". > And my ruuvitag sensors are doing fine:-) That's good to hear. The perfect workaround is easy to implement and easy to reverse. :-) > Thank you for all the research and support this was fun. > All the best for 2019! > Yours kindly > Roel Likewise. Could you attach the output from this command when booted with 4.19.10: # dtc -I fs /proc/device-tree > dtc-1.txt PhilE (Raspberry Pi Engineer & Forum Moderator) asked for it in a post at raspberrypi.org: https://www.raspberrypi.org/forums/viewtopic.php?p=1404200#p1298259 The documentation is here: $ rpm -qd dtc /usr/share/doc/dtc/manual.txt $ rpm -qi dtc ... Devicetree is a data structure for describing hardware. Rather than hard coding every detail of a device into an operating system, many aspects of the hardware can be described in a data structure that is passed to the operating system at boot time. The devicetree is used by OpenFirmware, OpenPOWER Abstraction Layer (OPAL), Power Architecture Platform Requirements (PAPR) and in the standalone Flattened Device Tree (FDT) form. (In reply to Steve from comment #17) > So the race could be between hci_uart and btsdio, as suggested by the > different ordering in Comment 8. Summarizing: Right, so this means that we likely need something similar to: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=b4cdaba274247c9ac841c6a682c08fa91fb3aa549 For the Raspberry Pi 3B+ bluetooth model If someone can do: cat /sys/bus/sdio/devices/mmc0\:0001\:1/vendor /sys/bus/sdio/devices/mmc0\:0001\:1/device Then that will tell you the vendor and product ids for the sdio wifi/bt on the Raspberry Pi 3B+. If someone can copy and paste the output for that command here, then writing a patch for this should be easy. Created attachment 1517970 [details] sudo dtc -I fs /proc/device-tree > dtc-1.txt I don't get any data in the outputfile dtc-1.txt dnf info dtc Installed Packages Name : dtc Version : 1.4.7 Release : 1.fc29 Arch : armv7hl Size : 264 k Source : dtc-1.4.7-1.fc29.src.rpm Repo : @System From repo : koji-override-0 Summary : Device Tree Compiler URL : https://devicetree.org/ License : GPLv2+ Description : Devicetree is a data structure for describing hardware. Rather than hard coding : every detail of a device into an operating system, many aspects of the hardware : can be described in a data structure that is passed to the operating system at : boot time. The devicetree is used by OpenFirmware, OpenPOWER Abstraction Layer : (OPAL), Power Architecture Platform Requirements (PAPR) and in the standalone : Flattened Device Tree (FDT) form. $cd /sys/bus/sdio/devices/ $ls -l lrwxrwxrwx. 1 root root 0 Dec 18 00:32 mmc1:0001:1 -> ../../../devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1 lrwxrwxrwx. 1 root root 0 Dec 18 00:32 mmc1:0001:2 -> ../../../devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:2 lrwxrwxrwx. 1 root root 0 Dec 18 00:32 mmc1:0001:3 -> ../../../devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:3 $ cd mmc1\:0001\:1 $ ls class device modalias of_node power subsystem uevent vendor For all de device and vendor in the directorys {mmc1:0001:1, mmc1:0001:2, mmc1:0001:3} $ cat device 0xa9a6 0xa9a6 0xa9a6 cat vendor 0x02d0 0x02d0 0x02d0 Roel (In reply to Hans de Goede from comment #21) ... > Right, so this means that we likely need something similar to: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ > commit?id=b4cdaba274247c9ac841c6a682c08fa91fb3aa549 ... Thanks for looking into this, Hans. That link doesn't work, but this appears to be the commit you are referring to: Bluetooth: btsdio: Do not bind to non-removable BCM43341 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b4cdaba274247c9c841c6a682c08fa91fb3aa549 (In reply to Roel from comment #22) > Created attachment 1517970 [details] > sudo dtc -I fs /proc/device-tree > dtc-1.txt > > I don't get any data in the outputfile dtc-1.txt ... Thanks for attaching the error output. Bug 1663054 - dtc "Segmentation fault" on Raspberry Pi 3B+ (In reply to Steve from comment #24) > Thanks for looking into this, Hans. That link doesn't work, but this appears > to be the commit you are referring to: > > Bluetooth: btsdio: Do not bind to non-removable BCM43341 > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/ > ?id=b4cdaba274247c9c841c6a682c08fa91fb3aa549 Yes, somehow an extra a snuck in there. (In reply to Roel from comment #23) > For all de device and vendor in the directorys {mmc1:0001:1, mmc1:0001:2, > mmc1:0001:3} > $ cat device > 0xa9a6 > 0xa9a6 > 0xa9a6 > cat vendor > 0x02d0 > 0x02d0 > 0x02d0 Thanks, ok so that means that this seems to be already fixed in 4.20, with this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=70ecdd3d8b36cd2bb960bc4e7ae047fbb94c031b I've started a 4.19 kernel scratch-build with that commit added as a patch: https://koji.fedoraproject.org/koji/taskinfo?taskID=31789531 ATM this is still building (this takes a couple of hours) when it is done please give this kernel a spin *without* blacklisting the btsdio module and let us know if it fixes this bug. Generic test instructions for installing a kernel from a koji build are here: https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt note you need to do replace the x86_64 in the instructions with aarch64. Ah, I just noticed you are using armv7hl not aarch64, armv7hl scratch-build with the patch added here: https://koji.fedoraproject.org/koji/taskinfo?taskID=31789628 (In reply to Hans de Goede from comment #26) ... > Generic test instructions for installing a kernel from a koji build are here: > https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt note you ... It might be useful to include instructions for removing the test kernel so testers don't encounter conflicts with released kernels. Something like this: After testing, the test kernel packages can be removed with: $ sudo rpm -ev kernel-core-4...x86_64 kernel-modules-4...x86_64 # NB: These are package names, not file names. Tip: All kernel packages can be listed with: $ rpm -qa kernel\* | sort (In reply to Steve from comment #28) ... > After testing, the test kernel packages can be removed with: > > $ sudo rpm -ev kernel-core-4...x86_64 kernel-modules-4...x86_64 # NB: These > are package names, not file names. ... Oops. Testers need to reboot with another kernel first ... (In reply to Steve from comment #29) ... > Oops. Testers need to reboot with another kernel first ... Oops.2: rpm lets you remove the current kernel. dnf is more rigorous: $ uname -r 4.20.0-1.fc30.x86_64 $ sudo dnf remove kernel-core-4.20.0-1.fc30.x86_64 kernel-modules-4.20.0-1.fc30.x86_64 Error: Problem: The operation would result in removing the following protected packages: kernel-core (In reply to Steve from comment #28) > It might be useful to include instructions for removing the test kernel so > testers don't encounter conflicts with released kernels. My kernels always have a local element added to their release fields, e.g. the: https://koji.fedoraproject.org/koji/taskinfo?taskID=31789628 rpms have a version-release of: 4.19.13-300.rhbz1661961.fc29.armv7hl so they won't conflict and once newer kernels are pushed to updates dnf will auto-clean them. So no need for manual removal, which as you're figuring out is actually not that easy. 1) Downloaded: https://kojipkgs.fedoraproject.org//work/tasks/9629/31789629/kernel-core-4.19.13-300.rhbz1661961.fc29.armv7hl.rpm and https://kojipkgs.fedoraproject.org//work/tasks/9629/31789629/kernel-modules-4.19.13-300.rhbz1661961.fc29.armv7hl.rpm 2) $ sudo rpm -ivh --oldpackage kernel*.rpm oke $ less /boot/extlinux/extlinux.conf ... label Fedora (4.19.13-300.rhbz1661961.fc29.armv7hl) 29 (Server Edition) kernel /vmlinuz-4.19.13-300.rhbz1661961.fc29.armv7hl append ro root=UUID=42252c78-88f3-4cb4-8302-f8e8343b6325 LANG=en_US.UTF-8 fdtdir /dtb-4.19.13-300.rhbz1661961.fc29.armv7hl/ initrd /initramfs-4.19.13-300.rhbz1661961.fc29.armv7hl.img ... looking good 3) "Un" blacklist btsdio in modprobe.d: $ less /etc/modprobe.d/raspi-blacklist.conf blacklist brcmfmac blacklist brcmutil #blacklist btsdio 4) $ sudo reboot $ uname -r 4.19.13-300.rhbz1661961.fc29.armv7hl $ lsmod | grep -i btsdio btsdio 16384 0 bluetooth 491520 27 btsdio,hci_uart,btintel,btqca,bnep,btbcm $ ls /sys/class/bluetooth/ hci0 NICE!:-) Thank you very much! Yours Kindly Roel (In reply to Roel from comment #32) Thank-you for your detailed report. And thank-you, Hans, for doing the scratch build. Roel, thank you for testing. I've added the patch fixing this as downstream patch to the Fedora kernels for now. This will be picked up by the next official Fedora kernel build. kernel-headers-4.19.14-200.fc28 kernel-4.19.14-200.fc28 kernel-tools-4.19.14-200.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-337484d88b kernel-headers-4.19.14-300.fc29 kernel-4.19.14-300.fc29 kernel-tools-4.19.14-300.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b0f7a7b74b kernel-4.19.14-300.fc29, kernel-headers-4.19.14-300.fc29, kernel-tools-4.19.14-300.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b0f7a7b74b kernel-4.19.14-200.fc28, kernel-headers-4.19.14-200.fc28, kernel-tools-4.19.14-200.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-337484d88b kernel-4.19.14-200.fc28, kernel-headers-4.19.14-200.fc28, kernel-tools-4.19.14-200.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. kernel-4.19.14-300.fc29, kernel-headers-4.19.14-300.fc29, kernel-tools-4.19.14-300.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. |