Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1874300 - USB devices lost when booting 5.8.4 kernel
Summary: USB devices lost when booting 5.8.4 kernel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 32
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-01 01:11 UTC by Ian Laurie
Modified: 2020-10-02 09:43 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-02 09:43:47 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
sudo journalctl --no-hostname -k > dmesg.txt (deleted)
2020-09-01 01:11 UTC, Ian Laurie
no flags Details
dmesg for 5.7.17-200 (deleted)
2020-09-01 14:10 UTC, Phil
no flags Details
lsusb -t for 5.7.17-200 (deleted)
2020-09-01 14:11 UTC, Phil
no flags Details
dmesg for 5.8.4-200 (deleted)
2020-09-01 14:11 UTC, Phil
no flags Details
lsusb -t for 5.8.4-200 (deleted)
2020-09-01 14:11 UTC, Phil
no flags Details
dmesg from working 5.7.17 kernel (deleted)
2020-09-01 23:38 UTC, Ian Laurie
no flags Details
william.garber@att.net dmesg (deleted)
2020-09-07 21:49 UTC, william.garber
no flags Details
william.garber@att.net dmesg (deleted)
2020-09-07 21:50 UTC, william.garber
no flags Details

Description Ian Laurie 2020-09-01 01:11:52 UTC
Created attachment 1713238 [details]
sudo journalctl --no-hostname -k > dmesg.txt

1. Please describe the problem:

When booting the 5.8.4-200.fc32.x86_64 kernel I have lost 3 USB device ports on a DELL Precicion T5610.

2. What is the Version-Release number of the kernel:

kernel-5.8.4-200.fc32.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

Previous kernel-5.7.17-200.fc32.x86_64 worked fine, problem appeared after an update to kernel-5.8.4-200.fc32.x86_64.  If I boot the older 5.7.17 kernel the USB ports work again.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Booting the system reproduces the issue.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

This doesn't apply, we are just past a Fedora beta branch and Rawhide is still at 5.8.0-1.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Attached.

Comment 1 Ian Laurie 2020-09-01 01:40:36 UTC
I tried 5.8.5-200.fc32.x86_64 downloaded from Koji and it has the same problem.

Comment 2 Phil 2020-09-01 07:14:00 UTC
Same here. I need keyboard input at initrd time to unlock my disks but no usb input devices are available. Works fine with kernel-5.7.17-200.fc32.x86_64.

Comment 3 Hans de Goede 2020-09-01 07:33:18 UTC
(In reply to Phil from comment #2)
> Same here. I need keyboard input at initrd time to unlock my disks but no
> usb input devices are available. Works fine with
> kernel-5.7.17-200.fc32.x86_64.

What hardware are you using ?  (Most people are not seeing this, so I'm trying to find a pattern in which hardware is affected)

Comment 4 Phil 2020-09-01 08:02:07 UTC
(In reply to Hans de Goede from comment #3)
> What hardware are you using ?

I'm using that hardware:

00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

Bus 002 Device 002: ID 8087:8001 Intel Corp. Integrated Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8009 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 05e3:0749 Genesys Logic, Inc. SD Card Reader and Writer
Bus 004 Device 002: ID 05e3:0612 Genesys Logic, Inc. Hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 005: ID 0781:5583 SanDisk Corp. Ultra Fit
Bus 003 Device 003: ID 0582:012a Roland Corp. UM-ONE
Bus 003 Device 008: ID 056e:010d Elecom Co., Ltd M-HT1DRBK HUGE Wireless Optical TrackBall
Bus 003 Device 007: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter (Planet UP-100, Genius G-Talk)
Bus 003 Device 006: ID 0dc6:2011 Precision Squared Technology Corp. <--- that's my keyboard
Bus 003 Device 004: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 003 Device 002: ID 05e3:0610 Genesys Logic, Inc. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Comment 5 Hans de Goede 2020-09-01 08:11:23 UTC
(In reply to Phil from comment #4)
> (In reply to Hans de Goede from comment #3)
> > What hardware are you using ?
> 
> I'm using that hardware:

Thank you, but that was not what I had in mind, what I meant is what manufacturer and product, e.g. "Dell Latitude E6400" are you using ?

Problems like the one from this bug are typically specific to how a specific product uses the involved chipsets,
if it was broken on such a popular chipset regardless of the product using the chipset this problem would
have been caught before the new kernel hit the updates repository.

Comment 6 Phil 2020-09-01 08:21:43 UTC
(In reply to Hans de Goede from comment #5)
> Thank you, but that was not what I had in mind, what I meant is what
> manufacturer and product, e.g. "Dell Latitude E6400" are you using ?

Oh sorry.
It's a custom desktop pc with a fairly old Gigabyte H97-D3H board w/i5-4690, using the onboard usb controller. The latest bios updates (AMI), version f7, are applied.

Comment 7 Hans de Goede 2020-09-01 08:41:12 UTC
Hmm, so that means this is happening on 2 quite different devices, what fun.

I've been looking at the kernel history to see if anything stands out, but I've not spotted anything.

I see that you have both XHCI and EHCI controllers.

Can you add the output of "lsusb -t" with the working kernel; as well as attach a dmesg for the working kernel ?

Comment 8 Phil 2020-09-01 14:10:38 UTC
Created attachment 1713312 [details]
dmesg for 5.7.17-200

Comment 9 Phil 2020-09-01 14:11:09 UTC
Created attachment 1713313 [details]
lsusb -t for 5.7.17-200

Comment 10 Phil 2020-09-01 14:11:36 UTC
Created attachment 1713314 [details]
dmesg for 5.8.4-200

Comment 11 Phil 2020-09-01 14:11:55 UTC
Created attachment 1713315 [details]
lsusb -t for 5.8.4-200

Comment 12 Ian Laurie 2020-09-01 23:35:48 UTC
@Hans this needs some explaining.  I booted 5.7.17 which works, and did the "lsusb -t" on the working system, but I had already moved all devices to USB ports that work in 5.8.4 (ie nothing plugged into the ports 5.8.4 doesn't like).  This is the result:

zuke$ uname -r
5.7.17-200.fc32.x86_64
zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 5: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

I then moved the mouse to a port that is non-functional in 5.8.4 (but works fine in 5.7.17), this is what I got:

zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 5: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

I then moved the keyboard to a port that is non-functional in 5.8.4, this is what I got:

zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Should there be 2 lines that say Port 3 on Bus 03 ?????  In any case this was in 5.7.17 and regardless of the odd looking listing, everything does work.

Comment 13 Ian Laurie 2020-09-01 23:38:13 UTC
Created attachment 1713383 [details]
dmesg from working 5.7.17 kernel

Comment 14 Hans de Goede 2020-09-02 08:23:17 UTC
Phil,

(In reply to Phil from comment #10)
> Created attachment 1713314 [details]
> dmesg for 5.8.4-200

Thank you, I don't see any missing usb-devices here vs the 5.7 kernel ?  Are you sure this is right ?  I guess it is possible that devices are detected but not working. You say that 3 ports stopped working, can you describe which 3 devices have stopped working ?

Comment 15 Hans de Goede 2020-09-02 08:45:04 UTC
Ian,

(In reply to Ian Laurie from comment #12)
> @Hans this needs some explaining.  I booted 5.7.17 which works, and did the
> "lsusb -t" on the working system, but I had already moved all devices to USB
> ports that work in 5.8.4 (ie nothing plugged into the ports 5.8.4 doesn't
> like).  This is the result:
<snip>

Thank you for the info. So it seems that the 5.8.4 kernel has an issue
with ports connected to the xhci controller, where as the ehci ports
work fine, right ?

> I then moved the mouse to a port that is non-functional in 5.8.4 (but works
> fine in 5.7.17), this is what I got:

Oh interesting, so the mouse (and later also the kbd) does get detected, as with
Phil's "lsusb -t" output, but they don't send events.

So the common pattern seems to be that:
1. You both have a "9 Series Chipset Family USB xHCI Controller"
2. You both have a lo-speed (1.5M) keyboard and mouse connected to that
XHCI controller
3. The kbd/mouse gets detected but does not send events. So there seems
to be an issue with lo-speed USB interrupt-transfers not working on the
"9 Series Chipset Family USB xHCI Controller".

I will report this upstream and then we will see from there.

> Should there be 2 lines that say Port 3 on Bus 03 ?????  In any case this
> was in 5.7.17 and regardless of the odd looking listing, everything does
> work.

That is normal your keyboard registers itself as 2 HID interfaces (likely one for the standard keys and one for the multimedia keys) and each interface gets its own line, if you look closely:

    |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M

You will see yhat it is Dev (USB address) 5 which gets listed twice, with "If 0", resp. "If 1". So yeah this part of "lsusb -t"'s output is a bit weird if you are not used to it I guess. But this is normal and this way of outputting things can actually be very helpful when debugging stuff.

Comment 16 Phil 2020-09-02 08:54:19 UTC
Hi Hans,


the lsusb is from when the system has finished booting and the gdm login screen is shown.

My problem is that the usb devices are missing while running from initrd.

When the PC powers on, all sata disks are security-locked. I'm using a dracut module that asks for the passphrases (read -s -r tmp < /dev/console), unlocks the disks, triggers a rescan and then continues booting. With kernel 5.8.4-200 I cannot enter my passphrase because at that point the keyboard is dead. With 5.7.17-200 it worked fine.

Now I thought that there might be a module missing in the initrd, but alas:

 $ diff <(lsinitrd /boot/initramfs-5.7.17-200.fc32.x86_64.img | sed -r 's|[^/]+fc32\.x86_64|KVER|g' | cut -c 56-) \
   <(lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | sed -r 's|[^/]+fc32\.x86_64|KVER|g' | cut -c 56-)
 1931,1933d1930
 < usr/lib/modules/KVER/kernel/crypto
 < usr/lib/modules/KVER/kernel/crypto/ecc.ko.xz
 < usr/lib/modules/KVER/kernel/crypto/ecdh_generic.ko.xz
 2084a2082,2085
 > usr/lib/modules/KVER/kernel/drivers/media
 > usr/lib/modules/KVER/kernel/drivers/media/cec
 > usr/lib/modules/KVER/kernel/drivers/media/cec/core
 > usr/lib/modules/KVER/kernel/drivers/media/cec/core/cec.ko.xz
 2106a2108
 > usr/lib/modules/KVER/kernel/drivers/pinctrl/intel/pinctrl-jasperlake.ko.xz
 2127a2130
 > usr/lib/modules/KVER/kernel/drivers/tty/serial/lantiq.ko.xz
 2130a2134,2135
 > usr/lib/modules/KVER/kernel/drivers/usb/host/xhci-pci.ko.xz
 > usr/lib/modules/KVER/kernel/drivers/usb/host/xhci-pci-renesas.ko.xz

Comment 17 Hans de Goede 2020-09-02 09:30:14 UTC
Phil,

(In reply to Phil from comment #16)
> the lsusb is from when the system has finished booting and the gdm login
> screen is shown.
> 
> My problem is that the usb devices are missing while running from initrd.
> 
> When the PC powers on, all sata disks are security-locked. I'm using a
> dracut module that asks for the passphrases (read -s -r tmp < /dev/console),
> unlocks the disks, triggers a rescan and then continues booting. With kernel
> 5.8.4-200 I cannot enter my passphrase because at that point the keyboard is
> dead. With 5.7.17-200 it worked fine.

I do see in the dmesg for 5.8 that for some reason 5.8 takes significantly
longer to initialize the USB stack. What happens if you boot 5.8 and then
wait 10 seconds before trying to type the passphrase?

And the keyboard does work once you are in gdm?  Also how did you get into
gdm with 5.8 if you cannot enter the passphrase?

Comment 18 Phil 2020-09-02 09:51:59 UTC
Hans,


(In reply to Hans de Goede from comment #17)
> What happens if you boot 5.8 and then
> wait 10 seconds before trying to type the passphrase?

I'll give it a try.

> And the keyboard does work once you are in gdm?  Also how did you get into
> gdm with 5.8 if you cannot enter the passphrase?

It's simple: Boot with 5.7.17-200, enter passphrases, press sysrq-b, boot with 5.8.4-200 :)

Comment 19 Hans de Goede 2020-09-02 11:40:40 UTC
(In reply to Phil from comment #18)
> It's simple: Boot with 5.7.17-200, enter passphrases, press sysrq-b, boot
> with 5.8.4-200 :)

Nifty. And then once at gdm the keyboard and mouse do work?

Comment 20 Phil 2020-09-02 12:41:20 UTC
(In reply to Hans de Goede from comment #19) 
> Nifty. And then once at gdm the keyboard and mouse do work?

Yes, when the system is booted 

Just for fun I added a "sleep 10; lsusb -t >&2" to the initrd and that's all I get:

 /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
     |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
 /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
     |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Bus 03 to 06 are missing. So ehci-pci is working, xhci_hcd not.

Also, at that point the module is not loaded. The difference between kernel 5.7.17-200 and 5.8.4-200 config is:

 $ diff <(grep -i 'usb_.hci' config-5.7.17-200.fc32.x86_64) <(grep -i 'usb_.hci' config-5.8.4-200.fc32.x86_64 )
 4c4,5
 < CONFIG_USB_XHCI_PCI=y
 ---
 > CONFIG_USB_XHCI_PCI=m
 > CONFIG_USB_XHCI_PCI_RENESAS=m

Do I need some rdrdinsmodpost magic?

Comment 21 Hans de Goede 2020-09-02 14:54:46 UTC
(In reply to Phil from comment #20)
> (In reply to Hans de Goede from comment #19) 
> > Nifty. And then once at gdm the keyboard and mouse do work?
> 
> Yes, when the system is booted 
> 
> Just for fun I added a "sleep 10; lsusb -t >&2" to the initrd and that's all
> I get:
> 
>  /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
>      |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
>  /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
>      |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
> 
> Bus 03 to 06 are missing. So ehci-pci is working, xhci_hcd not.
> 
> Also, at that point the module is not loaded. The difference between kernel
> 5.7.17-200 and 5.8.4-200 config is:
> 
>  $ diff <(grep -i 'usb_.hci' config-5.7.17-200.fc32.x86_64) <(grep -i
> 'usb_.hci' config-5.8.4-200.fc32.x86_64 )
>  4c4,5
>  < CONFIG_USB_XHCI_PCI=y
>  ---
>  > CONFIG_USB_XHCI_PCI=m
>  > CONFIG_USB_XHCI_PCI_RENESAS=m

Hmm, I think that that as an unintended change, I will discuss this with the Fedora kernel team.

> Do I need some rdrdinsmodpost magic?

First thing would be to run lsinitrd on the 5.8 initrd, e.g.:

sudo lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | grep xhci

And check if the xhci-pci module is there.

If it is there then the issue is the custom dracut magic you added for disk unlocking runs before udev gets started and modprobes the module; if it is not there, then this might be considered a dracut issue. I say might because I believe that the changing of the xhci-pci driver to no longer be builtin was unintentional. So the most likely fix here is to undo that change.

Comment 22 Hans de Goede 2020-09-02 15:17:51 UTC
Phil, Ian,

I've just started a scratchbuild of 5.8.5 with the XHCI PCI driver built into the kernel again (or at least it should be).
Can you both give this a try please?

ATM it is still building, this should be done in a couple of hours:
https://koji.fedoraproject.org/koji/taskinfo?taskID=50639874

See here for some generic instructions for testing kernels directly from koji:
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Note you need to (temporarily) disable secureboot to test this kernel, either in your BIOS or run:

sudo mokutil --disable-validation

And follow the instructions.

Comment 23 Hans de Goede 2020-09-02 17:26:59 UTC
FYI the scratch-build of 5.8.5 with XHCI builtin again has just completed building.

Comment 24 Phil 2020-09-02 18:39:54 UTC
Hi Hans,

(In reply to Hans de Goede from comment #21)

> First thing would be to run lsinitrd on the 5.8 initrd, e.g.:
> 
> sudo lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | grep xhci
> 
> And check if the xhci-pci module is there.

yep, as shown in https://bugzilla.redhat.com/show_bug.cgi?id=1874300#c16 the module is there.

> If it is there then the issue is the custom dracut magic you added for disk
> unlocking runs before udev gets started and modprobes the module;

Maybe. The priority of that dracut module is 6, I'll try setting it to a higher value.

Thanks a lot!

Comment 25 Hans de Goede 2020-09-02 19:29:14 UTC
Note the Fedora kernel team has agreed to set the XHCI module to be builtin again for the upcoming 5.8.6 kernel (whenever that comes out), matching what I've done for the 5.8.5 test / scratch-build.

Comment 26 Ian Laurie 2020-09-02 23:31:18 UTC
Unfortunately the test kernel hasn't worked for me.  Here is  the "lsusb -t" from it:

zuke$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 5: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.8.5-200.rhbz1874300.fc32.x86_64
zuke$

Comment 27 Ian Laurie 2020-09-02 23:50:10 UTC
With everything plugged in the same way as comment #26, the working kernel looks like this:

zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.7.17-200.fc32.x86_64
zuke$

Comment 28 Phil 2020-09-03 06:16:59 UTC
Hans,

thanks!

Unfortunately, the 5.8.5-200 kernel didn't help as xhci_pci is still compiled in as a module:

 $ grep XHCI_PCI /boot/config-5.8.5-200.rhbz1874300.fc32.x86_64 
 CONFIG_USB_XHCI_PCI=m
 CONFIG_USB_XHCI_PCI_RENESAS=m

Oh, regarding the dracut module's priority: I believe this issue also might affect users with LUKS-encrypted root devices, as the 'crypt' module gets started before the 'kernel-modules' module.

 $ ls -1d /lib/dracut/modules.d/*{crypt,kernel-modules}
 /lib/dracut/modules.d/90crypt
 /lib/dracut/modules.d/90kernel-modules

Comment 29 Phil 2020-09-03 06:32:38 UTC
FWIW – rdloaddriver doesn't help as that boot argument is being evaluated too late (S90kernel-modules). It might help Ian, though.
The only working solution for me so far (apart from compiling a custom kernel) is to include a 'modprobe xhci_pci' in my dracut module.

Comment 30 Hans de Goede 2020-09-03 07:21:41 UTC
Ugh, I forgot to run "make config-files" which is necessary when changing the Fedora kernel config templates, sorry.

Here is a new scratch-build where I did actually run "make config-files":

https://koji.fedoraproject.org/koji/taskinfo?taskID=50680004

Can you both please give this a try (once it has completed building) ?

Comment 31 Hans de Goede 2020-09-03 09:17:54 UTC
FWIW, the new, hopefully fixed, scratch-build is done now.

Comment 32 Phil 2020-09-03 09:22:35 UTC
Thanks Hans, the new build fixed it :)

Comment 33 Hans de Goede 2020-09-03 09:27:29 UTC
Phil, That is good to hear.

Ian, can you give this new build a try to please?

Ian, also I wonder if your problem like Phil's was with entering a password for disk-encryption at boot, or were you seeing issues once fully logged in too ?  The xhci driver being a module should not cause issues once the boot is past the initrd-phase (it shouldn't but this still is the most likely cause of your issues too).

Comment 34 Ian Laurie 2020-09-03 09:40:24 UTC
Hans,

I do use LUKS on some VMs I share between work and home in case my car get ripped off with VMs on a portable drive, but this issue is on real metal and I am not using any encryption.  Sadly we're victim of timezones, the issue I have is on a work machine and I'm home for the night.  Will test as soon as I get in.

Comment 35 Ian Laurie 2020-09-03 23:53:11 UTC
It still isn't working for me.  I booted with the keyboard and mouse plugged into the sockets that don't work in 5.8.4.  At the Grub kernel menu the keyboard works, and the LED on the mouse is on, but after I boot the keyboard and mouse were dead, and the LED on the mouse was off.

Plugged both into working sockets and this is the result:

zuke$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 5: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 6: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.8.5-200.rhbz1874300_2.fc32.x86_64
zuke$

zuke$ rpm -qa | grep rhbz1874300
kernel-modules-5.8.5-200.rhbz1874300_2.fc32.x86_64
kernel-devel-5.8.5-200.rhbz1874300_2.fc32.x86_64
kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64
kernel-modules-extra-5.8.5-200.rhbz1874300_2.fc32.x86_64
kernel-core-5.8.5-200.rhbz1874300_2.fc32.x86_64
zuke$

Comment 36 Ian Laurie 2020-09-04 01:09:40 UTC
Hans, I should point out this problem is not a show stopper for me as it only takes out 3 USB ports on the back panel, they just happened by coincidence to be the ports I was using for the mouse and keyboard, so when the 5.8.4 kernel arrived it broke the box until I worked out what was going on.  Technically I can live without these 3 ports, but obviously some other people may lose more than 3.

Does upstream have any ideas?  Willing to try more things if you want to pursue this.

Comment 37 Fedora Update System 2020-09-04 12:35:51 UTC
FEDORA-2020-708b23f2ce has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce

Comment 38 Fedora Update System 2020-09-04 15:28:39 UTC
FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-708b23f2ce`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 39 Ian Laurie 2020-09-07 00:55:22 UTC
What happened between kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64 and kernel-5.8.6-201.fc32.x86_64 USB wise?

zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.8.6-201.fc32.x86_64
zuke$ 

All my USB ports are back !!

Keyboard and mouse are back in the previously unworking ports in the listing above.

Comment 40 Hans de Goede 2020-09-07 07:46:55 UTC
(In reply to Ian Laurie from comment #39)
> What happened between kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64 and
> kernel-5.8.6-201.fc32.x86_64 USB wise?

The xhci controller driver got moved back to being built-in instead of being build as a module.

> All my USB ports are back !!
> 
> Keyboard and mouse are back in the previously unworking ports in the listing
> above.

That is good to know.

From Fedora's pov that means that this bug is resolved now, as the building of the XHCI driver as a module was done by accident and the plan is to keep it built-in.

The updates system will close this bug once kernel-5.8.6-201.fc32.x86_64 hits the stable updates repository.

I will follow-up with upstream about the ports disappearing when xhci is built as a module, as that should not happen.

Comment 41 Hans de Goede 2020-09-07 07:51:26 UTC
Ian one last question, upon reading your lsusb output, I now see that you did the lsusb-s all with the 5.7.17 kernel, right?

What is the output of "lsusb -t" with the mouse/kbd in the troublesome ports and a non working kernel ?

Comment 42 Hans de Goede 2020-09-07 07:56:40 UTC
Ian, one more question, do you have a usb device with a LED which always lights up when there is power (the red light at the bottom of a mouse typically turns off when not enumerated over USB) ? And if so can you try this in one of the trouble some ports with a non-working kernel?  I think the 5V on the ports gets turned off when the xhci driver is built as a module.

Comment 43 Ian Laurie 2020-09-07 10:30:07 UTC
(1) Comment #12, comment #27 and comment #39 were done with working kernels, the kernel appears in my "uname -r" in the comment.  Comment #26 and comment #35 were non-working kernels, again with a "uname -r" showing which one it was.

(2) I can test this in the morning, non-working kernel, keyboard and mouse in non-functional ports, I will get the "lsusb -t" over ssh.

(3) I don't have a device to confirm if power is removed from a non-working port, but I can test if the 5V is there with a multimeter.  Again I can do this in the morning.

Comment 44 Fedora Update System 2020-09-07 17:14:11 UTC
FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 45 william.garber 2020-09-07 21:46:31 UTC
garberw@electron> uname -a
Linux electron.localdomain 5.8.4-200.fc32.x86_64 #1 SMP Wed Aug 26 22:28:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I have two usb3.0gen1 cards with 4 ports each.
model sunix  USB4300NS
chipset renesas/nec uPD720201

I get this in dmesg:
[    2.933892] xhci_hcd 0000:17:00.0: FW has invalid version :8224
[    2.933908] xhci_hcd 0000:17:00.0: Direct firmware load for renesas_usb_fw.mem failed with error\
 -2
[    2.933910] xhci_hcd 0000:17:00.0: request_firmware failed: -2
[    2.933988] xhci_hcd: probe of 0000:17:00.0 failed with error -2
[    2.937826] xhci_hcd 0000:18:00.0: FW has invalid version :8224
[    2.937838] xhci_hcd 0000:18:00.0: Direct firmware load for renesas_usb_fw.mem failed with error\
 -2
[    2.937840] xhci_hcd 0000:18:00.0: request_firmware failed: -2
[    2.937910] xhci_hcd: probe of 0000:18:00.0 failed with error -2

root@electron# lsusb -t
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 1: Dev 2, If 1, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 1: Dev 2, If 0, Class=Printer, Driver=usblp, 480M
    |__ Port 7: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 7: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 8: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 8: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M
root@electron# 

root@electron# lsmod | grep xhci
xhci_pci               20480  0
xhci_pci_renesas       20480  1 xhci_pci
root@electron# 

the ports no longer work.  
rebooting into kernel 5.7.17-200.fc32.x86_64
they work again indicating sunix usb3 cards and
connected external hard drive and usb3 enclosure are not broken.
so it must be something to do with the kernel.

this seems to be a report of something similar:
https://bbs.archlinux.org/viewtopic.php?id=258184


root@electron# fdisk -l
Disk /dev/sdb: 223.58 GiB, 240057409536 bytes, 468862128 sectors
Disk model: SSD2SC240G1SA754
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7639e137

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdb1  *         2048   1026047   1024000   500M  7 HPFS/NTFS/exFAT
/dev/sdb2         1026048 467868301 466842254 222.6G  7 HPFS/NTFS/exFAT
/dev/sdb3       467869696 468856831    987136   482M 27 Hidden NTFS WinRE


Disk /dev/sda: 476.96 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: Crucial_CT512MX1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 756AA145-ECFC-4044-8BBF-4AF4E15F61BB

Device       Start        End   Sectors  Size Type
/dev/sda1     2048    2099199   2097152    1G Linux filesystem
/dev/sda2  2099200    6293503   4194304    2G EFI System
/dev/sda3  6293504 1000214527 993921024  474G Linux LVM


Disk /dev/mapper/main-root: 55 GiB, 59055800320 bytes, 115343360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/main-00: 48 GiB, 51539607552 bytes, 100663296 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/main-home: 355.96 GiB, 382184980480 bytes, 746455040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/main-var: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@electron# 

fdisk -l shows disk was not recognized (/dev/sdc missing).

attach dmesg.

Comment 46 william.garber 2020-09-07 21:49:14 UTC
Created attachment 1714009 [details]
william.garber  dmesg

dmesg invalid firmware

Comment 47 william.garber 2020-09-07 21:50:06 UTC
Created attachment 1714010 [details]
william.garber  dmesg

dmesg bug about firmware

Comment 48 Ian Laurie 2020-09-07 23:42:28 UTC
Late yesterday I converted the box over to Fedora 33 beta branch, however the initial kernel is 5.8.3 which is problematic in Fedora 33 as well, so it shouldn't disturb the results.

From comment #41

>What is the output of "lsusb -t" with the mouse/kbd in the troublesome ports and a non working kernel ?

zuke$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.8.3-300.fc33.x86_64
zuke$

For the above, keyboard/mouse in non-functioning ports, mouse LED is off, results obtained through SSH, kernel-5.8.3-300.fc33.x86_64 from Fedora 33 beta branch.

Comment 49 Ian Laurie 2020-09-08 00:04:32 UTC
From comment #42

>I think the 5V on the ports gets turned off when the xhci driver is built as a module.

Not the results I was expecting, but I have 5V on the problematic ports, all 3 of them, even though the keyboard and mouse appear dead when plugged into them.

Comment 50 Ian Laurie 2020-09-08 00:10:49 UTC
For the sake of completeness here is an "lsusb -t" from the [now] default kernel on the box which is 5.8.6-301:

zuke$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 3: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
zuke$ uname -r
5.8.6-301.fc33.x86_64
zuke$

For this listing the keyboard and mouse were in the same usb sockets that are non-functional in the problematic kernels.  Everything works well in kernel-5.8.6-301.fc33.x86_64.  In other words, the only difference between this listing and the one in comment #49 is the kernel.

Comment 51 william.garber 2020-09-08 00:27:00 UTC
possibly this uefi setting could help?  did not help me.

https://askubuntu.com/questions/729558/how-do-i-get-usb-3-0-driver-working-or-check-that-it-is-already-working

reposting from there:

it's very simple, i struggled with this issue using Ubuntu and Ubuntu flavored distros for years
(Mint, Elementary OS, etc). 
Go back into bios, have usb 3.0 turned on, an any other options turned on, but turn off legacy usb option.
The description of legacy usb is that if you have it off, that will disable it for any os that's not "usb aware". 
But I thought flip the switch, because it's 2018 and most os's are usb aware now. 
It wasn't supposed to work, but it fixed the issue that has baffled me for years.
My usb 3.0 works perfectly now. 
My theory is that usb legacy conflicts with the os's understanding of 3.0, so now there's no conflict. 
If it works for you, you're welcome.

I googled this much, and no one else seemed to have tried or had the same conclusion. 
I hope this helps others who struggled with it.

Comment 52 william.garber 2020-09-08 02:04:11 UTC
garberw@electron> uname -a
Linux electron.localdomain 5.8.6-201.fc32.x86_64 #1 SMP Fri Sep 4 03:27:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

at 2:30pst kernel was upgraded and now it works.
external hard disks automounted.
no error message about firmware in dmesg.

Comment 53 Hans de Goede 2020-09-08 08:50:29 UTC
(In reply to Ian Laurie from comment #48)
> zuke$ lsusb -t
> /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
>     |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
> /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
>     |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
> zuke$ uname -r
> 5.8.3-300.fc33.x86_64

Ok, thank you for all your patience in getting to the bottom of this.

So the problem is the xhci driver not loading / binding to the xhci-controller at all when not building as a module. Now that I know that, if I go back to your original dmesg (first attachment, 5.8.4 kernel) I see:

Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: FW has invalid version :8216
Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: Direct firmware load for renesas_usb_fw.mem failed with error -2
Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: request_firmware failed: -2
Sep 01 10:47:36 kernel: xhci_hcd: probe of 0000:05:00.0 failed with error -2

SO I now see that you are using a Renesas XHCI controller. I missed that at first, because I was looking at the "0000:00:05.0" pci-device-ids, where as your xhci controller is the "0000:05:00.0" pci-device, my bad.

So you seem to have the same problem as william.garber, which is the new renasas xhci controller firmware loading support breaking things.

The issue here is that the version check in the upstream code was no good, this has been fixed in this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d66a57be2f9a315fc10d0f524f670fec903e0fb4

This fix is available in the 5.8.6 kernel, which is in updates testing now, quoting from comment 38 :

FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-708b23f2ce`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Ian, you have already confirmed that 5.8.6 fixes things and the above actually explains why it fixes things, so I believe the issue on your system is fully resolved now.

William, can you test the 5.8.6 update from updates-testing (see above)? That should fix this for you too.

Comment 54 william.garber 2020-09-08 13:38:40 UTC
see comment 52.  it is working now.  fixed in 5.8.6-201.fc32.x86_64 
awesome; thank you :-)

Comment 55 Ian Laurie 2020-09-08 22:05:20 UTC
>Ian, you have already confirmed that 5.8.6 fixes things and the above actually explains why it fixes things...

The best conclusions are when in the end we understand fully the failure and fix scenario, this is a great result, thank you Hans.

Comment 56 bryanhoop 2020-09-11 15:18:25 UTC
I just ran into this with kernel 5.8.7. 5.8.6 works great, but USB devices stop working on 5.8.7.

Comment 57 Hans de Goede 2020-09-11 15:51:28 UTC
(In reply to bryanhoop from comment #56)
> I just ran into this with kernel 5.8.7. 5.8.6 works great, but USB devices
> stop working on 5.8.7.

This sounds like a different issue to me, please file a new bug for this.

In this new bug please attach the following:

1. dmesg output of working kernel
2. lsusb output on working kernel
3. "lsusb -t" output on working kernel
4. dmesg output of non-working kernel
5. "lsusb -t" output on non-working kernel

Comment 58 Ian Laurie 2020-10-02 02:27:51 UTC
I think this can be closed.

Comment 59 Hans de Goede 2020-10-02 09:43:47 UTC
(In reply to Ian Laurie from comment #58)
> I think this can be closed.

Agreed, closing.


Note You need to log in before you can comment on or make changes to this bug.