Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2063241

Summary: After update, ipp-usb package took over my printer
Product: [Fedora] Fedora Reporter: Nadav Har'El <nyh>
Component: golang-github-openprinting-ipp-usbAssignee: Zdenek Dohnal <zdohnal>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 35CC: zdohnal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-18 20:06:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nadav Har'El 2022-03-11 15:08:08 UTC
On a desktop system where I've been updating Fedora all the time for several years, a few days ago suddenly my connected HP printer (HP DeskJet 4675) stopped printing. The jobs hung in the queue, and journalctl showed errors like:

Mar 11 16:43:56 pooh hp[6292]: io/hpmud/musb.c 769: invalid deviceid ret=-9: Resource temporarily unavailable

Since nothing changed in my hardware or my wiring or configuration, there was no reason why the HP printer should suddenly have become unavailable, so I had a hunch that some irrelevant package took over this USB device. And sure enough, it seems the ipp-usb package did! Removing that package (rpm -e ipp-usb) made everything work again.

The strange thing is that this a very recent regression - a few days ago I was printing just fine. I didn't install the "ipp-usb" package myself, nor change anything in my printer configuration, so something must have happened during a recent dnf update.

Apparently I'm not the only one seeing this - a Google search also turned out:

https://ask.fedoraproject.org/t/ipp-usb-has-snatched-my-printer/20458

I'm guessing this ipp-usb thing is deliberate (see also https://lwn.net/Articles/857502/), but if it can't make a printer work correctly, it shouldn't take it over... First of all, my printer configuration wasn't using it, it was using the hpcups drivers (from the hplip package). Second, even when I did try with system-config-printer to choose the "driver-less" driver - presumably through ipp-usb), it didn't work.

Comment 1 Fedora Update System 2022-03-14 09:12:42 UTC
FEDORA-2022-016b7d39db has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-016b7d39db

Comment 2 Fedora Update System 2022-03-14 09:25:25 UTC
FEDORA-2022-352f2e3280 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-352f2e3280

Comment 3 Zdenek Dohnal 2022-03-14 12:52:37 UTC
Hi Nadav,

thank you for reporting the issue!

I'd added ipp-usb as a weak dependency of CUPS, which I've reverted for Fedora 35 and 34 now. The ipp-usb will be pulled in only for F36+ and it should affect only devices which support IPP-over-USB - otherwise ipp-usb doesn't start and won't take over the port.

It would be great if you attached output of `lsusb -v` command when your device is plugged in and turned on - it will give me info whether your device is really capable of IPP-over-USB and ipp-usb is really doing what is expected.

(In reply to Nadav Har'El from comment #0)
> I'm guessing this ipp-usb thing is deliberate (see also
> https://lwn.net/Articles/857502/), but if it can't make a printer work
> correctly, it shouldn't take it over...

Since your snap of the log shows an error from hp backend, I guess you've printed to the print queue created with hplip backend+driver - ipp-usb doesn't take over the old hplip queue, but unfortunately blocks it by occupying device's port. ipp-usb creates a new print queue - by default temporary - which you can see in print dialog which supports temp queues right away (f.e. in Libreoffice or any GTK dialog) or you can set a permanent queue like:

$ lpadmin -p queue -v ipp://localhost:60000/ipp/print -m everywhere -E

So the printing should work correctly once you send the job to the correct queue, which is generated by ipp-usb.

> First of all, my printer
> configuration wasn't using it, it was using the hpcups drivers (from the
> hplip package). Second, even when I did try with system-config-printer to
> choose the "driver-less" driver - presumably through ipp-usb), it didn't
> work.

Unless you print quite often and printer default options don't suit you, you don't need to install your USB printer. ipp-usb advertises it on localhost and modern print dialogs are able to pick it up and print to it.

Comment 4 Fedora Update System 2022-03-14 23:13:58 UTC
FEDORA-2022-352f2e3280 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-352f2e3280`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-352f2e3280

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2022-03-14 23:44:35 UTC
FEDORA-2022-016b7d39db has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-016b7d39db`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-016b7d39db

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Nadav Har'El 2022-03-15 07:51:59 UTC
(In reply to Zdenek Dohnal from comment #3)
> I'd added ipp-usb as a weak dependency of CUPS, which I've reverted for
> Fedora 35 and 34 now.
> The ipp-usb will be pulled in only for F36+ and it


Thanks, however I assume this reversion will not help users who already did "dnf update" and got ipp-usb.
Also, we'll eventually have exactly the same issue for people upgrading to Fedora 36.

> should affect only devices which support IPP-over-USB - otherwise ipp-usb
> doesn't start and won't take over the port.
> 
> It would be great if you attached output of `lsusb -v` command when your
> device is plugged in and turned on - it will give me info whether your
> device is really capable of IPP-over-USB and ipp-usb is really doing what is
> expected.

Bus 003 Device 011: ID 03f0:da11 HP, Inc DeskJet 4670 series
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x03f0 HP, Inc
  idProduct          0xda11 
  bcdDevice            1.00
  iManufacturer           1 HP
  iProduct                2 DeskJet 4670 series
  iSerial                 3 TH61R3H0XZ0663
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x00b1
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    204 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               7
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       1
      bNumEndpoints           2
      bInterfaceClass         7 Printer
      bInterfaceSubClass      1 Printer
      bInterfaceProtocol      4 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         7 Printer
      bInterfaceSubClass      1 Printer
      bInterfaceProtocol      2 Bidirectional
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x08  EP 8 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x89  EP 9 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       1
      bNumEndpoints           2
      bInterfaceClass         7 Printer
      bInterfaceSubClass      1 Printer
      bInterfaceProtocol      4 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x08  EP 8 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x89  EP 9 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      4 
      bInterfaceProtocol      1 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x85  EP 5 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      4 
      bInterfaceProtocol      1 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x0a  EP 10 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x8b  EP 11 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       1
      bNumEndpoints           2
      bInterfaceClass         7 Printer
      bInterfaceSubClass      1 Printer
      bInterfaceProtocol      4 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x0a  EP 10 OUT
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x8b  EP 11 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered


> Since your snap of the log shows an error from hp backend, I guess you've
> printed to the print queue created with hplip backend+driver - ipp-usb
> doesn't take over the old hplip queue, but unfortunately blocks it by
> occupying device's port. ipp-usb creates a new print queue - by default
> temporary - which you can see in print dialog which supports temp queues
> right away (f.e. in Libreoffice or any GTK dialog) or you can set a
> permanent queue like:
> 
> $ lpadmin -p queue -v ipp://localhost:60000/ipp/print -m everywhere -E
> 
> So the printing should work correctly once you send the job to the correct
> queue, which is generated by ipp-usb.

I see, I didn't know this. Yes, several years earlier I had set up a print queue (which I called "hp"),
and tried "lpr somefile.pdf" (hp is my default queue) as usual, and that didn't work. I also tried to modify
"hp"'s configuration (vi system-config-printer), not realizing that (perhaps? I didn't verify) all the time
I actually had other new queues names that did work.

I think we need to put some thought on how to treat upgrades in systems which already had print queues
set up. It's not just that I was used to doing "lpr -Php" and didn't think to check whether there are
other print queues available - the same existing queues were also used on other machines in the same
network, and users there were also surprised that they stopped working.


I'm not sure what to suggest should be done during upgrades :-( One option is to have ipp-usb
not run if other print queues exist. Another option can be for ipp-usb to somehow (?) figure out
that existing print queues use specific devices, and don't touch these devices. Another option
is to move way (e.g., move to *.rpmold) all the older print queues, and the user will notice
they disappeared - but then notice that new print queues appeared and start to use those.

Another thing that should perhaps be considered during upgrades is the "default" printer, the one
you get when you use "lpr somefile.pdf". If ipp-usb is the future, the default should probably
change to use its first printer - or something like that - not the pre-upgrade default printer
queue which probably no longer works.

Comment 7 Zdenek Dohnal 2022-03-15 10:52:58 UTC
Note for update:

The update removes the recommendation of ipp-usb - it is not able to remove ipp-usb itself, so users effected by the issue have to remove ipp-usb manually if you don't want to migrate to it yet.

The new update will make sure ipp-usb won't be brought in with next CUPS update, if user removed ipp-usb in the past.

Comment 8 Zdenek Dohnal 2022-03-15 12:01:10 UTC
(In reply to Nadav Har'El from comment #6)
> Thanks, however I assume this reversion will not help users who already did
> "dnf update" and got ipp-usb.
> Also, we'll eventually have exactly the same issue for people upgrading to
> Fedora 36.
> 

Yes, but there won't be no more broken setups in F35 and F34 at least :( . For F36 I'll send instructions to fedora-devel and fedora-users lists, hopefully it will help.

>       bInterfaceClass         7 Printer
>       bInterfaceSubClass      1 Printer
>       bInterfaceProtocol      4 

Ok, the device supports IPP-over-USB - https://docs.fedoraproject.org/en-US/quick-docs/cups-useful-tricks/#_how_to_find_out_if_my_usb_device_supports_ipp_over_usb .

> > So the printing should work correctly once you send the job to the correct
> > queue, which is generated by ipp-usb.
> 
> I see, I didn't know this. Yes, several years earlier I had set up a print
> queue (which I called "hp"),
> and tried "lpr somefile.pdf" (hp is my default queue) as usual, and that
> didn't work. I also tried to modify
> "hp"'s configuration (vi system-config-printer), not realizing that
> (perhaps? I didn't verify) all the time
> I actually had other new queues names that did work.

ipp-usb was added in Fedora 32, and until now nothing required it, so only users who proactively know about ipp-usb was using it, so I understand many people weren't aware of its existence.

Regarding the additional queues you have I cannot give any piece of advice - s-c-p shows only permanent queues, so it won't be the temp queue fro ipp-usb.

> 
> I think we need to put some thought on how to treat upgrades in systems
> which already had print queues
> set up. It's not just that I was used to doing "lpr -Php" and didn't think
> to check whether there are
> other print queues available - the same existing queues were also used on
> other machines in the same
> network, and users there were also surprised that they stopped working.

Aha, so the USB printer is plugged in a server, installed permanently there and then shared to other machines? Or do you use a USB hub, which the printer is plugged into, and other users are connected to the hub? Or users have set ServerName in /etc/cups/client.conf, so they connect to the CUPS server?

The first and the third way can get fix once the queue is reconfigured on the server. The second would need to reconfigure queues on the clients, unfortunately.

> 
> 
> I'm not sure what to suggest should be done during upgrades :-( One option
> is to have ipp-usb
> not run if other print queues exist. Another option can be for ipp-usb to
> somehow (?) figure out
> that existing print queues use specific devices, and don't touch these
> devices. Another option
> is to move way (e.g., move to *.rpmold) all the older print queues, and the
> user will notice
> they disappeared - but then notice that new print queues appeared and start
> to use those.
> 
> Another thing that should perhaps be considered during upgrades is the
> "default" printer, the one
> you get when you use "lpr somefile.pdf". If ipp-usb is the future, the
> default should probably
> change to use its first printer - or something like that - not the
> pre-upgrade default printer
> queue which probably no longer works.

We would have needed to have the printer turned on and plugged in during the upgrade, which IMO it is not something we can expect from users. Additionally, such stuff could be done during %post scriptlet, which does not work for Silverblue :( .
The main idea of the change is to possibly migrate users of IPP-over-USB printers to ipp-usb rather sooner than later to be able to catch the possible issues before there will be only ipp-usb and printer applications (since CUPS 3.0), so they won't be able to fallback to hplip itself in the future - so ignoring devices which have queues don't cut it. Moving away doesn't help either - if the device is not connected and turned on, you cannot find which queues are capable of IPP-over-USB, so you would have needed to move it blindly...

The default printer is the same case - you would have to find out whether the default printer is IPP-over-USB and then set it, which wouldn't work if the device is unplugged+turned off.

Unfortunately IMO creating automatic upgrade path for this is not possible right now :( .

Comment 9 Nadav Har'El 2022-03-16 21:01:02 UTC
(In reply to Zdenek Dohnal from comment #8)
> > Aha, so the USB printer is plugged in a server, installed permanently there
> and then shared to other machines? Or do you use a USB hub, which the
> printer is plugged into, and other users are connected to the hub? Or users
> have set ServerName in /etc/cups/client.conf, so they connect to the CUPS
> server?

The printer is plugged physically, over USB, to one machine in the house, and then all other machines in the house want to print to it.
If I remember correctly, all other machines connect to the USB-using one using CUPS, and get the list of queues from it. But still,
people on those machines (my family...) tried to print to "hp" because this is the queue which always worked.

> The first and the third way can get fix once the queue is reconfigured on
> the server. The second would need to reconfigure queues on the clients,
> unfortunately.

If the upgrade somehow eliminated my old "hp" queue and instead I got some new queue, I think we would manage
to try those new queues and see they work. I would lose some configuration (e.g., I also had a "hp2" queue that
printed to the same printer but forced double-side printing) but would have lived with that too. The main
problem was that after the upgrade, I still had the old "hp" queue just like I always had - it just didn't
work. Worse, the failure was completely silent... "lpq" and system-config-printers didn't tell me anything
was wrong with that queue - it seemed online and working - it just didn't print anything and just hung.
Only in journalctl could I find some signs that something is wrong in that queue.
 
> We would have needed to have the printer turned on and plugged in during the
> upgrade, which IMO it is not something we can expect from users.

One of the options I suggested was to delete the old queue I had (or rather,
move it out of the way, to some *.rpmold or something). This we could do
regardless of whether the printer is available during the upgrade.

> The default printer is the same case - you would have to find out whether
> the default printer is IPP-over-USB and then set it, which wouldn't work if
> the device is unplugged+turned off.

The concept of a "default printer" is useful especially for people like me who use the
command line a lot, and want to run "lpr somefile.pdf" and have it work. Could usb-ipp
change the default printer every time it comes up and sees just one printer? I don't think
this needs to happen during the upgrade itself.

Comment 10 Zdenek Dohnal 2022-03-17 15:18:18 UTC
(In reply to Nadav Har'El from comment #9)
> (In reply to Zdenek Dohnal from comment #8)
> The printer is plugged physically, over USB, to one machine in the house,
> and then all other machines in the house want to print to it.
> If I remember correctly, all other machines connect to the USB-using one
> using CUPS, and get the list of queues from it. But still,
> people on those machines (my family...) tried to print to "hp" because this
> is the queue which always worked.

If the clients had been creating its local queues which had pointed to the server, then once the server queue broke, they broke as well - but fixing the server queue would fix the issue at the clients as well.

> If the upgrade somehow eliminated my old "hp" queue and instead I got some
> new queue, I think we would manage
> to try those new queues and see they work. I would lose some configuration
> (e.g., I also had a "hp2" queue that
> printed to the same printer but forced double-side printing) but would have
> lived with that too.

If the update removes blindly all HP USB related queues, there will be reports about missing queues :( . For more specific removal you would need to have printer turned on to find out whether the printer is supported by ipp-usb. 

> The main
> problem was that after the upgrade, I still had the old "hp" queue just like
> I always had - it just didn't
> work. Worse, the failure was completely silent... "lpq" and
> system-config-printers didn't tell me anything
> was wrong with that queue - it seemed online and working - it just didn't
> print anything and just hung.
> Only in journalctl could I find some signs that something is wrong in that
> queue.
> 

It depends on the backend (hp here) whether it reports errors back to cupsd - otherwise CUPS cannot report anything :( .
 
> The concept of a "default printer" is useful especially for people like me
> who use the
> command line a lot, and want to run "lpr somefile.pdf" and have it work.
> Could usb-ipp
> change the default printer every time it comes up and sees just one printer?
> I don't think
> this needs to happen during the upgrade itself.

ipp-usb just advertises the queue on localhost - it is designed to create interface for USB, so it doesn't interact with CUPS on this level of communication.

Comment 11 Fedora Update System 2022-03-18 20:06:26 UTC
FEDORA-2022-016b7d39db has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 12 Fedora Update System 2022-03-29 02:30:23 UTC
FEDORA-2022-352f2e3280 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.