Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1344643 - default install of google-chrome, google-talkplugin or any other google app blocks users from getting *any* system updates
Summary: default install of google-chrome, google-talkplugin or any other google app b...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: gnome-software
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Richard Hughes
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
Depends On:
Blocks: F25FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2016-06-10 08:50 UTC by Kamil Páral
Modified: 2017-08-08 14:48 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-08 14:48:27 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)
PackageKit log while installing Chrome (2.55 MB, text/plain)
2016-06-10 08:52 UTC, Kamil Páral 		no flags Details
PackageKit log while downloading system updates (464.00 KB, text/plain)
2016-06-10 08:52 UTC, Kamil Páral 		no flags Details
journal from a failed offline update (87.04 KB, text/plain)
2016-06-10 08:54 UTC, Kamil Páral 		no flags Details
rpm -qa output (53.40 KB, text/plain)
2016-06-10 08:54 UTC, Kamil Páral 		no flags Details
chrome rpm scripts (26.02 KB, text/plain)
2016-06-10 09:07 UTC, Kamil Páral 		no flags Details
chrome's cron job (11.30 KB, text/plain)
2016-06-10 09:08 UTC, Kamil Páral 		no flags Details
software - "up to date" but not really (86.13 KB, image/png)
2016-08-30 14:14 UTC, Kamil Páral 		no flags Details
software - error screen during update refresh (128.42 KB, image/png)
2016-08-30 14:15 UTC, Kamil Páral 		no flags Details
View All

Description Kamil Páral 2016-06-10 08:50:29 UTC 
Description of problem:
Short version:
All users who installed Google Chrome and only use gnome-software (not dnf) are unable to install any system updates (including security fixes). That is due to a missing confirmation dialog in gnome-software regarding signed repos with not yet imported keys.

Long version:
Google Chrome is distributed as a RPM, which contains a post-install script which adds a google-chrome.repo and tries to import its key. That import fails, which is partly a bug in chrome's post-install script (they work around it by using cron on other systems, but they don't do it properly on Fedora) and partly rpm's fault for not providing a clean way to handle this and forcing third parties to resort to ugly hack. Some details here:
https://bugs.chromium.org/p/chromium/issues/detail?id=514013

So, after google-chrome RPM install, the system contains the repo (enabled), but does not trust it. This is not a problem if users use dnf, because on the next google-chrome update, dnf asks to user whether to trust the repo and the updated rpm, and imports the key once user confirms it.

However, this is a serious issue with gnome-software, because it does *not* ask the user whether to trust the repo before proceeding with the update. Instead, it downloads all updates (latest google-chrome and any other system updates available), does not ask for key confirmation, and asks to reboot the system to install updates. Once the system is rebooted and offline update is attempted, PackageKit notices that one of the RPMs (google-chrome) is untrusted and aborts the whole operation. On the next system boot, the user is not informed about failed update in any way, and none of the available updates were installed. Once in a while, the user is informed about new updates, but the system silently fails to install them, and such system is basically cut off from any system updates.

If you know where to look, you can find the error message:
$ pkcon offline-status 
Status:	Failed
ErrorCode:failed-initialization
ErrorDetails:package google-chrome-stable-51.0.2704.84-1.x86_64 cannot be verified and repo google-chrome is GPG enabled: failed to lookup digest in keyring for /var/cache/PackageKit/24/metadata/google-chrome/packages/google-chrome-stable-51.0.2704.84-1.x86_64.rpm

The workaround is to run "dnf update" in terminal once, and confirm importing google-chrome's key. Alternatively, you can do (in this specific case):
$ wget https://dl.google.com/linux/linux_signing_key.pub
$ sudo rpmkeys --import linux_signing_key.pub

It's difficult to say who's to blame here. Google Chrome doesn't try hard enough to import the key in their post-install script. RPM doesn't seem to offer any clean solution for packages which want to import a gpg key as part of the RPM installation (a common case for third parties). And packagekit and gnome-software should not silently download updates, don't ask anything, and then reject it without even telling the user what the problem is and offering a solution.

Overall, though, I believe we should not let one of the most commonly widespread third party repos (which is even trying to hack around the problem and do the right thing) completely break system updates (including critical security ones).

In any case, it seems that gnome-software should do gpg checking during the interactive mode, and allow people to react to untrusted packages. That seems like an omission after the offline update feature introduction, and also it would probably be the easiest way to fix this problem. After updates are downloaded, if there are any untrusted packages, it would show a system popup e.g. "additional action before update required" and after clicking on it, people would see the list of available updates and a dialog stating that some packages are untrusted and that you need either to trust this key (yes/no) or remove the package in order to continue with system update. Only after the key is trusted or the packages removed, gnome-software would make the system aware of available updates (so that "install updates" checkbox would appear in the poweroff dialog).

Please note that this issue most probably applies also to other signed third party repositories. I only tested google chrome, because it's hugely popular, but there's no reason this would be specific to chrome - the same problem should occur with other repos, as long as you only use gnome-software to interact with it.


Version-Release number of selected component (if applicable):
google-chrome-stable-50.0.2661.75-1.x86_64
gnome-software-3.20.3-1.fc24.x86_64
PackageKit-1.1.1-2.fc24.x86_64
libhif-0.2.2-4.fc24.x86_64
librepo-1.7.18-2.fc24.x86_64
dnf-1.1.9-2.fc24.noarch
rpm-4.13.0-0.rc1.27.fc24.x86_64

How reproducible:
always

Steps to Reproduce:
1. install clean F24 Workstation
2. download an older version of google-chrome. Because I found it very hard to find an older RPM for chrome on the internet, I uploaded chrome version 50 here:
https://kparal.fedorapeople.org/bugs/chrome-updates/google-chrome-stable_50_x86_64.rpm
3. double click on the RPM, it should open up in gnome-software. Click Install, wait until finished. (The progress bar is completely broken, you can watch progress with `pkmon`).
4. google-chrome rpm should fail to add any gpg keys during installation or during its scheduled cron job 2 minutes after installation. Use `sudo atq` to see the pending job and wait until it is executed. Then confirm that chrome's keys were not imported. Use
$ rpm -qa 'gpg-pubkey*'
gpg-pubkey-81b46521-55b3ca9a
which lists only one key (Fedora key).
5. make sure there are some other system updates available (not just chrome). You can use `dnf check-update` or gnome-software's Updates tab (Refresh button, again `pkmon` to see progress) to see a list of available updates. If you don't see anything except chrome, enable updates-testing:
$ sudo dnf config-manager --set-enabled updates-testing
Now you should see updates available. Use gnome-software's updates tab to trigger downloading them. Wait until they are downloaded and show up in gnome-software. Notice, that you'll not see any warning about untrusted packages.
6. Use the Restart&Install button and confirm reboot into offline updates mode. 
7. In offline updates mode, notice that the progress indicator jumps to ~14% and then immediately reboots.
8. After next boot, notice that you won't receive any failure notification (a separate bug, I assume).
9. Open up gnome-software and see that all the updates are still available and not applied. Not just google-chrome, but also all system ones.
10. See the failure message using `pkcon offline-status` or inspect `journalctl -b -1`

Actual results:
default installation of google-chrome prevents future system updates

Expected results:
default installation of google-chrome doesn't prevent future system updates
Comment 1 Kamil Páral 2016-06-10 08:52:14 UTC 
Created attachment 1166588 [details]
PackageKit log while installing Chrome
Comment 2 Kamil Páral 2016-06-10 08:52:43 UTC 
Created attachment 1166589 [details]
PackageKit log while downloading system updates
Comment 3 Kamil Páral 2016-06-10 08:54:16 UTC 
Created attachment 1166590 [details]
journal from a failed offline update

Full journal from offline update attached. However pkcon detects the error correctly:

$ pkcon offline-status 
Status:	Failed
ErrorCode:failed-initialization
ErrorDetails:package google-chrome-stable-51.0.2704.84-1.x86_64 cannot be verified and repo google-chrome is GPG enabled: failed to lookup digest in keyring for /var/cache/PackageKit/24/metadata/google-chrome/packages/google-chrome-stable-51.0.2704.84-1.x86_64.rpm
Comment 4 Kamil Páral 2016-06-10 08:54:57 UTC 
Created attachment 1166591 [details]
rpm -qa output
Comment 5 Kamil Páral 2016-06-10 09:07:02 UTC 
Created attachment 1166594 [details]
chrome rpm scripts

There are the scripts chrome executed during rpm operations. The important one is line 101 install_rpm_key(). At line 151 there is
>     rpm --import "$TMPKEY"
which you can see to fail if you install the rpm using dnf:

>   Installing  : google-chrome-stable-50.0.2661.75-1.x86_64              135/136 
> error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)
> error: /tmp/google.sig.5TVPxG: key 1 import failed.
> Redirecting to /bin/systemctl start  atd.service

They are actually aware that sometimes you can't import a key during post-install, which is why they try to postpone it at line 485:
> echo "sh /etc/cron.daily/google-chrome" | at now + 2 minute > /dev/null 2>&1

However, there is a logical error in that script which causes it to no longer try to import the key (it just verifies the repo was added), at least on Fedora.


I'm posting this here to explain in detail what's going on, but I don't think this should be used to claim "it's a bug in google-chrome, not our problem". First, they're really trying to make this work, and second I don't even think they should be doing such dirty hacks in post-install. Our package managers should be able to deal with unimported keys in a friendly fashion and should not force third-parties to invent ugly hacks in their RPMs.
Comment 6 Kamil Páral 2016-06-10 09:08:26 UTC 
Created attachment 1166597 [details]
chrome's cron job

This script is executed 2 minutes after RPM install, and regularly once a day. However, it will not fix the missing key import as it's currently written.
Comment 7 Kamil Páral 2016-06-10 09:41:19 UTC 
After talking to Richard, if chrome installed its key to /etc/pki/rpm-gpg/ this would not happen, because libhif autoimports everything from that directory. It does not automatically download the repo keys, though.
Comment 8 Elliott Sales de Andrade 2016-07-01 03:02:35 UTC 
Isn't this true of all packages in Google's repository, not just Chrome? I'm pretty sure I saw issues on F23 from the Google Talk plugin, which were fixed after installing the GPG key.
Comment 9 Kamil Páral 2016-07-01 07:01:37 UTC 
(In reply to Elliott Sales de Andrade from comment #8)
> Isn't this true of all packages in Google's repository, not just Chrome?

Most likely, yes. Thanks for mentioning that.
Comment 10 Kamil Páral 2016-08-30 14:01:46 UTC 
The behavior seems to have changed slightly with gnome-software-3.21.90-2.fc25. It no longer fails during offline update, it fails now during transaction verification in gnome-software (which is an improvement).

If you let packagekit do its thing and run in the background, it will download all updates, fail the transaction test, and keep silent. Therefore no "updates are available" notification is posted, probably never (as long as you have google-chrome installed). If you open up gnome-software, it claims "software is up to date" (which is, of course, its usual lie). See screenshot.

If you trigger refresh with the top left button, it will re-run the transaction test and show an error message:
"package google-chrome-* cannot be verified and repo google-chrome is GPG enabled: failed to lookup digest in keyring in /var/cache/PackageKit/*"
See screenshot.

If certain people reach this screen and realize they could remove google-chrome (but would they want to do that?), it will "fix" the issue. However, most people won't even see this screen (because even if they wander into the Updates tab, it will say "software is up to date" every time).
Comment 11 Kamil Páral 2016-08-30 14:14:32 UTC 
Created attachment 1195928 [details]
software - "up to date" but not really

This is the screen that you see when you open gnome-software and your system update is already broken due to google-chrome.
Comment 12 Kamil Páral 2016-08-30 14:15:20 UTC 
Created attachment 1195929 [details]
software - error screen during update refresh

This is the error screen that you see when you manually refresh available updates.
Comment 13 Kamil Páral 2016-08-30 14:16:29 UTC 
I'm proposing this as a Final blocker, because it violates these criteria:
"The installed system must be able to download and install updates with the default graphical package manager in all release-blocking desktops. "
https://fedoraproject.org/wiki/Fedora_25_Beta_Release_Criteria#Updates
and
"Release-blocking desktops must notify the user of available updates, but must not do so when running as a live image. "
https://fedoraproject.org/wiki/Fedora_25_Final_Release_Criteria#Update_notification

This is a conditional violation in case the user installed google-chrome. Neither the update notification will arrive, nor the update can be performed manually inside gnome-software.

Some ideas how to deal with this are provided in comment 0. 

Please not that due to bug 1367780 you need to install chrome RPM using "pkcon install-local" instead of gnome-software. Also, because there are oh so many bugs in packagekit, in case packagekit already downloaded some updates for you before you installed chrome, you need to refresh the updates list in order to see this bug.
Comment 14 Kamil Páral 2016-08-31 08:07:05 UTC 
I have tested other Google apps [1] as well. This problem most probably affects all of them (because they use the same post-install scripts). I have verified this with google-talkplugin and google-webdesigner.

With google-talkplugin, simply install an older version [2] and follow the usual procedure (try to update). Same error.

With google-webdesigner, I don't have an older RPM for it, so I used this trick:
1. Installed current google-webdesigner (1.3.5.0)
2. Create a fake google-webdesigner rpm [3] with rpmfluff with version 1.0, downgraded to it
3. Tried to update to the version in repos (1.3.5.0)

Adjusting title to make it clear this affects all google apps distributed as RPMs.

[1] https://www.google.com/linuxrepositories/
[2] https://kparal.fedorapeople.org/bugs/chrome-updates/google-talkplugin_5.41.0.0_x86_64.rpm
[3] https://kparal.fedorapeople.org/bugs/chrome-updates/google-webdesigner-1.0-1.x86_64.rpm
Comment 15 Kalev Lember 2016-09-07 15:35:56 UTC 
https://github.com/rpm-software-management/libhif/pull/187 should fix this
Comment 16 Kamil Páral 2016-09-09 08:27:57 UTC 
(In reply to Kalev Lember from comment #15)
> https://github.com/rpm-software-management/libhif/pull/187 should fix this

And it does! Thanks.
Comment 17 Kamil Páral 2016-09-09 08:28:47 UTC 
(The PK build people can test with is this: http://koji.fedoraproject.org/koji/buildinfo?buildID=798635 )
Comment 18 Fedora Update System 2016-09-09 08:50:23 UTC 
PackageKit-1.1.4-0.4.20160901.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-360d537639
Comment 19 Fedora Update System 2016-09-10 07:57:43 UTC 
PackageKit-1.1.4-0.4.20160901.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-360d537639
Comment 20 Fedora Update System 2016-09-12 13:17:42 UTC 
PackageKit-1.1.4-0.4.20160901.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Comment 21 Adam Williamson 2016-11-29 01:52:22 UTC 
This is still not fixed on F24, is it? I guess not, as there has not been an F24 update to PK since 1.1.3-2. Re-opening, since this was filed against F24 and is on the F24 Common Bugs list. Please fix for F24 if possible. Thanks!
Comment 22 Fedora End Of Life 2017-07-25 21:03:31 UTC 
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Comment 23 Fedora End Of Life 2017-08-08 14:48:27 UTC 
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Note You need to log in before you can comment on or make changes to this bug.