Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 963810
Summary: | Could not get update details: Error Type: <type 'exceptions.UnicodeDecodeError'> | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Michael Schwendt <bugs.michael> | ||||||||||
Component: | PackageKit | Assignee: | Richard Hughes <rhughes> | ||||||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||
Severity: | urgent | Docs Contact: | |||||||||||
Priority: | unspecified | ||||||||||||
Version: | 19 | CC: | awilliam, jonathan, kevin, kparal, rdieter, rhughes, robatino, rtguille, smparrish | ||||||||||
Target Milestone: | --- | ||||||||||||
Target Release: | --- | ||||||||||||
Hardware: | Unspecified | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | AcceptedBlocker | ||||||||||||
Fixed In Version: | PackageKit-0.8.9-1.fc19 | Doc Type: | Bug Fix | ||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2013-05-22 01:59:14 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 834087, 964352 | ||||||||||||
Attachments: |
|
Hmm, I can't reproduce. What's the output of "echo $LANG" on your machine (logged in as your user)? It trips over the %description of the policycoreutils package, which uses a 0xc2 0xae Registered Trade Mark symbol character. In packagekit/backend.py, line 239, in details that is the "desc" variable. >>> s = 'Linux® kernel' >>> print isinstance(s,str) True >>> print isinstance(s,unicode) False >>> print s.encode('utf-8',errors='replace') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5: ordinal not in range(128) $ echo $LANG en_US.UTF-8 $ locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= Brilliant, thanks for narrowing it down. I don't think s.encode is what's tripping up PK tho, I think is a more reliable testcase: --- #!/usr/bin/python2 # -*- coding: utf-8 -*- def _to_utf8(txt, errors='replace'): if isinstance(txt, str): return txt if isinstance(txt, unicode): return txt.encode('utf-8', errors=errors) return str(txt) s = 'Linux® kernel' print isinstance(s,str) print isinstance(s,unicode) print _to_utf8(s) --- This prints for me: True False Linux® kernel Just to check the obvious, what version of PackageKit is this? Thanks. Created attachment 749354 [details]
gpk-u-v output on F18
I see the same traceback on F18:
PackageKit-0.8.8-1.fc18.x86_64
gnome-packagekit-3.6.2-1.fc18.x86_64
I have cs_CZ.UTF-8 locale, but it also fails with en_US. No policycoreutils update is available. I don't see any valuable info in pkmon or on stdout.
When I try to update anyway, it fails: 16:21:37 GnomePackageKit status download 16:23:02 GnomePackageKit status sig-check 16:23:02 GnomePackageKit status test-commit 16:23:04 GnomePackageKit status update 16:23:04 GnomePackageKit already active 16:23:04 GnomePackageKit already active 16:23:05 GnomePackageKit already active 16:23:05 GnomePackageKit already active 16:23:05 GnomePackageKit already active 16:23:06 GnomePackageKit status finished 16:23:06 GnomePackageKit status check-executable-files 16:23:06 GnomePackageKit status finished 16:23:06 GnomePackageKit Unknown error, please report a bug at https://bugs.freedesktop.org/. More information is available in the detailed report. 16:23:06 GnomePackageKit Unknown error 16:23:06 GnomePackageKit (null): There are unfinished transactions remaining. Please run yum-complete-transaction as root. $ rpm -q PackageKit
PackageKit-0.8.8-2.fc19.x86_64
> This prints for me:
>
> True
> False
> Linux® kernel
The same here, of course, because _to_utf8() returns the unchanged string in the first if-statement.
Hey Kamil: Can you directly edit: /usr/lib/python2.7/site-packages/packagekit/backend.py and add before line 239: print package_id print package_license print group print desc print url print bytes and try to work out: a) which package it is (so I have a hope of reproducing it locally) b) which parameter is encoded in a weird way (uncommenting them one by one) Thanks dude, Richard Oh, last comment is confusing, of course. What I mean is that in packagekit backend.py, the isinstance() results for "desc" are the opposite. This is what I get during debugging: str = 0 unicode = 1 Security-enhanced Linux is a feature of the Linux® kernel [...] In turn, encode() is called and throws an exception. Restructuring the code shows that print() throws the exception for the utf-8 encoded strings: uni_desc = _to_utf8(desc) uni_group = _to_utf8(group) print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout) KK tells me https://bugzilla.redhat.com/show_bug.cgi?id=964352 is the same as this. If so, this is preventing installation of updates on F19 systems currently, and leaving the yum database in an inconsistent and apparently unrecoverable state. Serious issue, Beta blocker. > leaving the yum database in an inconsistent and apparently
> unrecoverable state. Serious issue
I performed a 'yum clean all' then a 'yum-complete-transaction --cleanup-only' then a 'yum-complete-transaction' with redirection to a file, i leaved it some hours and produced a +200mb output file (!). I interrupted it with ctrl+c. As Adam said, it really looks bad. (KDE Plasma Workspaces + apper).
*** Bug 964352 has been marked as a duplicate of this bug. *** Created attachment 750427 [details]
output of pkcon
Ok, another try.
I reinstalled the guest, KDE Plasma Workspaces Base Environment without Add-ONs.
I will try packagekit command line 'pkcon' utility instead of KDE 'apper'.
Ssh from another host:
# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
# localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: es
X11 Layout: es
# pkcon --version
0.8.8
The attached file contains the output of:
# pkcon -p -v get-updates
# pkcon -p -v update
Something to highlight from the pkcon's output is this:
20:04:19 PackageKit cannot translate 'cleanup', please report!
cleanup
20:04:19 PackageKit cannot translate 'obsoleting', please report!
obsoleting
I don't know if this is related to the issue, there are a lot of this messages.
The command 'pkcon -p -v update' ends in:
Status: Waiting in queue
20:06:15 PackageKit emit transaction-list-changed
Status: Starting
Percentage: 0
Status: Running
Status: Resolving dependencies
Percentage: 10
Status: Downloading packages
Percentage: 40
Status: Checking signatures
Software source signature required
Package: vim-minimal-2:7.3.944-1.fc19.x86_64
Software source name: updates-testing
Key URL: /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-x86_64
Key user: Fedora (19) <fedora>
Key ID: FB4B18E6
Key fingerprint: ca81b2c85e4f4d4a1a3f723407477e65fb4b18e6
Key Timestamp: Sat Dec 1 00:31:25 2012
Do you accept this signature? [N/y] y
20:39:23 PackageKit emit transaction-list-changed
20:39:23 PackageKit notify::connected
The daemon crashed mid-transaction!
From this point, a 'yum-complete-transaction --cleanup-only' is required, if executed, all packages look like they were installed while there were not. :-(
A regular 'yum-complete-transaction' might work but i aborted it when its output
exceeded 200mb after several hours (i was using redirection to a file), so maybe
leaving it all night might work.
Can anyone who can reproduce this bug successfully run: pkcon get-details libselinux It should display: description: Security-enhanced Linux is a feature of the Linux® kernel.. I'm struggling to work out why printing a UTF-8 string should suddenly start throwing exceptions. Richard I'm going to need the help of an experienced python developer here. Something funky is going on. It's nothing to do with the daemon, and everything to do with the yum backend. Since Nils got reassigned, we don't actually have an official yum backend maintainer anymore. Ideas welcome. Richard (In reply to Richard Hughes from comment #7) Richard, it doesn't happen with PackageKit-0.8.7-1.fc18.x86_64, but it happens with PackageKit-0.8.8-1.fc18.x86_64. The problematic package is "opencv;2.4.3-5.fc18;x86_64;updates-testing" and its "desc" attribute. > Error Value: 'ascii' codec can't encode character u'\xae' in position 18: ordinal not in range(128) This is the value, retrieved by repr(): u'OpenCV means Intel\xae Open Source Computer Vision Library. It is a collection of;C functions and a few C++ classes that implement some popular Image Processing;and Computer Vision algorithms.' Just to answer comment 14, it reproduces the issue in the same way (except that for me it displays my modifications to packagekit/backend.py in the traceback). Also, one cannot simply print() in there for debugging purposes, since there's some redirection taking place. type(desc) = <type 'unicode'> sys.stdout.encoding = 'None' [...] $ pkcon get-details libselinux More than one package matches: 1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing] 2. libselinux-2.1.13-15.fc19.i686 [fedora] 3. libselinux-2.1.13-15.fc19.i686 [updates-testing] Please choose the correct package: 3 Getting details [=========================] Waiting in queue [=========================] Starting [=========================] Getting information [=========================] Fatal error: Error Type: <type 'exceptions.UnicodeDecodeError'> Error Value: 'ascii' codec can't decode byte 0xc2 in position 49: ordinal not in range(128) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3590, in <module> main() File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3587, in main backend.dispatcher(sys.argv[1:]) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 760, in dispatcher self.dispatch_command(args[0], args[1:]) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 620, in dispatch_command self.get_details(package_ids) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2546, in get_details self._show_details_pkg(pkg) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2572, in _show_details_pkg self.details(package_id, license, group, desc, url, size) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 303, in details PackageKitBaseBackend.details(self, package_id, package_license, group, desc, url, bytes) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 268, in details print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout) (In reply to Richard Hughes from comment #14) > Can anyone who can reproduce this bug successfully run: > > pkcon get-details libselinux This is throwing the usual exception with PackageKit-0.8.8-1.fc18.x86_64 and PackageKit-0.8.8-2.fc19.x86_64. With PackageKit-0.8.7-1.fc18.x86_64 it doesn't crash, but doesn't print anything: $ pkcon get-details libselinux More than one package matches: 1. libselinux-2.1.12-7.3.fc18.x86_64 [installed:updates] 2. libselinux-2.1.12-7.3.fc18.i686 [updates] Please choose the correct package: 2 Getting details [=========================] Waiting in queue [=========================] Starting [=========================] Getting information [=========================] $ Re: comment 16 > u'OpenCV means Intel\xae Open Source 0xae is ISO 8859-1 for the Trade Mark symbol. The f18 spec file contains UTF-8 0xc2 0xae. PackageKit 0.8.8 changelog mentions "Do not use _UTF8Writer when using python3 (Richard Hughes)". Actually using an UTF8Writer based on eithers codecs.getwriter() or kitchen.text.converters is something I've tried while taking a first look at the problem. Hasn't worked, and that's where I discovered the 'sys.stdout.encoding = None', too. Created attachment 750552 [details]
test patch
Okay, so I've got something I think works (basically, rip out all the print() encoding stuff).
Can someone try doing:
/usr/lib/python2.7/site-packages/packagekit
sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch
If that fixes things, I'll commit upstream and roll a new update.
Thanks,
Richard.
Works for me. I've also run gpk-update-viewer and applied the pending updates. $ pkcon get-details libselinux More than one package matches: 1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing] 2. libselinux-2.1.13-15.fc19.i686 [fedora] 3. libselinux-2.1.13-15.fc19.i686 [updates-testing] Please choose the correct package: 3 Getting details [=========================] Waiting in queue [=========================] Starting [=========================] Getting information [=========================] Package description package: libselinux-2.1.13-15.fc19.i686 license: Public Domain group: other description: Security-enhanced Linux is a feature of the Linux® kernel and a number of utilities with enhanced security functionality designed to add mandatory access controls to Linux. The Security-enhanced Linux kernel contains new architectural components originally developed to improve the security of the Flask operating system. These architectural components provide general support for the enforcement of many kinds of mandatory access control policies, including those based on the concepts of Type Enforcement®, Role-based Access Control, and Multi-level Security. libselinux provides an API for SELinux applications to get and set process and file security contexts and to obtain security policy decisions. Required for any applications that use the SELinux API. size: 141244 bytes url: http://oss.tresys.com/git/selinux.git (In reply to Richard Hughes from comment #21) > /usr/lib/python2.7/site-packages/packagekit > sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch Fixes the unicode problem in F18. I still can't perform updates because of some other bug, but that was present before. I'll file that separately. The encoding issue seems to be gone. "pkcon get-details libselinux" also works. PackageKit-0.8.9-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/PackageKit-0.8.9-1.fc19 Discussed at 2013-05-20 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-20/f19beta-blocker-review-7.2013-05-20-16.07.log.txt . Accepted as a blocker per criterion https://fedoraproject.org/wiki/Fedora_19_Alpha_Release_Criteria#Updates : "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops." While this backend problem causes the the apper crash ( bug #964352 ), dannti thinks we can fix apper/kded separately (undup'ing) Package PackageKit-0.8.9-1.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing PackageKit-0.8.9-1.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-8637/PackageKit-0.8.9-1.fc19 then log in and leave karma (feedback). Richard, I need an F18 build as well. Thanks. On F19, "pkcon get-details libselinux" works well with the new version. I can't test real policycoreutils update, because there is no version to downgrade to. PackageKit-0.8.9-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report. |
Created attachment 748898 [details] console output of gpk-update-viewer -v $ rpm -qf $(which gpk-update-viewer) gnome-packagekit-3.8.1-1.fc19.x86_64 Reproducibility: currently always An error dialog is displayed, but there is no hint about which package (perhaps a %changelog entry) it refers to. Verbose output attached. Could not get update details Failed to process request Error Type: <type 'exceptions.UnicodeDecodeError'> Error Value: 'ascii' codec can't decode byte 0xc2 in position 49: ordinal not in range(128) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3590, in <module> main() File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3587, in main backend.dispatcher(sys.argv[1:]) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 722, in dispatcher self.dispatch_command(args[0], args[1:]) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 582, in dispatch_command self.get_details(package_ids) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2546, in get_details self._show_details_pkg(pkg) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2572, in _show_details_pkg self.details(package_id, license, group, desc, url, size) File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 303, in details PackageKitBaseBackend.details(self, package_id, package_license, group, desc, url, bytes) File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 239, in details print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, _to_utf8(group), _to_utf8(desc), url, bytes), file=sys.stdout)