Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1710576

Summary: /etc/cron.hourly/0yum-hourly.cron crashes - unhashable type: 'dict'
Product: [Fedora] Fedora Reporter: Tom Chiverton <bugzilla.redhat.com>
Component: python-urlgrabberAssignee: Michal Domonkos <mdomonko>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: dmach, mdomonko, ngompa13, packaging-team-maint, sheltren
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-urlgrabber-4.0.0-3.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-06 01:05:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tom Chiverton 2019-05-15 19:42:17 UTC
Description of problem:
yum-hourly cron task fails

Version-Release number of selected component (if applicable):

# rpm -qf /etc/yum/yum-cron-hourly.conf
yum-cron-3.4.3-521.fc30.noarch
# rpm -qa urlgrabber
python2-urlgrabber-4.0.0-1.fc30.noarch

How reproducible:
every time 

Steps to Reproduce:
1. run /etc/cron.hourly/0yum-hourly.cron

Actual results:
crash

Expected results:
no crash

Additional info:

# /etc/cron.hourly/0yum-hourly.cron 
Traceback (most recent call last):
  File "/usr/sbin/yum-cron", line 713, in <module>
    main()
  File "/usr/sbin/yum-cron", line 710, in main
    base.updatesCheck()
  File "/usr/sbin/yum-cron", line 602, in updatesCheck
    self.populateUpdateMetadata()
  File "/usr/sbin/yum-cron", line 409, in populateUpdateMetadata
    self.pkgSack # honor skip_if_unavailable
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1095, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 799, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1460, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 698, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 693, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 659, in _setupGrab
    failure_callback=mirror_failure)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 288, in __init__
    self.mirrors.sort(key=estimate, reverse=True)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 282, in estimate
    speed, fail = _TH.estimate(m['mirror'])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2547, in estimate
    host = urlparse.urlsplit(baseurl).netloc.split(b'@')[-1] or baseurl
  File "/usr/lib64/python2.7/urlparse.py", line 176, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'dict'

Comment 1 Tom Chiverton 2019-05-16 17:10:30 UTC
Found an upstream bug (linked), closed sometime ago saying they don't care.
so fedora should drop urlgrabber? Means rewriting core yum features..

Comment 2 Michal Domonkos 2019-05-17 08:15:30 UTC
Fedora actually *is* dropping yum (the original implementation, version 3) starting with F31.

That said, in F30, we still have yum-3, so this should really be fixed.  I'll take that on.

Comment 3 Michal Domonkos 2019-05-20 13:55:18 UTC
Fixed upstream: https://github.com/rpm-software-management/urlgrabber/pull/13

Comment 4 Fedora Update System 2019-05-21 16:30:18 UTC
python-urlgrabber-4.0.0-3.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-372212061d

Comment 5 Fedora Update System 2019-05-22 01:44:20 UTC
python-urlgrabber-4.0.0-3.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-372212061d

Comment 6 Tom Chiverton 2019-05-23 20:52:48 UTC
Sorted.

Comment 7 Tom Chiverton 2019-05-27 14:40:35 UTC
However, dnf upgrade now reports issues :

[root@bookcase boinc]# dnf upgrade
ROCm                                                                                            16 kB/s | 2.9 kB     00:00    
Fedora Modular 30 - x86_64                                                                      14 kB/s |  24 kB     00:01    
Fedora Modular 30 - x86_64 - Updates                                                            45 kB/s |  23 kB     00:00    
Fedora 30 - x86_64 - Updates                                                                    47 kB/s |  24 kB     00:00    
Fedora 30 - x86_64                                                                              29 kB/s |  24 kB     00:00    
RPM Fusion for Fedora 30 - Free - Updates                                                       57 kB/s |  10 kB     00:00    
RPM Fusion for Fedora 30 - Free                                                                 24 kB/s |  10 kB     00:00    
Dependencies resolved.

 Problem: package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
===============================================================================================================================
 Package                         Architecture               Version                          Repository                   Size
===============================================================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 fuse-common                     x86_64                     3.4.2-3.fc30                     fedora                       19 k
 fuse-common                     x86_64                     3.5.0-1.fc30                     updates                     7.4 k

Transaction Summary
===============================================================================================================================
Skip  2 Packages

Nothing to do.
Complete!
[root@bookcase boinc]# dnf upgrade --best --allowerasing
Last metadata expiration check: 0:00:30 ago on Mon 27 May 2019 15:36:09 BST.
Error: 
 Problem: cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
  - problem with installed package fuse-2.9.9-3.fc30.x86_64
  - package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - problem with installed package fuse-common-3.4.2-6.fc30.x86_64
(try to add '--skip-broken' to skip uninstallable packages)
[root@bookcase boinc]# dnf upgrade --best --allowerasing --skip-broken
Last metadata expiration check: 0:01:13 ago on Mon 27 May 2019 15:36:09 BST.
Error: 
 Problem: cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
  - problem with installed package fuse-2.9.9-3.fc30.x86_64
  - package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - problem with installed package fuse-common-3.4.2-6.fc30.x86_64
[root@bookcase boinc]# dnf remove fuse-common
Dependencies resolved.
===============================================================================================================================
 Package                          Architecture             Version                            Repository                  Size
===============================================================================================================================
Removing:
 fuse-common                      x86_64                   3.4.2-6.fc30                       @updates                    38  
Removing dependent packages:
 borgbackup                       x86_64                   1.1.10-2.fc30                      @updates                   4.8 M
Removing unused dependencies:
 fuse                             x86_64                   2.9.9-3.fc30                       @fedora                    224 k
 python3-llfuse                   x86_64                   1.3.5-2.fc30                       @fedora                    1.5 M

Transaction Summary
===============================================================================================================================
Remove  4 Packages

Freed space: 6.5 M
Is this ok [y/N]: n
Operation aborted.
[root@bookcase boinc]#

Comment 8 Tom Chiverton 2019-05-27 14:43:46 UTC
Looks like you need the fuse package from updates-testing as well:

[root@bookcase boinc]# sudo dnf upgrade --enablerepo=updates-testing fuse
Last metadata expiration check: 0:00:50 ago on Mon 27 May 2019 15:40:57 BST.
Dependencies resolved.
===============================================================================================================================
 Package                 Architecture              Version                            Repository                          Size
===============================================================================================================================
Upgrading:
 fuse                    x86_64                    2.9.9-7.fc30                       updates-testing                     79 k

Transaction Summary
===============================================================================================================================
Upgrade  1 Package

Total download size: 79 k
Is this ok [y/N]: y
Downloading Packages:
fuse-2.9.9-7.fc30.x86_64.rpm                                                                   280 kB/s |  79 kB     00:00    
-------------------------------------------------------------------------------------------------------------------------------
Total                                                                                           86 kB/s |  79 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                       1/1 
  Upgrading        : fuse-2.9.9-7.fc30.x86_64                                                                              1/2 
update
  Cleanup          : fuse-2.9.9-3.fc30.x86_64                                                                              2/2 
  Running scriptlet: fuse-2.9.9-3.fc30.x86_64                                                                              2/2 
  Verifying        : fuse-2.9.9-7.fc30.x86_64                                                                              1/2 
  Verifying        : fuse-2.9.9-3.fc30.x86_64                                                                              2/2 

Upgraded:
  fuse-2.9.9-7.fc30.x86_64                                                                                                     

Complete!
[root@bookcase boinc]# dnf update
Last metadata expiration check: 0:06:13 ago on Mon 27 May 2019 15:36:09 BST.
Dependencies resolved.
===============================================================================================================================
 Package                         Architecture               Version                          Repository                   Size
===============================================================================================================================
Upgrading:
 fuse-common                     x86_64                     3.5.0-1.fc30                     updates                     7.4 k

Transaction Summary
===============================================================================================================================
Upgrade  1 Package

Total download size: 7.4 k
Is this ok [y/N]: y
Downloading Packages:
fuse-common-3.4.2-6.fc30_3.5.0-1.fc30.x86_64.drpm                                               26 kB/s | 4.6 kB     00:00    
[DRPM] fuse-common-3.4.2-6.fc30_3.5.0-1.fc30.x86_64.drpm: done                                                                
-------------------------------------------------------------------------------------------------------------------------------
Total                                                                                          8.7 kB/s | 4.6 kB     00:00     
Delta RPMs reduced 0.0 MB of updates to 0.0 MB (38.1% saved)
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                       1/1 
  Upgrading        : fuse-common-3.5.0-1.fc30.x86_64                                                                       1/2 
  Cleanup          : fuse-common-3.4.2-6.fc30.x86_64                                                                       2/2 
  Verifying        : fuse-common-3.5.0-1.fc30.x86_64                                                                       1/2 
  Verifying        : fuse-common-3.4.2-6.fc30.x86_64                                                                       2/2 

Upgraded:
  fuse-common-3.5.0-1.fc30.x86_64                                                                                              

Complete!
[root@bookcase boinc]# dnf update
Last metadata expiration check: 0:06:43 ago on Mon 27 May 2019 15:36:09 BST.
Dependencies resolved.
Nothing to do.
Complete!
[root@bookcase boinc]#

Comment 9 Michal Domonkos 2019-05-27 15:06:57 UTC
Please note that DNF does not use python-urlgrabber at all.

Comment 10 Ben Cotton 2019-05-28 18:55:09 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 11 Fedora Update System 2019-06-06 01:05:58 UTC
python-urlgrabber-4.0.0-3.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.