Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1789200 - python-unittest2 fails to build with Python 3.9
Summary: python-unittest2 fails to build with Python 3.9
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-unittest2
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Alfredo Moralejo
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-09 02:48 UTC by Charalampos Stratakis
Modified: 2020-01-17 11:26 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-17 11:26:20 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Charalampos Stratakis 2020-01-09 02:48:07 UTC
python-unittest2 fails to build with Python 3.9.0a2.

Error message:

running install
Traceback (most recent call last):
  File "/builddir/build/BUILD/unittest2-1.1.0/setup.py", line 87, in <module>
    setup(**params)
  File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.9/distutils/dist.py", line 984, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib64/python3.9/distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/usr/lib/python3.9/site-packages/setuptools/command/install.py", line 38, in finalize_options
    orig.install.finalize_options(self)
  File "/usr/lib64/python3.9/distutils/command/install.py", line 292, in finalize_options
    'dist_fullname': self.distribution.get_fullname(),
  File "/usr/lib64/python3.9/distutils/dist.py", line 1166, in get_fullname
    return "%s-%s" % (self.get_name(), self.get_version())
  File "/builddir/build/BUILD/unittest2-1.1.0/setup.py", line 15, in __str__
    from unittest2 import __version__ as VERSION
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/__init__.py", line 40, in <module>
    from unittest2.collector import collector
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/collector.py", line 3, in <module>
    from unittest2.loader import defaultTestLoader
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/loader.py", line 13, in <module>
    from unittest2 import case, suite, util
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/case.py", line 18, in <module>
    from unittest2 import result
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/result.py", line 10, in <module>
    from unittest2.compatibility import wraps
  File "/builddir/build/BUILD/unittest2-1.1.0/unittest2/compatibility.py", line 143, in <module>
    class ChainMap(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

This is due to MutableMapping being now accessible only through collections.abc.

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01141957-python-unittest2/

For all our attempts to build python-unittest2 with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-unittest2/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33, but the initial bootstrapping has already started.
A build failure this early in the bootstrap sequence blocks us very much.

Comment 1 Charalampos Stratakis 2020-01-09 02:48:44 UTC
This can be fixed with something like:

diff --git a/unittest2/compatibility.py b/unittest2/compatibility.py
index 9e5f1a5..0eedfa7 100644
--- a/unittest2/compatibility.py
+++ b/unittest2/compatibility.py
@@ -1,3 +1,4 @@
+import collections.abc as abc
 import collections
 import os
 import sys
@@ -140,7 +141,7 @@ except ImportError:
 ###  ChainMap (helper for configparser and string.Template)
 ########################################################################
 
-class ChainMap(collections.MutableMapping):
+class ChainMap(abc.MutableMapping):
     ''' A ChainMap groups multiple dicts (or other mappings) together
     to create a single, updateable view.

Comment 2 Avram Lubkin 2020-01-09 11:42:34 UTC
Why would we fix this? Shouldn't we fix the 20 packages that incorrectly require unittest2 and make unittest2 EPEL-only?

For some background, unittest2 is a backport of unittest for Python 2.6 since significant changes were added in 2.7. So why would we need this for Python 3.9? We don't.

I checked a few of these and for most of them we can just remove the line in the spec file for unittest2. The packages either don't use it or they have 2.6 compatibility and do a conditional import. One or two may need a small patch. All of these are srpms except python3-testtools.


$ dnf repoquery --disablerepo=* --enablerepo=fedora{,-source} --releasever=rawhide --whatrequires python3-unittest2 --refresh

ProDy-0:1.10.10-4.fc32.src
beaker-0:27.0-1.fc32.src
fedpkg-0:1.37-10.fc32.src
obs-service-tar_scm-0:0.10.10-2.fc32.src
pyephem-0:3.7.6.0-17.fc32.src
python-agate-0:1.6.1-7.fc32.src
python-billiard-1:3.6.1.0-3.fc32.src
python-case-0:1.5.3-4.fc32.src
python-funcsigs-0:1.0.2-17.fc32.src
python-leather-0:0.3.3-13.gite85dd30.fc32.src
python-linecache2-0:1.0.0-24.fc32.src
python-mimerender-0:0.6.0-5.fc32.src
python-pyclipper-0:1.1.0-8.fc32.src
python-pynwb-0:1.1.2-1.fc32.src
python-rsa-0:3.4.2-12.fc32.src
python-social-auth-core-0:1.7.0-8.fc32.src
python-testtools-0:2.3.0-15.fc32.src
python-traceback2-0:1.4.0-25.fc32.src
python3-testtools-0:2.3.0-15.fc32.noarch
tuned-0:2.13.0-1.fc32.src

Comment 3 Miro Hrončok 2020-01-09 12:37:33 UTC
> Shouldn't we fix the 20 packages that incorrectly require unittest2 and make unittest2 EPEL-only?

Yes! I've contacted the maintainers repeatedly about this, but this is what we have (left). Somebody can do a heroic effort to fix all of them, but it won't be me.

In the mean time, before somebody does, let's fix this one instead?

Comment 4 Petr Viktorin (pviktori) 2020-01-09 12:41:04 UTC
> A build failure this early in the bootstrap sequence blocks us very much.

Which dependencies are the blocking ones?

Comment 5 Miro Hrončok 2020-01-09 12:51:19 UTC
We bootstrap to fedpkg - in order to have a working rawhide development environment once the side tag is merged.

So the obvious one is fedpkg, but that has been fixed: https://src.fedoraproject.org/rpms/fedpkg/c/245d4d31c29352f975c74a92d4ba2cb650e6f9b7?branch=master



testrepository and testscenario need testtools are needed to build pbr and runtime for fixtures. pbr and testools combined are needed very much, mostly by the oslo stack, but also other packages.

$ repoquery --repo=rawhide{,-source} --whatrequires python3-fixtures | pkgname
dlrn
python-bashate
python-congressclient
python-debtcollector
python-gear
python-keystoneauth1
python-keystoneclient
python-linecache2
python-magnumclient
python-mox3
python-openstackclient
python-os-client-config
python-osc-lib
python-oslo-concurrency
python-oslo-config
python-oslo-context
python-oslo-db
python-oslo-i18n
python-oslo-utils
python-pyvmomi
python-requests-mock
python-sphinxcontrib-spelling
python-stestr
python-subunit2sql
python-tackerclient
python-testrepository
python-testresources
python-testscenarios
python-tosca-parser
python-traceback2
python3-congressclient-tests
python3-keystoneclient-tests
python3-magnumclient-tests
python3-mox3
python3-octaviaclient-tests
python3-osc-lib-tests
python3-oslo-concurrency-tests
python3-oslo-db-tests
python3-oslo-utils-tests
python3-oslotest
python3-pifpaf
python3-stestr
python3-tackerclient-tests-unit
python3-testrepository
python3-testtools
subunit

$ repoquery --repo=rawhide{,-source} --whatrequires python3-pbr | pkgname
ara
bandit
dib-utils
diskimage-builder
diskimage-builder
dlrn
git-pull-request
git-review
guake
guake
heat-cfntools
heat-cfntools
nodepool
nodepool
pylast
python-ZEO
python-aioresponses
python-aodhclient
python-automaton
python-barbicanclient
python-bashate
python-cinderclient
python-cliff
python-congressclient
python-cotyledon
python-couchbase
python-cradox
python-daiquiri
python-debtcollector
python-designateclient
python-distroinfo
python-django-helpdesk
python-fedmsg-meta-fedora-infrastructure
python-fixtures
python-freecell_solver
python-futurist
python-gabbi
python-gear
python-gerritlib
python-gertty
python-glanceclient
python-gnocchiclient
python-hacking
python-heatclient
python-ironicclient
python-jenkins
python-jenkins-job-builder
python-jira
python-jsonpath-rw-ext
python-keystoneauth1
python-keystoneclient
python-ldappool
python-linecache2
python-lockfile
python-magnumclient
python-manilaclient
python-migrate
python-mistralclient
python-molecule
python-mox3
python-murano-pkg-check
python-neutronclient
python-novaclient
python-octaviaclient
python-openstack-doc-tools
python-openstackclient
python-openstackdocstheme
python-openstacksdk
python-os-client-config
python-os-service-types
python-os-testr
python-osc-lib
python-oslo-concurrency
python-oslo-config
python-oslo-context
python-oslo-db
python-oslo-i18n
python-oslo-log
python-oslo-serialization
python-oslo-sphinx
python-oslo-utils
python-oslotest
python-pifpaf
python-positional
python-pyghmi
python-pymod2pkg
python-pysol-cards
python-renderspec
python-reno
python-requests-mock
python-requests-unixsocket
python-requestsexceptions
python-rsd-lib
python-saharaclient
python-shade
python-sphinx-click
python-sphinxcontrib-apidoc
python-sphinxcontrib-fulltoc
python-sphinxcontrib-pecanwsme
python-sphinxcontrib-spelling
python-stestr
python-stevedore
python-subunit2sql
python-sushy
python-swiftclient
python-tackerclient
python-taskflow
python-tenacity
python-testinfra
python-testscenarios
python-testtools
python-tosca-parser
python-traceback2
python-troveclient
python-virtualbmc
python-virtualenvwrapper
python-vitrageclient
python-yaql
python-zaqarclient
python-zuul-sphinx
python3-aodhclient
python3-ara
python3-automaton
python3-barbicanclient
python3-bashate
python3-cinderclient
python3-congressclient
python3-debtcollector
python3-designateclient
python3-distroinfo
python3-django-helpdesk
python3-dlrn
python3-etcd3gw
python3-fixtures
python3-freecell_solver
python3-futurist
python3-gabbi
python3-gear
python3-gerritlib
python3-gertty
python3-glanceclient
python3-gnocchiclient
python3-hacking
python3-heatclient
python3-ironicclient
python3-jenkins
python3-jira
python3-jsonpath-rw-ext
python3-keystoneauth1
python3-keystoneclient
python3-magnumclient
python3-manilaclient
python3-migrate
python3-mistralclient
python3-molecule
python3-mox3
python3-murano-pkg-check
python3-neutronclient
python3-novaclient
python3-octaviaclient
python3-openstack-doc-tools
python3-openstackclient
python3-openstackdocstheme
python3-openstacksdk
python3-os-service-types
python3-os-testr
python3-osc-lib
python3-oslo-concurrency
python3-oslo-config
python3-oslo-context
python3-oslo-db
python3-oslo-log
python3-oslo-sphinx
python3-pifpaf
python3-positional
python3-pymod2pkg
python3-pysol-cards
python3-pywbem
python3-rdopkg
python3-reno
python3-rsd-lib
python3-rsd-lib-tests
python3-rsdclient
python3-saharaclient
python3-shade
python3-sphinx-click
python3-sphinxcontrib-apidoc
python3-stestr
python3-stevedore
python3-subunit2sql
python3-sushy
python3-tackerclient
python3-tackerclient-tests-unit
python3-taskflow
python3-testinfra
python3-testscenarios
python3-testtools
python3-troveclient
python3-virtualbmc
python3-vitrageclient
python3-yaql
python3-zaqarclient
python3-zuul-sphinx
pywbem
rdopkg

Comment 6 Miro Hrončok 2020-01-09 12:52:21 UTC
> pbr and testools combined

I meant: pbr and fixtures combined

Comment 7 Zbigniew Jędrzejewski-Szmek 2020-01-09 13:12:04 UTC
I'll start by trying to remove the dependency in python-pbr.

Comment 8 Avram Lubkin 2020-01-09 13:15:48 UTC
Does python-pbr have a dependency on unittest2? It wasn't in the list.

Can one of you guys with commit access remove python-unittest2 and python-traceback2 from python-testtools? That should go a long way in fixing the dependencies.

Comment 9 Avram Lubkin 2020-01-09 13:21:10 UTC
Happy to do a pull request if needed, but in testing looks like a simple fix.

Comment 10 Zbigniew Jędrzejewski-Szmek 2020-01-09 13:23:11 UTC
> Does python-pbr have a dependency on unittest2?
Yeah, pbr doesn't. It must fail for some other reason.

> Happy to do a pull request if needed, but in testing looks like a simple fix.
If you already have the local patch, please do.

Comment 11 Miro Hrončok 2020-01-09 13:33:53 UTC
(In reply to Avram Lubkin from comment #8)
> Does python-pbr have a dependency on unittest2? It wasn't in the list.

pbr has:

BuildRequires:  python3-testrepository
BuildRequires:  python3-testresources
BuildRequires:  python3-testscenarios

It has a transitive dependency on unittest2.

Comment 12 Avram Lubkin 2020-01-09 13:37:07 UTC
PR created for python-testtools

https://src.fedoraproject.org/rpms/python-testtools/pull-request/5

Comment 13 Avram Lubkin 2020-01-09 13:46:23 UTC
Initial list had 19 packages (python-testtools was listed as rpm and srpm). fedpkg is done, so that's 19.

python-traceback2 seems to be only used by python-testtools and python-unittest so it should be easy to drop.

If python-testtools is done that should fix a lot of dependencies and leaves the list at 17.

Most of these are easy, just removing a line in the spec. It's possible that something might require a patch, but I haven't seen it yet.

If they will get merged, I can work on PRs today for some more of them.

Comment 14 Avram Lubkin 2020-01-09 13:47:58 UTC
Count is one off. I meant we're at 18 now, after python-testtools patch is merged, that's an effective 16.

Comment 15 Alfredo Moralejo 2020-01-09 14:03:14 UTC
(In reply to Miro Hrončok from comment #3)
> > Shouldn't we fix the 20 packages that incorrectly require unittest2 and make unittest2 EPEL-only?
> 
> Yes! I've contacted the maintainers repeatedly about this, but this is what
> we have (left). Somebody can do a heroic effort to fix all of them, but it
> won't be me.
> 
> In the mean time, before somebody does, let's fix this one instead?

I can fix unittest2 in the meanwhile to remove some pressure on the heroic effort.

Comment 16 Miro Hrončok 2020-01-09 14:06:02 UTC
Thanks, Alfredo.

Comment 17 Avram Lubkin 2020-01-09 14:07:12 UTC
I was advocating we don't fix it. I wouldn't call the effort heroic. It's 17 packages. We should be able to knock this out.

Comment 18 Miro Hrončok 2020-01-09 14:10:49 UTC
> I was advocating we don't fix it

This is not a good reason not to fix it. Please, don't do that.

Comment 19 Avram Lubkin 2020-01-09 14:14:46 UTC
Explain why this is not a good reason. This package should be retired. The dependencies are incorrect and are either ignored or will cause the tests to use a version of unittest that doesn't match the standard library of the Python version being used. It's technical debt, so we should fix it. By fixing unittest2 it removes the incentive to do it quickly.

Comment 20 Alfredo Moralejo 2020-01-09 14:28:21 UTC
(In reply to Avram Lubkin from comment #19)
> Explain why this is not a good reason. This package should be retired. The
> dependencies are incorrect and are either ignored or will cause the tests to
> use a version of unittest that doesn't match the standard library of the
> Python version being used. It's technical debt, so we should fix it. By
> fixing unittest2 it removes the incentive to do it quickly.

I agree it's technical debt but iiuc some packages may need upstream changes to replace imports on unittest2 that will need time to get released?, i still think it's good to fix it in unittest2 until we can remove it everywhere.

Comment 21 Avram Lubkin 2020-01-09 14:35:07 UTC
Almost all of these are simple spec file fixes. There may be a couple that need a patch. If we run into a roadblock then it would make sense to carry unittest2 forward, but doesn't it make more sense to see if we hit a roadblock than to assume we will?

Comment 22 Miro Hrončok 2020-01-09 15:14:22 UTC
> Explain why this is not a good reason.

Because the failure is blocking us now. We need to carry a downstream-downstream patch in order to proceed at the moment.

This effort (getting rid of unittest2 for good) is very well needed, due time and appreciated, but it is not blocking anything now.

It is like refusing to fix a leaking pipe because you need a new boiler anyway. Sure, let's use the leaking pipe as a reason to actually go and get a new boiler, but at least duct tape it or turn off the water before you go shopping.

Comment 23 Avram Lubkin 2020-01-09 15:31:22 UTC
Your metaphor is flawed. In this case the boiler is python-unittest2. It's old and you don't need it any more, but it's plumbed into a bunch of rooms even though the building also has a new boiler. The old boiler is broken and you want to fix it instead of re-plumbing the rooms. I get it, but it's why these things stick around for years. It's ok for things to be broken in rawhide, that's what it's for. So what if we need a downstream patch? We have a lot of those.

Comment 24 Miro Hrončok 2020-01-09 15:35:19 UTC
Rawhide is not for being broken. Broken rawhide blocks us.

I need a downstream patch. Alfredo offered to carry it. Please let him.

Until that happens I need to carry a local patch over the Fedora package.

Comment 25 Avram Lubkin 2020-01-09 15:39:36 UTC
Oh, I thought you were talking about a patch for testtools. No one is stopping him .I just think it's a waste.

Comment 26 Miro Hrončok 2020-01-09 16:10:10 UTC
Thanks. You are most likely correct an I see your point.

Comment 27 Alfredo Moralejo 2020-01-09 16:14:57 UTC
https://src.fedoraproject.org/rpms/python-unittest2/pull-request/2 should be fine. I just rebuilt it using python 3.9 from the copr.

Comment 28 Zbigniew Jędrzejewski-Szmek 2020-01-09 16:19:35 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1789416 for retirement of python-traceback2.

Comment 29 Zbigniew Jędrzejewski-Szmek 2020-01-09 21:55:51 UTC
ProDy-0:1.10.10-4.fc32.src               dist-git
beaker-0:27.0-1.fc32.src                 koji
fedpkg-0:1.37-10.fc32.src                already fixed by Miro
obs-service-tar_scm-0:0.10.10-2.fc32.src dist-git
pyephem-0:3.7.6.0-17.fc32.src            https://src.fedoraproject.org/rpms/pyephem/pull-request/1
python-agate-0:1.6.1-7.fc32.src          koji
python-billiard-1:3.6.1.0-3.fc32.src     koji
python-case-0:1.5.3-4.fc32.src           koji
python-funcsigs-0:1.0.2-17.fc32.src      koji
python-leather-0:0.3.3-13.gite85dd30.fc32.src koji
python-linecache2-0:1.0.0-24.fc32.src    koji
python-mimerender-0:0.6.0-5.fc32.src     koji
python-pyclipper-0:1.1.0-8.fc32.src      koji
python-pynwb-0:1.1.2-1.fc32.src          koji
python-rsa-0:3.4.2-12.fc32.src           koji
python-social-auth-core-0:1.7.0-8.fc32.src  koji
python-testtools-0:2.3.0-15.fc32.src     TBD
python-traceback2-0:1.4.0-25.fc32.src    https://bugzilla.redhat.com/show_bug.cgi?id=1789416 
tuned-0:2.13.0-1.fc32.src                koji

(koji means built in koji without BR:unittest2, dist-git means pushed but not built)

Comment 30 Avram Lubkin 2020-01-09 22:14:47 UTC
Thanks for all that! I did start looking at a patch for testtools.

python-rsa
https://src.fedoraproject.org/rpms/python-rsa/pull-request/2

Comment 31 Miro Hrončok 2020-01-17 11:26:20 UTC
Fixed. Feel free to open a separate bug for the retirement, but please don't reopen this one unless there is a failure with 3.9, we use the bugzillas for tracking. Thanks.


Note You need to log in before you can comment on or make changes to this bug.