1792956 – python-passlib fails to build with Python 3.9: imports abc from collections + _crypt.crypt [Errno 22] Invalid argument

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1792956 - python-passlib fails to build with Python 3.9: imports abc from collections + _crypt.crypt [Errno 22] Invalid argument

Summary: python-passlib fails to build with Python 3.9: imports abc from collections +...

Keywords:
Status:	CLOSED RAWHIDE
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	python-passlib
Sub Component:
Version:	rawhide
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Assignee:	Alan Pevec (Fedora)
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	PYTHON39
TreeView+	depends on / blocked

Reported:	2020-01-20 12:52 UTC by Miro Hrončok
Modified:	2020-02-28 15:23 UTC (History)
CC List:	6 users (show)
Fixed In Version:	python-passlib-1.7.2-1.fc33
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-02-28 15:23:34 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Miro Hrončok 2020-01-20 12:52:43 UTC

python-passlib fails to build with Python 3.9.0a2.

======================================================================
ERROR: ldap_sha1_crypt (os_crypt backend): test hash() / genconfig() honors min_salt_size
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/tests/utils.py", line 1293, in test_12_min_salt_size
    self.do_encrypt('stub', salt_size=min_size)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/tests/utils.py", line 878, in do_encrypt
    return (handler or self.handler).using(**settings).hash(secret, **context)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/handlers.py", line 2669, in hash
    return self._wrap_hash(self.wrapped.hash(secret, **kwds))
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/handlers.py", line 748, in hash
    self.checksum = self._calc_checksum(secret)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/handlers.py", line 2361, in _calc_checksum
    return self._calc_checksum_backend(secret)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/handlers/sha1_crypt.py", line 111, in _calc_checksum_os_crypt
    hash = safe_crypt(secret, config)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/__init__.py", line 789, in safe_crypt
    result = _crypt(secret, hash)
  File "/usr/lib64/python3.9/crypt.py", line 82, in crypt
    return _crypt.crypt(word, salt)
OSError: [Errno 22] Invalid argument
======================================================================
ERROR: sha1_crypt (os_crypt backend): test hash() / genconfig() honors min_salt_size
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/tests/utils.py", line 1293, in test_12_min_salt_size
    self.do_encrypt('stub', salt_size=min_size)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/tests/utils.py", line 878, in do_encrypt
    return (handler or self.handler).using(**settings).hash(secret, **context)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/handlers.py", line 748, in hash
    self.checksum = self._calc_checksum(secret)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/handlers.py", line 2361, in _calc_checksum
    return self._calc_checksum_backend(secret)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/handlers/sha1_crypt.py", line 111, in _calc_checksum_os_crypt
    hash = safe_crypt(secret, config)
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/utils/__init__.py", line 789, in safe_crypt
    result = _crypt(secret, hash)
  File "/usr/lib64/python3.9/crypt.py", line 82, in crypt
    return _crypt.crypt(word, salt)
OSError: [Errno 22] Invalid argument
======================================================================
ERROR: Failure: ImportError (cannot import name 'MutableMapping' from 'collections' (/usr/lib64/python3.9/collections/__init__.py))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.9/site-packages/nose/loader.py", line 416, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/usr/lib/python3.9/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.9/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib64/python3.9/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib64/python3.9/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 786, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/tests/test_pwd.py", line 60, in <module>
    from passlib.pwd import genword, default_charsets
  File "/builddir/build/BUILD/passlib-1.7.1/passlib/pwd.py", line 8, in <module>
    from collections import defaultdict, MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/lib64/python3.9/collections/__init__.py)
----------------------------------------------------------------------
Ran 3013 tests in 165.558s
FAILED (SKIP=1267, errors=3)


See https://docs.python.org/3.9/whatsnew/3.9.html#removed

"The abstract base classes in collections.abc no longer are exposed in the regular collections module. This will help create a clearer distinction between the concrete classes and the abstract base classes."


For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01161815-python-passlib/

For all our attempts to build python-passlib with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-passlib/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Karthikeyan Singaravelan 2020-01-22 10:25:27 UTC

collections import seems to fixed with https://bitbucket.org/ecollins/passlib/commits/ff226f9dd5d892546973802ae45004bb88f1efc8 . Thanks.

Comment 2 Miro Hrončok 2020-02-05 13:58:14 UTC

Alan, could you please backport the fix? This is blocking the rebuild of ansible.

Comment 3 Ben Cotton 2020-02-11 17:18:50 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 4 Miro Hrončok 2020-02-25 17:54:27 UTC

Please?

Comment 5 Alan Pevec 2020-02-26 22:32:55 UTC

 ff226f9 is included in 1.7.2

*** This bug has been marked as a duplicate of bug 1775827 ***

Comment 6 Alan Pevec 2020-02-27 09:22:44 UTC

1.7.2 does not solve ERROR: sha1_crypt (os_crypt backend): test hash() / genconfig() honors min_salt_size
...
  File "/usr/lib64/python3.9/crypt.py", line 82, in crypt
    return _crypt.crypt(word, salt)
OSError: [Errno 22] Invalid argument

Comment 7 Alan Pevec 2020-02-27 11:00:04 UTC

library docs don't show changes in 3.9:
https://docs.python.org/3.9/library/crypt.html#crypt.crypt

but it is more strict in 3.9, I could reproduce this error with empty salt

Python 3.7.6 (default, Jan 30 2020, 09:44:41) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import crypt
>>> crypt.crypt("sikrit","")
>>>

Python 3.9.0a3 (default, Jan 27 2020, 00:00:00) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import crypt
>>> crypt.crypt("sikrit","")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/crypt.py", line 82, in crypt
    return _crypt.crypt(word, salt)
OSError: [Errno 22] Invalid argument

Comment 8 Alan Pevec 2020-02-27 11:03:14 UTC

While at it, 3.9 breaks nose, passlib master branch python ./setup.py test in py39 venv fails with
...
  File "/home/apevec/src/passlib/.eggs/nose-1.3.7-py3.9.egg/nose/suite.py", line 106, in _set_tests
    if isinstance(tests, collections.Callable) and not is_suite:
AttributeError: module 'collections' has no attribute 'Callable'

Comment 9 Miro Hrončok 2020-02-27 11:13:34 UTC

Fedora's nose is patched. Upstream nose is broken for ages, dead and you should run away form it fast.

See also: https://fedoraproject.org/wiki/Changes/DeprecateNose


As a side note, the collections.abc thing was reverted for 3.9.0a4 (but will hit again in 3.10 and this time, they made it pretty clear that there will be no more delaying).

Comment 10 Alan Pevec 2020-02-27 23:53:38 UTC

ack for Nose - it's up to upstream to decide

re. crypt: in 3.9 it errors with some salt values e.g.
Python 3.9.0a3 (default, Jan 27 2020, 00:00:00)
...
>>> crypt.crypt('stub','$sha1$60000$')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/crypt.py", line 83, in crypt
    return _crypt.crypt(word, salt)
OSError: [Errno 22] Invalid argument

which silently returns nothing in 3.8 and earlier:
Python 3.8.1 (default, Dec 19 2019, 00:00:00) 
...
>>> crypt.crypt('stub','$sha1$60000$')
>>> 

$ git log -p v3.8.1..v3.9.0a3 -- Modules/_cryptmodule.c
commit 0d3fe8ae4961bf551e7d5e42559e2ede1a08fd7c
Author: Antonio Gutierrez <chibby0ne>
Date:   Tue Oct 8 06:22:17 2019 +0200

    closes bpo-38402: Check error of primitive crypt/crypt_r. (GH-16599)
    
    Checks also for encryption algorithms methods not supported in different
    OSs.
    
    Signed-off-by: Antonio Gutierrez <chibby0ne>

diff --git a/Modules/_cryptmodule.c b/Modules/_cryptmodule.c
index 5d03f45f64..00c1f4f698 100644
--- a/Modules/_cryptmodule.c
+++ b/Modules/_cryptmodule.c
@@ -42,6 +42,9 @@ crypt_crypt_impl(PyObject *module, const char *word, const char *salt)
 #else
     crypt_result = crypt(word, salt);
 #endif
+    if (crypt_result == NULL) {
+        return PyErr_SetFromErrno(PyExc_OSError);
+    }
     return Py_BuildValue("s", crypt_result);
 }

Comment 11 Alan Pevec 2020-02-28 01:02:22 UTC

With this patch test suite passes with Python 3.9:

diff -r bd0da1bdf0fd passlib/utils/__init__.py
--- a/passlib/utils/__init__.py Sun Feb 16 10:56:06 2020 -0500
+++ b/passlib/utils/__init__.py Fri Feb 28 02:01:08 2020 +0100
@@ -791,7 +791,10 @@
                 raise ValueError("null character in secret")
             if isinstance(hash, bytes):
                 hash = hash.decode("ascii")
-            result = _crypt(secret, hash)
+            try:
+                result = _crypt(secret, hash)
+            except OSError:
+                result = None
             # NOTE: per issue 113, crypt() may return bytes in some odd cases.
             #       assuming it should still return an ASCII hash though,
             #       or there's a bigger issue at hand.

Comment 12 Alan Pevec 2020-02-28 01:12:05 UTC

rebased to 1.7.2

diff -r 48058c4309dd passlib/utils/__init__.py
--- a/passlib/utils/__init__.py Fri Nov 22 16:08:33 2019 -0500
+++ b/passlib/utils/__init__.py Fri Feb 28 02:10:14 2020 +0100
@@ -791,7 +791,10 @@
                 raise ValueError("null character in secret")
             if isinstance(hash, bytes):
                 hash = hash.decode("ascii")
-            result = _crypt(secret, hash)
+            try:
+                result = _crypt(secret, hash)
+            except OSError:
+                result = None
             if not result or result[0] in _invalid_prefixes:
                 return None
             return result

Comment 13 Alan Pevec 2020-02-28 01:24:44 UTC

Reported upstream https://bitbucket.org/ecollins/passlib/issues/115

Comment 14 Alan Pevec 2020-02-28 01:32:07 UTC

Included in https://src.fedoraproject.org/rpms/python-passlib/pull-request/5

Note You need to log in before you can comment on or make changes to this bug.