Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1897148 - m2crypto fails to build with Python 3.10: PyObject_AsReadBuffer() is removed
Summary: m2crypto fails to build with Python 3.10: PyObject_AsReadBuffer() is removed
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: m2crypto
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Neal Gompa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1968836 (view as bug list)
Depends On:
Blocks: PYTHON3.10 F35FTBFS, RAWHIDEFTBFS F35FailsToInstall, RAWHIDEFailsToInstall 1968820 1968843 1968854
TreeView+ depends on / blocked
 
Reported: 2020-11-12 12:45 UTC by Tomáš Hrnčiar
Modified: 2021-06-25 03:21 UTC (History)
7 users (show)

Fixed In Version: m2crypto-0.38.0-1.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-25 03:21:47 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)
An incomplete patch (4.17 KB, patch)
2021-06-09 10:38 UTC, Miro Hrončok 		no flags Details | Diff
An incomplete patch (3.60 KB, patch)
2021-06-09 10:44 UTC, Miro Hrončok 		no flags Details | Diff
An incomplete patch (3.76 KB, patch)
2021-06-09 10:49 UTC, Miro Hrončok 		no flags Details | Diff
Patch that owkrs with Python 3.10 (3.75 KB, patch)
2021-06-09 13:33 UTC, Miro Hrončok 		no flags Details | Diff
Show Obsolete (3) View All

Description Tomáš Hrnčiar 2020-11-12 12:45:36 UTC 
m2crypto fails to build with Python 3.10.0a2.

ImportError: /builddir/build/BUILD/M2Crypto-0.35.2/M2Crypto/_m2crypto.cpython-310-x86_64-linux-gnu.so: undefined symbol: PyObject_AsReadBuffer

bpo-41103: PyObject_AsCharBuffer(), PyObject_AsReadBuffer(), PyObject_CheckReadBuffer(), and PyObject_AsWriteBuffer() are removed. Please migrate to new buffer protocol; PyObject_GetBuffer() and PyBuffer_Release().

https://docs.python.org/3.10/whatsnew/changelog.html#id16

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01757146-m2crypto/

For all our attempts to build m2crypto with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/m2crypto/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.
Comment 1 Miro Hrončok 2020-11-27 14:18:28 UTC 
BTW 0.36.0 does not fix this, see https://src.fedoraproject.org/rpms/m2crypto/pull-request/4 and https://copr.fedorainfracloud.org/coprs/g/python/python3.10/builds/?dirname=python3.10:pr:4
Comment 2 Ben Cotton 2021-02-09 15:23:49 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.
Comment 3 Tomáš Hrnčiar 2021-03-29 12:17:21 UTC 
Hello Neal,

is there any update on this? There are two packages depending on m2crypto (ipsilon and module-build-service), we cannot test them with Python 3.10 without it. 

Thank you
Comment 4 Neal Gompa 2021-03-29 14:10:25 UTC 
Sorry, I forgot to close this when I updated m2crypto 0.37.1. This should be fixed now: https://gitlab.com/m2crypto/m2crypto/-/commit/6bb65c27f01dd69f08c4654ce531fb283c7bcc07
Comment 5 Miro Hrončok 2021-03-29 14:13:17 UTC 
But it is not. See https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/m2crypto/
Comment 6 Miro Hrončok 2021-06-04 20:14:12 UTC 
This is a mass-posted update. Sorry if it is not 100% accurate to this bugzilla.


The Python 3.10 rebuild is in progress in a Koji side tag. If you manage to fix the problem, please commit the fix in the rawhide branch, but don't build the package in regular rawhide.

You can either build the package in the side tag, with:

    $ fedpkg build --target=f35-python

Or you can the build and we will eventually build it for you.

Note that the rebuild is still in progress, so not all (build) dependencies of this package might be available right away.

Thanks.

See also https://fedoraproject.org/wiki/Changes/Python3.10

If you have general questions about the rebuild, please use this mailing list thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/G47SGOYIQLRDTWGOSLSWERZSSHXDEDH5/
Comment 7 Miro Hrončok 2021-06-07 22:58:52 UTC 
The f35-python side tag has been merged to Rawhide. From now on, build as you would normally build.
Comment 8 Miro Hrončok 2021-06-08 11:17:09 UTC 
*** Bug 1968836 has been marked as a duplicate of this bug. ***
Comment 9 Neal Gompa 2021-06-09 09:42:27 UTC 
I don't know how to fix this, Miro. :(
Comment 10 Miro Hrončok 2021-06-09 10:23:59 UTC 
SWIG/_lib.i has:

...
    if (PyObject_CheckBuffer(obj)) {
...
    } else {
...
         PyObject_AsReadBuffer(obj, buffer, &len)
...
    }


The else branch cannot use PyObject_AsReadBuffer. I don't really know what should be done if PyObject_CheckBuffer() is 0. What is the idea here?
Comment 11 Miro Hrončok 2021-06-09 10:38:24 UTC 
Created attachment 1789543 [details]
An incomplete patch

The attached patch makes it compile and import, but I guess some alternate else branches must be provided, otherwise there is some missing exception handling.



======================================================================
ERROR: test_private_encrypt (tests.test_rsa.RSATestCase)
----------------------------------------------------------------------
TypeError: a bytes-like object is required, not 'method'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builddir/build/BUILD/M2Crypto-0.37.1/tests/test_rsa.py", line 114, in test_private_encrypt
    priv.private_encrypt(self.gen_callback, RSA.pkcs1_padding)
  File "/builddir/build/BUILD/M2Crypto-0.37.1/M2Crypto/RSA.py", line 77, in private_encrypt
    return m2.rsa_private_encrypt(self.rsa, data, padding)
SystemError: <built-in function rsa_private_encrypt> returned a result with an exception set

======================================================================
ERROR: test_public_encrypt (tests.test_rsa.RSATestCase)
----------------------------------------------------------------------
TypeError: a bytes-like object is required, not 'method'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builddir/build/BUILD/M2Crypto-0.37.1/tests/test_rsa.py", line 133, in test_public_encrypt
    priv.public_encrypt(self.gen_callback, RSA.pkcs1_padding)
  File "/builddir/build/BUILD/M2Crypto-0.37.1/M2Crypto/RSA.py", line 67, in public_encrypt
    return m2.rsa_public_encrypt(self.rsa, data, padding)
SystemError: <built-in function rsa_public_encrypt> returned a result with an exception set

----------------------------------------------------------------------
Comment 12 Miro Hrončok 2021-06-09 10:44:13 UTC 
Created attachment 1789544 [details]
An incomplete patch

An updated but still incomplete patch
Comment 13 Miro Hrončok 2021-06-09 10:49:16 UTC 
Created attachment 1789545 [details]
An incomplete patch

Progress:

======================================================================
ERROR: test_private_encrypt (tests.test_rsa.RSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/M2Crypto-0.37.1/tests/test_rsa.py", line 114, in test_private_encrypt
    priv.private_encrypt(self.gen_callback, RSA.pkcs1_padding)
  File "/builddir/build/BUILD/M2Crypto-0.37.1/M2Crypto/RSA.py", line 77, in private_encrypt
    return m2.rsa_private_encrypt(self.rsa, data, padding)
ValueError: help me, I am trapped in this patch

======================================================================
ERROR: test_public_encrypt (tests.test_rsa.RSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/M2Crypto-0.37.1/tests/test_rsa.py", line 133, in test_public_encrypt
    priv.public_encrypt(self.gen_callback, RSA.pkcs1_padding)
  File "/builddir/build/BUILD/M2Crypto-0.37.1/M2Crypto/RSA.py", line 67, in public_encrypt
    return m2.rsa_public_encrypt(self.rsa, data, padding)
ValueError: help me, I am trapped in this patch

----------------------------------------------------------------------
Ran 370 tests in 70.509s


I still have no idea what should be done if PyObject_CheckBuffer fails :/
Comment 14 Miro Hrončok 2021-06-09 13:33:10 UTC 
Created attachment 1789565 [details]
Patch that owkrs with Python 3.10

Thanks to Petr Viktorin, I have a working patch:

Attached and in https://src.fedoraproject.org/rpms/m2crypto/pull-request/6

The problem was in the exception type I was rising.


Not sure if applicable upstream as is, I leave that you you. It is likely not Python 2 compatible and not even tested on Python 3.9 by me.
Comment 15 Miro Hrončok 2021-06-09 13:33:43 UTC 
> Patch that owkrs with Python 3.10

Should have been "works"
Comment 16 Petr Viktorin 2021-06-09 13:39:21 UTC 
The changes to m2_PyObject_AsReadBuffer should be good for any Python.

Raising TypeError rather than trying PyObject_AsReadBuffer is OK on Python 3: up to 3.9, the old buffer protocol is just a wrapper over the new one. But in Python 2, I think it is possible for an object to implement the old protocol but not the new one.
So in those 2 places (m2_PyObject_AsReadBuffer*), upstream you might want to add #ifdef on the Python version.
Comment 17 Matěj Cepl 2021-06-13 12:27:13 UTC 
Recognized upstream https://gitlab.com/m2crypto/m2crypto/-/issues/297
Comment 18 Matěj Cepl 2021-06-14 21:03:12 UTC 
M2Crypto 0.38.0 should be what you are looking for. I hope.
Comment 19 Neal Gompa 2021-06-25 03:21:47 UTC 
Released into Rawhide: https://src.fedoraproject.org/rpms/m2crypto/c/e7d562393c5a1981cb5cad00e6ce55e6c4d1e2aa?branch=rawhide
Note You need to log in before you can comment on or make changes to this bug.