Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1791765 - coccinelle fails to build with Python 3.9: unable to find the Python library
Summary: coccinelle fails to build with Python 3.9: unable to find the Python library
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: coccinelle
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-16 12:45 UTC by Miro Hrončok
Modified: 2020-06-01 19:13 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-30 09:16:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Miro Hrončok 2020-01-16 12:45:31 UTC
coccinelle fails to build with Python 3.9.0a2.

+ /builddir/build/BUILDROOT/coccinelle-1.0.8-2.fc32.x86_64/usr/lib64/coccinelle/spatch --sp-file /builddir/build/SOURCES/testpy.cocci /builddir/build/SOURCES/test.c
init_defs_builtins: /builddir/build/BUILDROOT/coccinelle-1.0.8-2.fc32.x86_64/usr/lib64/coccinelle/standard.h
Py.find_library: unable to find the Python library [/lib64/libpython3.9.so.1.0 returned Cannot resolve PyImport_Cleanup.
]

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01148357-coccinelle/

For all our attempts to build coccinelle with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/coccinelle/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Victor Stinner 2020-01-16 14:30:18 UTC
It smells like something is not linked explicitly to libpython.

Comment 2 Richard W.M. Jones 2020-01-16 16:30:21 UTC
Could it be we need to use python-3.X-embed instead?  See also
a similar bug for nbdkit: https://bugzilla.redhat.com/show_bug.cgi?id=1718294

Comment 3 Richard W.M. Jones 2020-01-16 16:37:25 UTC
Ignore previous comment.

It looks as if the code shouldn't using using PyImport_Cleanup.  Note
that it is marked as deprecated here:

https://docs.python.org/3/c-api/import.html

I have raised this issue with upstream.

Comment 4 Victor Stinner 2020-01-17 13:22:22 UTC
> It looks as if the code shouldn't using using PyImport_Cleanup.

Oh wait, I misread the error "/lib64/libpython3.9.so.1.0 returned Cannot resolve PyImport_Cleanup".

I removed PyImport_Cleanup() from libpython on purpose: you should really not call it.

"The C function PyImport_Cleanup() has been removed. It was documented as: “Empty the module table. For internal use only.” (Contributed by Victor Stinner in bpo-36710.)"

https://docs.python.org/dev/whatsnew/3.9.html#removed

Maybe Coccinelle has a good reason to call it? If yes, I'm open to discuss reverting my change in Python 3.9 :-)

> I have raised this issue with upstream.

Did you open a public issue? I don't see anything at https://github.com/coccinelle/coccinelle/issues

Comment 5 Richard W.M. Jones 2020-01-17 13:41:18 UTC
I mentioned it here: https://systeme.lip6.fr/pipermail/cocci/2020-January/006785.html
I did not open a formal bug.

It sounds as if Cocci probably should not be calling this private
API, and if you removed it then you exposed that previously hidden
bug.  Anyway we can see what they say.

Comment 6 Ben Cotton 2020-02-11 17:33:57 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 7 Richard W.M. Jones 2020-03-25 19:08:26 UTC
Still present in coccinelle-1.0.9-0.1.111d328fee1303f14a5b9def835301d849e41331.fc33

Comment 8 Victor Stinner 2020-03-26 13:01:23 UTC
I reported the bug upstream: https://github.com/coccinelle/coccinelle/issues/189

I also proposed a fix: https://github.com/coccinelle/coccinelle/pull/190 (not tested)

Comment 9 Miro Hrončok 2020-05-25 13:01:02 UTC
This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.


The Python 3.9 rebuild is in progress in a Koji side tag.

If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:

    $ fedpkg build --target=f33-python

The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.

You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.

If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Comment 10 Richard W.M. Jones 2020-05-26 08:42:59 UTC
Note mainly for my own benefit.

This has been fixed upstream:
https://github.com/thierry-martinez/pyml/commit/8b26fdb9f60c3a66b58f2ecc1b2136a833f55612

We probably need to add pyml as a separate BR so that coccinelle stops using the
bundled version, and make sure the fix is in Fedora.

Since this is a complex multi-package fix I think I prefer to wait until Python 3.9
is actually in Rawhide.

Comment 11 Miro Hrončok 2020-05-29 07:11:11 UTC
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.

Comment 12 Richard W.M. Jones 2020-05-30 09:16:41 UTC
It turns out that the pyml library isn't packaged in Fedora and we are
carrying the right Bundled tags & exceptions.  I have built the package
in Fedora Rawhide now against Python 3.9.0~b1-4.fc33:

https://koji.fedoraproject.org/koji/taskinfo?taskID=45171978

Comment 13 Victor Stinner 2020-06-01 19:13:06 UTC
Thanks for fixing the package!


Note You need to log in before you can comment on or make changes to this bug.