Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1792062 - python-jep fails to build with Python 3.9: daemon thread are not supported in subinterpreters
Summary: python-jep fails to build with Python 3.9: daemon thread are not supported in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-jep
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Raphael Groner
QA Contact: Fedora Extras Quality Assurance
URL: https://src.fedoraproject.org/rpms/py...
Whiteboard:
: 1842128 (view as bug list)
Depends On: 1885098
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-16 23:15 UTC by Miro Hrončok
Modified: 2021-02-10 01:19 UTC (History)
7 users (show)

Fixed In Version: python-jep-3.9.1-1.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-01 18:52:58 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ninia jep issues 202 0 None open Features for Jep 4.0 2021-02-01 16:54:56 UTC
Github ninia jep issues 229 0 None closed jep is incompatible with Python 3.9: daemon threads can no longer be used in subinterpreters 2021-02-01 16:54:57 UTC
Python 37266 0 None None None 2020-02-07 00:14:07 UTC

Description Miro Hrončok 2020-01-16 23:15:23 UTC 
python-jep fails to build with Python 3.9.0a2.

FAIL: test_shared_modules_threads (test_shared_modules.TestSharedModules)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/jep-3.9.0/src/test/python/test_shared_modules.py", line 15, in test_shared_modules_threads
    jep_pipe(build_java_process_cmd('jep.test.TestSharedModulesThreads'))
  File "/usr/lib64/python3.9/contextlib.py", line 240, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/lib64/python3.9/contextlib.py", line 83, in __init__
    self.gen = func(*args, **kwds)
  File "/builddir/build/BUILD/jep-3.9.0/src/test/python/jep_pipe.py", line 36, in jep_pipe
    assert False, stderr
AssertionError: b'Exception in thread "main" jep.JepException: <class \'RuntimeError\'>: daemon thread are not supported in subinterpreters\n\tat /usr/lib64/python3.9/threading.start(threading.py:858)\n\tat <string>.<module>(<string>:1)\n\tat jep.Jep.eval(Native Method)\n\tat jep.Jep.eval(Jep.java:451)\n\tat jep.test.TestSharedModulesThreads.main(TestSharedModulesThreads.java:53)\n'

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01149085-python-jep/

For all our attempts to build python-jep with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-jep/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.
Comment 1 Raphael Groner 2020-02-07 00:14:07 UTC 
Upstream issue: https://bugs.python.org/issue37266

Because of the reason given there, I doubt we can do anything as downstream to fix (CANTFIX).
Comment 2 Raphael Groner 2020-02-07 00:14:47 UTC 
Upstream issue: https://bugs.python.org/issue37266

Because of the reason given there, I doubt we can *not* do anything as downstream to fix (CANTFIX).
Comment 3 Miro Hrončok 2020-02-07 00:19:58 UTC 
Raphael, I suggest you contact jep upstream about this.

Victor, what can they do to fix this?
Comment 4 Victor Stinner 2020-02-07 10:37:48 UTC 
> Victor, what can they do to fix this?

python-jep should use regular threads rather than daemon threads in subinterpreters.

Daemon threads in subinterpreters can crash at Python exit.
Comment 5 Victor Stinner 2020-02-07 10:41:52 UTC 
I reported the issue to jep: https://github.com/ninia/jep/issues/229
Comment 6 Raphael Groner 2020-02-10 09:01:49 UTC 
py4j as an alternative seems fixed, so we'd do as well for python-jep.
Comment 7 Ben Cotton 2020-02-11 16:30:59 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.
Comment 8 Raphael Groner 2020-02-15 09:55:48 UTC 
Upstream has plans for a new major release 4.0, maybe without support for older python versions.
https://github.com/ninia/jep/issues/202
Comment 9 Raphael Groner 2020-04-04 14:00:28 UTC 
Upstream patch available and promised for upcoming 3.9.1:
https://github.com/ninia/jep/commit/a31d461c6cacc96de68d68320eaa83e19a45d0cc
Comment 10 Miro Hrončok 2020-05-25 13:02:00 UTC 
This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.


The Python 3.9 rebuild is in progress in a Koji side tag.

If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:

    $ fedpkg build --target=f33-python

The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.

You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.

If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/
Comment 11 Petr Viktorin 2020-05-28 16:27:36 UTC 
There is a new failure, see https://koji.fedoraproject.org/koji/taskinfo?taskID=45052014

test_restricted_classloader (test_import.TestImport) ... #
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff26424d40, pid=3660980, tid=0x00007fff06fdf100
#
# JRE version: OpenJDK Runtime Environment (8.0_252-b01) (build 1.8.0_252-ea-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b01 mixed mode linux-ppc64le compressed oops)
# Problematic frame:
# C  [libpython3.9.so.1.0+0x74d40]  PyEval_ReleaseLock+0x18
#
# Core dump written. Default location: /builddir/build/BUILD/jep-3.9.0/core or core.3660980
#
# An error report file with more information is saved as:
# /builddir/build/BUILD/jep-3.9.0/hs_err_pid3660980.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Comment 12 Victor Stinner 2020-05-28 16:53:45 UTC 
The jep crash may be related to https://github.com/ninia/jep/issues/229#issuecomment-633154702
Comment 13 Miro Hrončok 2020-05-29 07:11:53 UTC 
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.
Comment 14 Raphael Groner 2020-05-30 21:17:16 UTC 
*** Bug 1842128 has been marked as a duplicate of this bug. ***
Comment 15 Raphael Groner 2020-06-03 18:00:57 UTC 
There's now an issue with hard linkage to libpython without versioning.

I tend to skip any support for python 3.9 for the moment. Might be of interest when the mass orphaning gets initiated due to FTI.
Comment 16 Raphael Groner 2020-06-06 14:22:28 UTC 
Upstream is going to drop any support for backwards compatibility to python 2, introduced for announced (still in development) 4.0 major release.

https://github.com/ninia/jep/commit/51c2ff3619247ae091eda45eb90e26376d3646cc
Comment 17 Raphael Groner 2020-06-06 15:46:51 UTC 
Fixed in rawhide. Can you please retry?

Added a patch from upstream to avoid daemon thread. But there's now an odd crash with (external) core dump from jdk, I'm adding all to build.log in case. Suspiction is to see some unhandled error/exceptin coming from inside libpython.

https://koji.fedoraproject.org/koji/taskinfo?taskID=45486722

Please reopen if there's still an issue.
Comment 18 Raphael Groner 2020-06-06 15:53:00 UTC 
Relevant snippet from build.log:

Event: 0.309 Thread 0xb6246bf8 Exception <a 'java/security/PrivilegedActionException'> (0xa40fd240) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.252.b02-0.0.ea.fc33.arm/openjdk/hotspot/src/share/vm/prims/jvm.cpp, line 1499]
Event: 1.024 Thread 0xb6246bf8 Exception <a 'jep/JepException'> (0xa40e0d68) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.252.b02-0.0.ea.fc33.arm/openjdk/hotspot/src/share/vm/prims/jni.cpp, line 710]

Maybe that thrown JepException indicates a badly written unit test that fails to handle this invalid error.
Comment 19 Raphael Groner 2020-07-28 06:32:41 UTC 
Reopening for further tracking. Upstream writes:
"We will need to release Jep 3.9.1 as soon as Python 3.9 is released since Python 3.9 will not be compatible with any existing versions of Jep."

New patch: https://github.com/ninia/jep/commit/83b7e654c3efb6b5fc906e877c6b0060eb8b83aa
Comment 20 Igor Raits 2020-08-03 19:31:20 UTC 
Hello,

Please note that this comment was generated automatically. If you feel that this output has mistakes, please contact me via email (ignatenkobrain).

All subpackages of a package against which this bug was filled are now installable or removed from Fedora 33.

Thanks for taking care of it!
Comment 21 Charalampos Stratakis 2020-08-25 12:30:58 UTC 
(In reply to Raphael Groner from comment #19)
> Reopening for further tracking. Upstream writes:
> "We will need to release Jep 3.9.1 as soon as Python 3.9 is released since
> Python 3.9 will not be compatible with any existing versions of Jep."
> 
> New patch:
> https://github.com/ninia/jep/commit/83b7e654c3efb6b5fc906e877c6b0060eb8b83aa

Sent a PR with the patch. In koji rawhide it builds successfully.
Comment 22 Charalampos Stratakis 2020-08-25 12:31:23 UTC 
And of course forgot the link: https://src.fedoraproject.org/rpms/python-jep/pull-request/2
Comment 23 Raphael Groner 2020-09-03 09:58:54 UTC 
Thanks.
Comment 24 Raphael Groner 2020-10-12 00:41:53 UTC 
Is your patch part of the new release 0.9.1 from upstream? See bug #1885098.
Comment 25 Victor Stinner 2020-10-12 08:43:50 UTC 
In jep-3.9.1.tar.gz found at https://pypi.org/project/jep/#files, src/test/java/jep/test/TestSharedModulesThreads.java no longer contains "t.daemon = True". The downstream a31d461c6cacc96de68d68320eaa83e19a45d0cc.patch is no longer needed if the package is updated to jep 3.9.1.
Comment 26 Raphael Groner 2021-02-01 17:44:45 UTC 
Fixed in rawhide. Please retry.
Comment 27 Raphael Groner 2021-02-01 18:52:58 UTC 
Bodhi should add link, but slow performance currently.
Comment 28 Fedora Update System 2021-02-02 02:19:50 UTC 
FEDORA-2021-35ae800ff3 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-35ae800ff3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-35ae800ff3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 29 Fedora Update System 2021-02-10 01:19:20 UTC 
FEDORA-2021-35ae800ff3 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.