Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1965584
Summary: | gevent crash on Python 3.10 (python-x2go fails to build): ABI issue with outdated vendored copy of greenlet.h | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Tomáš Hrnčiar <thrnciar> |
Component: | python3.10 | Assignee: | Python Maintainers <python-maint> |
Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | rawhide | CC: | cstratak, mhroncok, orion, python-maint, python-sig, thrnciar, torsava, vstinner |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-06-02 10:26:41 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1890881 |
Description
Tomáš Hrnčiar
2021-05-28 08:06:30 UTC
This seems to be python or possibly python-gevent crashing when running sphinx-build-3. Running with gdb just results in a corrupted stack trace. valgrind reports: [mockbuild@d4b708504a514540a8c1ae53566cb8a6 ~]$ cd build/BUILD/python-x2go-0.6.1.3/docs [mockbuild@d4b708504a514540a8c1ae53566cb8a6 docs]$ valgrind /usr/bin/sphinx-build-3 -b html -d build/doctrees source build/html ==35== Memcheck, a memory error detector ==35== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==35== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info ==35== Command: /usr/bin/sphinx-build-3 -b html -d build/doctrees source build/html ==35== Running Sphinx v3.5.4 ==84== Warning: invalid file descriptor 1024 in syscall close() ==84== Warning: invalid file descriptor 1025 in syscall close() ==84== Warning: invalid file descriptor 1026 in syscall close() ==84== Warning: invalid file descriptor 1027 in syscall close() ==84== Use --log-fd=<number> to select an alternative log fd. ==84== Warning: invalid file descriptor 1028 in syscall close() ==84== Warning: invalid file descriptor 1029 in syscall close() <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject loading translations [en]... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 46 source files that are out of date updating environment: [new config] 46 added, 0 changed, 0 removed reading sources... [ 2%] index reading sources... [ 4%] modules reading sources... [ 6%] x2go reading sources... [ 8%] x2go.backends reading sources... [ 10%] x2go.backends.control reading sources... [ 13%] x2go.backends.control.plain reading sources... [ 15%] x2go.backends.info reading sources... [ 17%] x2go.backends.info.plain reading sources... [ 19%] x2go.backends.printing reading sources... [ 21%] x2go.backends.printing.file reading sources... [ 23%] x2go.backends.profiles reading sources... [ 26%] x2go.backends.profiles.base reading sources... [ 28%] x2go.backends.profiles.file reading sources... [ 30%] x2go.backends.profiles.httpbroker reading sources... [ 32%] x2go.backends.profiles.sshbroker reading sources... [ 34%] x2go.backends.proxy reading sources... [ 36%] x2go.backends.proxy.base reading sources... [ 39%] x2go.backends.proxy.nx3 reading sources... [ 41%] x2go.backends.settings reading sources... [ 43%] x2go.backends.settings.file reading sources... [ 45%] x2go.backends.terminal reading sources... [ 47%] x2go.backends.terminal.plain ==91== Warning: invalid file descriptor 1024 in syscall close() ==91== Warning: invalid file descriptor 1025 in syscall close() ==91== Warning: invalid file descriptor 1026 in syscall close() ==91== Warning: invalid file descriptor 1027 in syscall close() ==91== Use --log-fd=<number> to select an alternative log fd. ==91== Warning: invalid file descriptor 1028 in syscall close() ==91== Warning: invalid file descriptor 1029 in syscall close() ==91== ==91== HEAP SUMMARY: ==91== in use at exit: 56,452,639 bytes in 499,640 blocks ==91== total heap usage: 7,604,380 allocs, 7,104,740 frees, 991,297,424 bytes allocated ==91== ==35== Invalid read of size 8 ==35== at 0xA87926F: __pyx_f_6gevent_24_gevent_c_hub_primitives__primitive_wait.lto_priv.0 (_hub_primitives.c:6044) ==35== by 0xA87AE57: __pyx_f_6gevent_24_gevent_c_hub_primitives_wait_on_watcher.lto_priv.0 (_hub_primitives.c:6661) ==35== by 0xA87AFA8: UnknownInlinedFun (_hub_primitives.c:6801) ==35== by 0xA87AFA8: __pyx_pw_6gevent_24_gevent_c_hub_primitives_9wait_on_watcher.lto_priv.0 (_hub_primitives.c:6776) ==35== by 0x49627F3: _PyObject_MakeTpCall (call.c:215) ==35== by 0x495ED5D: UnknownInlinedFun (abstract.h:112) ==35== by 0x495ED5D: UnknownInlinedFun (abstract.h:99) ==35== by 0x495ED5D: UnknownInlinedFun (abstract.h:123) ==35== by 0x495ED5D: UnknownInlinedFun (ceval.c:5885) ==35== by 0x495ED5D: _PyEval_EvalFrameDefault (ceval.c:4214) ==35== by 0x4958943: UnknownInlinedFun (pycore_ceval.h:46) ==35== by 0x4958943: _PyEval_Vector (ceval.c:5069) ==35== by 0x4959D0D: UnknownInlinedFun (abstract.h:114) ==35== by 0x4959D0D: UnknownInlinedFun (abstract.h:123) ==35== by 0x4959D0D: UnknownInlinedFun (ceval.c:5885) ==35== by 0x4959D0D: _PyEval_EvalFrameDefault (ceval.c:4199) ==35== by 0x4958943: UnknownInlinedFun (pycore_ceval.h:46) ==35== by 0x4958943: _PyEval_Vector (ceval.c:5069) ==35== by 0x496F192: UnknownInlinedFun (call.c:342) ==35== by 0x496F192: UnknownInlinedFun (abstract.h:114) ==35== by 0x496F192: method_vectorcall (classobject.c:61) ==35== by 0x49C410B: UnknownInlinedFun (abstract.h:114) ==35== by 0x49C410B: _PyObject_CallNoArg.lto_priv.0 (abstract.h:168) ==35== by 0x49E10BF: UnknownInlinedFun (bufferedio.c:1551) ==35== by 0x49E10BF: UnknownInlinedFun (bufferedio.c:888) ==35== by 0x49E10BF: _io__Buffered_read (bufferedio.c.h:169) ==35== by 0x496728F: cfunction_vectorcall_FASTCALL (methodobject.c:426) ==35== Address 0x1ffeff8298 is on thread 1's stack ==35== 1656 bytes below stack pointer ==35== ==35== Invalid write of size 8 ==35== at 0x4948D59: _PyObject_GC_Alloc (gcmodule.c:2257) ==35== Address 0x1ffeff8288 is on thread 1's stack ==35== 1664 bytes below stack pointer ==35== ==35== Use of uninitialised value of size 8 ==35== at 0x4948D71: _PyObject_GC_Alloc (gcmodule.c:2272) ==35== by 0xA70FB2F: ??? ==35== ==35== Jump to the invalid address stated on the next line ==35== at 0x0: ??? ==35== by 0xA70FB2F: ??? ==35== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==35== ==35== ==35== Process terminating with default action of signal 11 (SIGSEGV) ==35== Bad permissions for mapped region at address 0x0 ==35== at 0x0: ??? ==35== by 0xA70FB2F: ??? reading sources... [ 47%] x2go.backends.terminal.plain make: *** [Makefile:53: html] Segmentation fault (core dumped) It would be nice to retrieve the stack trace here. ==35== Invalid read of size 8 ==35== at 0xA87926F: __pyx_f_6gevent_24_gevent_c_hub_primitives__primitive_wait.lto_priv.0 (_hub_primitives.c:6044) ==35== by 0xA87AE57: __pyx_f_6gevent_24_gevent_c_hub_primitives_wait_on_watcher.lto_priv.0 (_hub_primitives.c:6661) ==35== by 0xA87AFA8: UnknownInlinedFun (_hub_primitives.c:6801) ==35== by 0xA87AFA8: __pyx_pw_6gevent_24_gevent_c_hub_primitives_9wait_on_watcher.lto_priv.0 (_hub_primitives.c:6776) These 4 functions come from the following file of gevent compiled with Cython: https://github.com/gevent/gevent/blob/master/src/gevent/_hub_primitives.py The crash comes from the gevent code compiled with Cython. Reproducer code: --- # Disable C accelerator import gevent._compat #gevent._compat.PURE_PYTHON = True from gevent import _greenlet_primitives # Explicitly import the Cython code #from gevent import _gevent_c_greenlet_primitives as _greenlet_primitives print(_greenlet_primitives) SwitchOutGreenletWithLoop = _greenlet_primitives.SwitchOutGreenletWithLoop class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable ... --- The pure Python code of gevent works well: uncomment "PURE_PYTHON = True" line, it no longer crashes. When using Python 3.10 built in debug mode, I get an assertion error at gevent/_hub_primitives.py, line 44: "class WaitOperationsGreenlet(SwitchOutGreenletWithLoop):" gevent simplified code: --- from gevent import _greenlet_primitives # In Cython, we define these as 'cdef inline' functions. The # compilation unit cannot have a direct assignment to them (import # is assignment) without generating a 'lvalue is not valid target' # error. locals()['SwitchOutGreenletWithLoop'] = _greenlet_primitives.SwitchOutGreenletWithLoop class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable ... --- _greenlet_primitives.py simplified code: --- import greenlet class TrackedRawGreenlet(greenlet.greenlet): ... class SwitchOutGreenletWithLoop(TrackedRawGreenlet): ... --- Assertion error: python: Objects/typeobject.c :2219 : extra_ivars: l'assertion « t_size >= b_size » a échoué. Program received signal SIGABRT, Aborted. 0x00007ffff7c662a2 in raise () from /lib64/libc.so.6 (gdb) frame 4 #4 0x0000000000495baa in extra_ivars (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>, base=0x7fffe9d01620 <PyGreenlet_Type>) at Objects/typeobject.c:2219 2219 assert(t_size >= b_size); /* Else type smaller than base! */ (gdb) p t_size $1 = 152 (gdb) p b_size $2 = 160 (gdb) l 2214 extra_ivars(PyTypeObject *type, PyTypeObject *base) 2215 { 2216 size_t t_size = type->tp_basicsize; 2217 size_t b_size = base->tp_basicsize; 2218 2219 assert(t_size >= b_size); /* Else type smaller than base! */ 2220 if (type->tp_itemsize || base->tp_itemsize) { 2221 /* If itemsize is involved, stricter rules */ 2222 return t_size != b_size || 2223 type->tp_itemsize != base->tp_itemsize; (gdb) p base->tp_name $3 = 0x7fffe9cfea2c "greenlet.greenlet" (gdb) p type->tp_name $4 = 0x7fffe78b8938 "gevent._gevent_c_greenlet_primitives.TrackedRawGreenlet" (gdb) where #0 0x00007ffff7c662a2 in raise () from /lib64/libc.so.6 #1 0x00007ffff7c4f8a4 in abort () from /lib64/libc.so.6 #2 0x00007ffff7c4f789 in __assert_fail_base.cold () from /lib64/libc.so.6 #3 0x00007ffff7c5ea16 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000495baa in extra_ivars (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>, base=0x7fffe9d01620 <PyGreenlet_Type>) at Objects/typeobject.c:2219 #5 0x0000000000495cf5 in solid_base (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>) at Objects/typeobject.c:2246 #6 0x0000000000495cd4 in solid_base (type=0x7fffe78bd600 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_SwitchOutGreenletWithLoop>) at Objects/typeobject.c:2243 #7 0x0000000000495aae in best_base (bases=(<type at remote 0x7fffe78bd600>,)) at Objects/typeobject.c:2189 #8 0x0000000000497fb5 in type_new_get_bases (ctx=0x7ffffffde1a0, type=0x7ffffffde198) at Objects/typeobject.c:3269 #9 0x0000000000498168 in type_new (metatype=0x893bc0 <PyType_Type>, args=('WaitOperationsGreenlet', (<type at remote 0x7fffe78bd600>,), {'__module__': 'gevent._hub_primitives', '__qualname__': 'WaitOperationsGreenlet', 'wait': <function at remote 0x7fffe790b280>, 'cancel_waits_close_and_then': <function at remote 0x7fffe790b330>, '_cancel_waits_then': <function at remote 0x7fffe790b3e0>, 'cancel_wait': <function at remote 0x7fffe790b490>, '_cancel_wait': <function at remote 0x7fffe790b540>}), kwds=0x0) at Objects/typeobject.c:3311 #10 0x0000000000493518 in type_call (type=0x893bc0 <PyType_Type>, args=('WaitOperationsGreenlet', (<type at remote 0x7fffe78bd600>,), {'__module__': 'gevent._hub_primitives', '__qualname__': 'WaitOperationsGreenlet', 'wait': <function at remote 0x7fffe790b280>, 'cancel_waits_close_and_then': <function at remote 0x7fffe790b330>, '_cancel_waits_then': <function at remote 0x7fffe790b3e0>, 'cancel_wait': <function at remote 0x7fffe790b490>, '_cancel_wait': <function at remote 0x7fffe790b540>}), kwds=0x0) at Objects/typeobject.c:1127 ... (gdb) py-bt Traceback (most recent call first): <built-in method __build_class__ of module object at remote 0x7fffea697c50> File "/home/vstinner/bug/python-x2go-0.6.1.3/env/lib/python3.10/site-packages/gevent-21.1.2-py3.10-linux-x86_64.egg/gevent/_hub_primitives.py", line 44, in <module> class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable <built-in method exec of module object at remote 0x7fffea697c50> File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "/home/vstinner/bug/python-x2go-0.6.1.3/env/lib/python3.10/site-packages/gevent-21.1.2-py3.10-linux-x86_64.egg/gevent/__init__.py", line 87, in <module> from gevent._hub_primitives import iwait_on_objects as iwait (...) Before the crash, I see warnings like that: <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject This warning comes from Cython __Pyx_ImportType() function. Oh, at May 6, 2021, binary wheel files of greenlet for Python 3.10 have been published, even if Python 3.10 is still a beta version: https://pypi.org/project/greenlet/#files I simplified src/gevent/_greenlet_primitives.py to: --- import greenlet._greenlet class TrackedRawGreenlet(greenlet._greenlet.greenlet): ... class SwitchOutGreenletWithLoop(TrackedRawGreenlet): ... --- where greenlet._greenlet is a C extension. Script to reproduce the crash, bug.py: --- from gevent._gevent_c_greenlet_primitives import SwitchOutGreenletWithLoop class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): ... --- Commands to reproduce the issue on Fedora 34: --- mkdir bug; cd bug # Development branch of Python 3.10 built in debug mode ~/python/3.10/python -m venv ../env # Install Cython 0.29.23 env/bin/python -m pip install Cython wget https://files.pythonhosted.org/packages/47/6d/be10df2b141fcb1020c9605f7758881b5af706fb09a05b737e8eb7540387/greenlet-1.1.0.tar.gz tar -xf greenlet-1.1.0.tar.gz && cd greenlet-1.1.0/ ../env/bin/python setup.py install cd .. wget https://files.pythonhosted.org/packages/0b/50/1b1175ea3a269b5fa3f0f7fed11149888180695bef5fbf568adbb196efaf/gevent-21.1.2.tar.gz tar -xf gevent-21.1.2.tar.gz && cd gevent-21.1.2/ ../env/bin/python setup.py install cd .. ./env/bin/python -c 'import gevent' --- where ~/python/3.10/ is a checkout of the 3.10 branch of git:python/cpython.git and the ~/python/3.10/python was built with "./configure --with-pydebug CFLAGS=-O0 --with-system-expat --with-system-ffi && make". Output: --- <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject <frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject python: Objects/typeobject.c:2219: extra_ivars: Assertion `t_size >= b_size' failed. Abandon (core dumped) --- Oh, gevent 21.1.2 contains an outdated copy of greenlet.h in deps/greenlet/greenlet.h, the old copy is ABI incompatible with greenlet 1.1.0. greenlet 1.1.0 contains a fix for Python 3.10: https://greenlet.readthedocs.io/en/latest/changes.html#id1 gevent copy doesn't the greenlet fix for Python 3.10. Example of __Pyx_ImportType() call in gevent-21.1.2/src/gevent/_hub_local.c, code run when the module is imported (Py_mod_exec slot of the module): --- #include "greenlet/greenlet.h" ... __pyx_ptype_6gevent_29_gevent_c_greenlet_primitives_greenlet = __Pyx_ImportType(__pyx_t_1, "greenlet", "greenlet", sizeof(PyGreenlet), __Pyx_ImportType_CheckSize_Warn); --- In env/lib/python3.10/site-packages/greenlet-1.1.0-py3.10-linux-x86_64.egg/greenlet/greenlet.h, PyGreenlet structure is defined as: typedef struct _greenlet { PyObject_HEAD char* stack_start; char* stack_stop; char* stack_copy; intptr_t stack_saved; struct _greenlet* stack_prev; struct _greenlet* parent; PyObject* run_info; struct _frame* top_frame; int recursion_depth; PyObject* weakreflist; #if PY_VERSION_HEX >= 0x030700A3 _PyErr_StackItem* exc_info; _PyErr_StackItem exc_state; #else PyObject* exc_type; PyObject* exc_value; PyObject* exc_traceback; #endif PyObject* dict; #if PY_VERSION_HEX >= 0x030700A3 PyObject* context; #endif #if PY_VERSION_HEX >= 0x30A00B1 CFrame* cframe; #endif } PyGreenlet; Cython compiles src/gevent/_hub_local.py to src/gevent/_hub_local.c. GCC compiles src/gevent/_hub_local.c to src/gevent/_hub_local.o using -I/home/vstinner/bug/gevent-21.1.2/deps option. Problem: gevent-21.1.2/deps/ contains a copy of greenlet/greenlet.h which is outdated: --- typedef struct _greenlet { PyObject_HEAD char* stack_start; char* stack_stop; char* stack_copy; intptr_t stack_saved; struct _greenlet* stack_prev; struct _greenlet* parent; PyObject* run_info; struct _frame* top_frame; int recursion_depth; PyObject* weakreflist; #if PY_VERSION_HEX >= 0x030700A3 _PyErr_StackItem* exc_info; _PyErr_StackItem exc_state; #else PyObject* exc_type; PyObject* exc_value; PyObject* exc_traceback; #endif PyObject* dict; #if PY_VERSION_HEX >= 0x030700A3 PyObject* context; #endif } PyGreenlet; --- It doesn't contain the last cframe member of Python 3.10 beta1 and newer. > Oh, gevent 21.1.2 contains an outdated copy of greenlet.h in deps/greenlet/greenlet.h, the old copy is ABI incompatible with greenlet 1.1.0.
Maybe the gevent specfile should remove its vendored copy to get greenlet.h from a build dependency, to avoid such ABI issue.
> Maybe the gevent specfile should remove its vendored copy to get greenlet.h from a build dependency, to avoid such ABI issue. Oh, ignore my previous comments, the Fedora python-gevent package is not affected. It already removes deps/ directory in its specfile: https://src.fedoraproject.org/rpms/python-gevent/blob/rawhide/f/python-gevent.spec --- %prep %autosetup -p1 -n %{modname}-%{version} # Remove bundled libraries rm -r deps --- I've deleted the build with Python 3.10.0b1 from https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-gevent/ and verified the only latest successful build was done with 3.10.0b2. I also see a successful build of https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-x2go/ This seems like a simple matter of misscompilation with an older Python 3.10 pre-release. Stuff like that happens in our Copr and we have no easy way to solve this, except to keep rebuilding everything until it does. Sorry for the noise. |