Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1898157
Summary: | scipy fails to build with Python 3.10: TypeError: 'float' object cannot be interpreted as an integer | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Miro Hrončok <mhroncok> |
Component: | scipy | Assignee: | Nikola Forró <nforro> |
Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | rawhide | CC: | cstratak, david08741, mhroncok, nforro, python-sig, thrnciar, tomspur, vstinner |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-11-27 08:17:05 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1890881 |
Description
Miro Hrončok
2020-11-16 14:42:24 UTC
I reported the issue to scipy: https://github.com/scipy/scipy/issues/13122 I proposed a fix upstream: https://github.com/scipy/scipy/pull/13121 (In reply to Victor Stinner from comment #1) > I reported the issue to scipy: https://github.com/scipy/scipy/issues/13122 > > I proposed a fix upstream: https://github.com/scipy/scipy/pull/13121 Thanks, but that only solves a subset of the issues, AFAICT. There are also multiple crashes during test runs, similar to this: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fbd50d9337a in vgetargskeywordsfast_impl () from /lib64/libpython3.10.so.1.0 #1 0x00007fbd50e1b60e in _PyArg_ParseStackAndKeywords_SizeT () from /lib64/libpython3.10.so.1.0 #2 0x00007fbd0c0f9f0d in zlib_Decompress_decompress () from /usr/lib64/python3.10/lib-dynload/zlib.cpython-310-x86_64-linux-gnu.so #3 0x00007fbd0af746f4 in __Pyx_PyObject_CallOneArg () from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so #4 0x00007fbd0af77b5e in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream__fill_buffer.part.0 () from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so #5 0x00007fbd0af77de6 in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream_read_into () from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so #6 0x00007fbd0af916aa in __pyx_f_5scipy_2io_6matlab_10mio5_utils_10VarReader5_cread_full_tag.lto_priv.0 () from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so #7 0x00007fbd0af923ec in __pyx_pw_5scipy_2io_6matlab_10mio5_utils_10VarReader5_9read_full_tag.lto_priv.0 () from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so #8 0x00007fbd50d3f0f2 in method_vectorcall_NOARGS () from /lib64/libpython3.10.so.1.0 #9 0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0 #10 0x00007fbd50d3bb83 in function_code_fastcall () from /lib64/libpython3.10.so.1.0 #11 0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0 #12 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0 #13 0x00007fbd50d3b8d6 in _PyFunction_Vectorcall () from /lib64/libpython3.10.so.1.0 #14 0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0 #15 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0 #16 0x00007fbd50d3b8d6 in _PyFunction_Vectorcall () from /lib64/libpython3.10.so.1.0 #17 0x00007fbd50d2e2fb in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0 #18 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0 #19 0x00007fbd50dabba5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.10.so.1.0 #20 0x00007fbd50dabb3d in PyEval_EvalCodeEx () from /lib64/libpython3.10.so.1.0 #21 0x00007fbd50dabaef in PyEval_EvalCode () from /lib64/libpython3.10.so.1.0 #22 0x00007fbd50dd340d in run_eval_code_obj () from /lib64/libpython3.10.so.1.0 #23 0x00007fbd50dd2456 in run_mod () from /lib64/libpython3.10.so.1.0 #24 0x00007fbd50d1b707 in PyRun_InteractiveOneObjectEx () from /lib64/libpython3.10.so.1.0 #25 0x00007fbd50d1bffb in PyRun_InteractiveLoopFlags () from /lib64/libpython3.10.so.1.0 #26 0x00007fbd50cb520c in PyRun_AnyFileExFlags.cold () from /lib64/libpython3.10.so.1.0 #27 0x00007fbd50cb3d42 in Py_RunMain.cold () from /lib64/libpython3.10.so.1.0 #28 0x00007fbd50d9ccfd in Py_BytesMain () from /lib64/libpython3.10.so.1.0 #29 0x00007fbd50a8eba2 in __libc_start_main () from /lib64/libc.so.6 #30 0x000055e51ae4909e in _start () > Thanks, but that only solves a subset of the issues, AFAICT. There are also multiple crashes during test runs, (...)
Oh, I didn't know. Good to know.
Are you interested to investigate, or do you want me to have a look?
My first PR was wrong. I wrote a second PR which has been merged into master: https://github.com/scipy/scipy/commit/277dbab1612ae32f02038a6a0df7e061c927a5cc It simply skips the test on Python 3.10. scipy is going to deprecate and then remove float support in scipy.special.factorial(). (In reply to Victor Stinner from comment #3) > Are you interested to investigate, or do you want me to have a look? Sure, you can have a look, thanks. All the crashes seem to have the same cause, I tracked it down to this so far: https://github.com/scipy/scipy/blob/v1.5.4/scipy/io/matlab/streams.pyx#L137 Here is a simplified reproducer: python3.10 -c 'import scipy.io.matlab.mio; scipy.io.matlab.mio.loadmat("scipy/io/matlab/tests/data/little_endian.mat")' > python3.10 -c 'import scipy.io.matlab.mio; scipy.io.matlab.mio.loadmat("scipy/io/matlab/tests/data/little_endian.mat")'
Thanks for the reproducer.
gdb traceback:
(gdb) where
#0 0x00007ffff7de337a in vgetargskeywordsfast_impl (args=0x1, nargs=0, kwargs=0x0, kwnames=<unknown at remote 0x7ffff583cea4>, parser=0x7ffff5843660 <_parser.19>, p_va=0x7fffffffd220, flags=2)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:2034
#1 0x00007ffff7e6b60e in _PyArg_ParseStackAndKeywords_SizeT (args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>, parser=parser@entry=0x7ffff5843660 <_parser.19>)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:1519
#2 0x00007ffff583cf0d in zlib_Decompress_decompress (self=0x7ffff45e1df0, cls=0x7fffffffd3e8, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Modules/clinic/zlibmodule.c.h:394
#3 0x00007ffff46276f4 in __Pyx_PyObject_CallOneArg ()
from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#4 0x00007ffff462ab5e in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream__fill_buffer.part.0 ()
from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#5 0x00007ffff462ade6 in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream_read_into ()
from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#6 0x00007ffff46446aa in __pyx_f_5scipy_2io_6matlab_10mio5_utils_10VarReader5_cread_full_tag.lto_priv.0 ()
from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#7 0x00007ffff46453ec in __pyx_pw_5scipy_2io_6matlab_10mio5_utils_10VarReader5_9read_full_tag.lto_priv.0 ()
from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#8 0x00007ffff7d8f0f2 in method_vectorcall_NOARGS (func=<method_descriptor at remote 0x7ffff467f630>, args=0x7ffff77513e0, nargsf=<optimized out>, kwnames=0x0)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Objects/descrobject.c:434
#9 0x00007ffff7d7e559 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff77513e0, callable=<method_descriptor at remote 0x7ffff467f630>, tstate=0x555555560cd0)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/cpython/abstract.h:114
#10 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff77513e0, callable=<method_descriptor at remote 0x7ffff467f630>)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/cpython/abstract.h:123
#11 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555560cd0) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/ceval.c:5341
#12 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/ceval.c:3753
#13 0x00007ffff7d8bb83 in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7ffff7751240, for file /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py, line 267, in read_var_header (self=<MatFile5Reader(mat_stream=<_io.BufferedReader at remo
te 0x7ffff77ab250>, dtypes={}, byte_order='<', struct_as_record=True, squeeze_me=False, chars_as_strings=True, mat_dtype=False, verify_compressed_data_integrity=True, simplify_cells=False, uint16_codec='utf-8', _file_reader=<scipy.io.matl
ab.mio5_utils.VarReader5 at remote 0x7ffff4f63a00>, _matrix_reader=<scipy.io.matlab.mio5_utils.VarReader5 at remote--Type <RET> for more, q to quit, c to continue without paging--
0x7ffff4c15040>) at remote 0x7ffff781f730>, mdtype=15, byte_count=48, next_pos=184, stream=<scipy.io.matlab.streams.ZlibInputStream at remote 0x7ffff45daf20>, check_stream_limit=True),
tstate=0x555555560cd0) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/internal/pycore_ceval.h:40
(...)
(gdb) py-bt
Traceback (most recent call first):
File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py", line 267, in read_var_header
mdtype, byte_count = self._matrix_reader.read_full_tag()
File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py", line 313, in get_variables
hdr, next_position = self.read_var_header()
File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio.py", line 224, in loadmat
matfile_dict = MR.get_variables(variable_names)
File "<string>", line 1, in <module>
(gdb) frame 0
#0 0x00007ffff7de337a in vgetargskeywordsfast_impl (args=0x1, nargs=0, kwargs=0x0, kwnames=<unknown at remote 0x7ffff583cea4>, parser=0x7ffff5843660 <_parser.19>, p_va=0x7fffffffd220, flags=2)
at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:2034
2034 if (kwnames != NULL && !PyTuple_Check(kwnames)) {
(gdb) p *kwnames
$3 = {ob_refcnt = -8554210076332781581, ob_type = 0x41ff3145c2894cd2}
The zlib_Decompress_decompress() function is called with the wrong calling convention. It gets a kwnames object which is not a tuple of strings but looks like an invalid PyObject*.
In Python 3.10, this function is declared as:
#define ZLIB_DECOMPRESS_DECOMPRESS_METHODDEF \
{"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__},
static PyObject *
zlib_Decompress_decompress(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
{
PyObject *return_value = NULL;
static const char * const _keywords[] = {"", "max_length", NULL};
static _PyArg_Parser _parser = {"y*|n:decompress", _keywords, 0};
Py_buffer data = {NULL, NULL};
Py_ssize_t max_length = 0;
if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
&data, &max_length)) {
goto exit;
}
return_value = zlib_Decompress_decompress_impl(self, cls, &data, max_length);
exit:
/* Cleanup for data */
if (data.obj) {
PyBuffer_Release(&data);
}
return return_value;
}
My bet is an issue in Cython optimized functions which doesn't support properly METH_METHOD. The surprising part is that METH_METHOD was introduced in Python 3.9, not in Python 3.10.
The difference in Python 3.9, the function didn't use METH_METHOD:
#define ZLIB_DECOMPRESS_DECOMPRESS_METHODDEF \
{"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__},
static PyObject *
zlib_Decompress_decompress(compobject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
{
...
}
About Python 3.10, the following calling convention: {"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__}, comes from this Argument Clinic code: /*[clinic input] zlib.Decompress.decompress cls: defining_class data: Py_buffer The binary data to decompress. / max_length: Py_ssize_t = 0 The maximum allowable length of the decompressed data. Unconsumed input data will be stored in the unconsumed_tail attribute. ... [clinic start generated code]*/ The zlib module started to use "defining_class" argument type. > My bet is an issue in Cython optimized functions which doesn't support properly METH_METHOD. I reported the issue to Cython: https://github.com/cython/cython/issues/3917 scipy only works on Python 3.9 because zlib didn't use METH_METHOD flag. It started to use it in Python 3.10. Thanks Victor. I added your patch to skip the factorial test in Rawhide: https://src.fedoraproject.org/rpms/scipy/c/138e3931a60cebfa5c0d3163f9da90e8bf705ced?branch=master With this: https://src.fedoraproject.org/rpms/Cython/pull-request/24 scipy builds! Thank you all. |