Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1737011 - python-theano: FTBFS in Fedora rawhide/f31
Summary: python-theano: FTBFS in Fedora rawhide/f31
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-theano
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Jerry James
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F31FTBFS PYTHON38 1732841
TreeView+ depends on / blocked
 
Reported: 2019-08-02 09:50 UTC by Fedora Release Engineering
Modified: 2019-09-16 08:52 UTC (History)
4 users (show)

Fixed In Version: python-theano-1.0.4-4.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-16 08:52:12 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
build.log (32.00 KB, text/plain)
2019-08-02 09:50 UTC, Fedora Release Engineering
no flags Details
root.log (32.00 KB, text/plain)
2019-08-02 09:50 UTC, Fedora Release Engineering
no flags Details
state.log (653 bytes, text/plain)
2019-08-02 09:50 UTC, Fedora Release Engineering
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github Theano Theano issues 6715 0 None open Numpy 1.17.0 issues 2020-02-11 04:41:30 UTC

Description Fedora Release Engineering 2019-08-02 09:50:38 UTC
python-theano failed to build from source in Fedora rawhide/f31

https://koji.fedoraproject.org/koji/taskinfo?taskID=36749963


For details on the mass rebuild see:

https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild
Please fix python-theano at your earliest convenience and set the bug's status to
ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks,
python-theano will be orphaned. Before branching of Fedora 32,
python-theano will be retired, if it still fails to build.

For more details on the FTBFS policy, please visit:
https://fedoraproject.org/wiki/Fails_to_build_from_source

Comment 1 Fedora Release Engineering 2019-08-02 09:50:41 UTC
Created attachment 1599981 [details]
build.log

file build.log too big, will only attach last 32768 bytes

Comment 2 Fedora Release Engineering 2019-08-02 09:50:43 UTC
Created attachment 1599982 [details]
root.log

file root.log too big, will only attach last 32768 bytes

Comment 3 Fedora Release Engineering 2019-08-02 09:50:45 UTC
Created attachment 1599983 [details]
state.log

Comment 4 Miro Hrončok 2019-08-05 08:28:11 UTC
This blocks the Python 3.8 rebuild.

Comment 5 Jerry James 2019-08-05 20:45:35 UTC
The numpy 1.17.0 release is involved somehow.  If I build with numpy 1.16.4, all is well.  With 1.17.0, there are errors and test failures:

======================================================================
ERROR: theano.tensor.tests.test_var.test_numpy_method
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/tensor/tests/test_var.py", line 23, in test_numpy_method
    y = fct(x)
TypeError: must be real number, not TensorVariable

======================================================================
FAIL: test_scan_err1 (theano.gof.tests.test_compute_test_value.TestComputeTestValue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/gof/tests/test_compute_test_value.py", line 283, in test_scan_err1
    n_steps=k)
ValueError: shapes (5,3) and (5,3) not aligned: 3 (dim 1) != 5 (dim 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/gof/tests/test_compute_test_value.py", line 292, in test_scan_err1
    assert os.path.split(frame_info[0])[1] == expected, frame_info
AssertionError: <FrameSummary file /builddir/build/BUILD/Theano-rel-1.0.4/theano/tensor/basic.py, line 6105 in dot>

======================================================================
FAIL: test_good (theano.tensor.tests.test_basic.ClipTester)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/tensor/tests/test_basic.py", line 425, in test_good
    np.allclose(variable, expected)))
AssertionError: Test Elemwise{clip,no_inplace}::correct7: Output 0 gave the wrong value. With inputs [array([[-1.70586872,  2.46664369, -0.52189608,  4.84714482,  1.9208813 ],
       [ 1.28983643, -2.38914901, -1.67933224,  2.10808673,  3.58489085],
       [ 4.45900732, -3.70413992, -3.7683781 ,  1.87495988, -2.58993616],
       [-2.6853165 ,  2.19925062,  2.69281351,  1.39211517,  1.29298677],
       [ 4.43245431,  3.63622689, -2.58812035, -3.32239248,  0.24149971]]), array(1.), array(-1.)], expected [[-1. -1. -1. -1. -1.]
 [-1. -1. -1. -1. -1.]
 [-1. -1. -1. -1. -1.]
 [-1. -1. -1. -1. -1.]
 [-1. -1. -1. -1. -1.]] (dtype float64), got [[ 1. -1.  1. -1. -1.]
 [-1.  1.  1. -1. -1.]
 [-1.  1.  1. -1.  1.]
 [ 1. -1. -1. -1. -1.]
 [-1. -1.  1.  1.  1.]] (dtype float64). eps=0.000000 np.allclose returns False False

======================================================================
FAIL: test_vector_arguments (theano.tensor.tests.test_raw_random.T_random_function)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/tensor/tests/test_raw_random.py", line 681, in test_vector_arguments
    self.assertRaises(ValueError, fc, post1c, [-4., -2], [-1, 0], [1])
AssertionError: ValueError not raised by <theano.compile.function_module.Function object at 0x7fcab1fdf890>

======================================================================
FAIL: test_vector_arguments (theano.tensor.tests.test_shared_randomstreams.T_SharedRandomStreams)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Theano-rel-1.0.4/theano/tensor/tests/test_shared_randomstreams.py", line 469, in test_vector_arguments
    self.assertRaises(ValueError, fc, [-4., -2], [-1, 0], [1])
AssertionError: ValueError not raised by <theano.compile.function_module.Function object at 0x7fcab178de90>

----------------------------------------------------------------------

I have only a shallow understanding of how numpy works.  If there are any numpy experts in the audience, advice on fixing these would be much appreciated.

Comment 6 Miro Hrončok 2019-08-05 22:03:00 UTC
I suggest to file an upstream issue.

Comment 7 Ben Cotton 2019-08-13 17:12:43 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 8 Ben Cotton 2019-08-13 18:32:24 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 9 Miro Hrončok 2019-08-14 22:24:42 UTC
The coordinated rebuild of Python 3.8 has started in the `f32-python` side tag.

If you figure out how to rebuild this package, please don't rebuild it in regular rawhide, but use the side tag instead:

    on branch master:
    $ fedpkg build --target=f32-python

To wait for a build to show up in the side tag, do:

    $ koji wait-repo f32-python --build=<nvr>

Where <nvr> is name-version-release of the source package, e.g. python-foo-1.1-2.fc32.

An updated mock config is posted at:
http://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Note that it will take a while before the essential packages are rebuilt, so don't expect all your dependencies to be available right away.

Thanks. Let us know if you need up to date info, or if you have any questions.



PS this message is mass posted to all the bugs that block the PYTHON38 bug. If this is also a Fedora 31 FTBFS bug and you manage to fix it, you can do a f31 build as usual:

    on branch f31:
    $ fedpkg build

Comment 10 Jerry James 2019-08-16 19:33:26 UTC
I have patched python-theano and built it in Rawhide (after checking with Miro) and F31.  I will leave this open for now to see if upstream responds with different approaches to fixing the numpy 1.17.0 incompatibilities.

Comment 11 Miro Hrončok 2019-08-21 16:35:27 UTC
The f32-python side tag has been merged. In order to rebuild the package, do it in regular rawhide, but please wait until python3-3.8 is tagged:

  $ koji wait-repo f32-build --build python3-3.8.0~b3-3.fc32


If your built already started in f32-python, after it is finished, please tag it to rawhide with:

  $ koji tag-build f32-pending <nvr>

For example:

  $ koji tag-build f32-pending libreoffice-6.3.0.4-3.fc32

Thanks!

(This comment is mass posted to all bugzillas blocking the PYTHON38 tracking bug.)

Comment 12 Miro Hrončok 2019-08-21 17:29:27 UTC
(Python 3.8 has landed in the rawhide buildroot.)

Comment 13 Robert-André Mauchin 🐧 2019-08-22 19:07:07 UTC
(In reply to Jerry James from comment #10)
> I have patched python-theano and built it in Rawhide (after checking with
> Miro) and F31.  I will leave this open for now to see if upstream responds
> with different approaches to fixing the numpy 1.17.0 incompatibilities.

This doesn't seem to build with Py 3.8 in Rawhide? Anything else must be done?

Comment 14 Miro Hrončok 2019-08-22 22:36:09 UTC
This hasn't been rebuild with Python 3.8 and it blocks quite a stack:

$ repoquery --repo=rawhide{,-source} --whatrequires python3-theano
python3-lasagne-0:0.1-13.fc31.noarch
sympy-0:1.4-2.fc31.src

$ repoquery --repo=rawhide{,-source} --whatrequires python3-sympy
octave-symbolic-0:2.8.0-3.fc31.noarch
octave-symbolic-0:2.8.0-3.fc31.src
python-brian2-0:2.2.2.1-2.fc31.src
python-nineml-0:1.0.1-2.fc31.src
python-nipy-0:0.4.2-3.fc31.src
python-texext-0:0.6.1-4.fc32.src
python-trimesh-0:2.37.12-2.fc31.src
python3-brian2-0:2.2.2.1-2.fc31.x86_64
python3-fiat-0:2018.1.0-3.fc31.noarch
python3-nineml-0:1.0.1-2.fc31.noarch
python3-nipy-0:0.4.2-3.fc31.x86_64
python3-pysb-0:1.9.1-2.fc31.noarch
python3-trimesh-easy-0:2.37.12-2.fc31.noarch
sagemath-0:8.8-1.fc31.src
sagemath-0:8.8-1.fc31.x86_64
sympy-examples-0:1.4-2.fc31.noarch

$ repoquery --repo=rawhide{,-source} --whatrequires python3-texext
python-nb2plots-0:0.6-5.20190809.dfa3ad2.fc32.src
python-networkx-0:2.3-3.fc31.src
python3-nb2plots-0:0.6-5.20190809.dfa3ad2.fc32.noarch

$ repoquery --repo=rawhide{,-source} --whatrequires python3-networkx
cjdns-graph-0:20.3-6.fc31.noarch
diskimage-builder-0:2.20.3-3.fc31.noarch
python-chaospy-0:3.0.7-1.fc32.src
python-libpysal-0:4.1.0-2.fc31.src
python-nitime-0:0.8.1-1.fc31.src
python-scikit-image-0:0.14.2-2.fc31.src
python-taskflow-0:3.7.0-2.fc31.src
python-trimesh-0:2.37.12-2.fc31.src
python-vitrageclient-0:2.7.0-2.fc31.src
python3-biopython-0:1.74-1.fc32.x86_64
python3-chaospy-0:3.0.7-1.fc32.noarch
python3-nitime-0:0.8.1-1.fc31.x86_64
python3-prov-0:1.5.2-5.fc31.noarch
python3-pysb-0:1.9.1-2.fc31.noarch
python3-scikit-image-0:0.14.2-2.fc31.x86_64
python3-taskflow-0:3.7.0-2.fc31.noarch
python3-trimesh-0:2.37.12-2.fc31.noarch
python3-tvb-data-0:1.5.6-6.20191229git7d2d05b.fc31.noarch
python3-vitrageclient-0:2.7.0-2.fc31.noarch
pythran-0:0.9.2-2.fc31.src
pythran-0:0.9.2-2.fc31.x86_64
sagemath-0:8.8-1.fc31.src
sagemath-0:8.8-1.fc31.x86_64
setools-console-analyses-0:4.2.2-1.fc31.x86_64
setools-gui-0:4.2.2-1.fc31.x86_64

... and so on

Comment 15 Robert-André Mauchin 🐧 2019-08-23 12:03:28 UTC
Patching the bug reveals another set of failures afterwards
https://koji.fedoraproject.org/koji/taskinfo?taskID=37227614

Comment 16 Jerry James 2019-08-23 14:06:34 UTC
Robert-André, would you mind sharing your initial patch?  I have patched several other problems in Theano, and have time to take a look at this latest problem, but I don't know what you've already done.

Comment 17 Jerry James 2019-08-23 19:11:55 UTC
I continue to confess that I know little about numpy.  But here is the crux of the problem, a difference in how numpy behaves with python 3.7 and 3.8.  With python3-3.7.4-5.fc31.x86_64 and python3-numpy-1.17.0-3.fc32.x86_64 from the last Rawhide compose:

>>> import numpy as np
>>> a = np.array([[0.70043712, 0.84418664], [0.67651434, 0.72785806], [0.95145796, 0.0127032 ]])
>>> axis = np.array(0.)
>>> np.sort(a, axis, 'quicksort')
array([[0.67651434, 0.0127032 ],
       [0.70043712, 0.72785806],
       [0.95145796, 0.84418664]])

With python3-3.8.0~b3-4.fc32.x86_64 and python3-numpy-1.17.0-3.fc32.x86_64 from the latest koji builds:

>>> import numpy as np
>>> a = np.array([[0.70043712, 0.84418664], [0.67651434, 0.72785806], [0.95145796, 0.0127032 ]])
>>> axis = np.array(0.)
>>> np.sort(a, axis, 'quicksort')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<__array_function__ internals>", line 5, in sort
  File "/usr/lib64/python3.8/site-packages/numpy/core/fromnumeric.py", line 970, in sort
    a.sort(axis=axis, kind=kind, order=order)
TypeError: only integer scalar arrays can be converted to a scalar index

So floating point scalar arrays used to be convertible to scalar indexes with python 3.7, but are not so convertible with python 3.8.  I do not know whether they *should* be convertible.  Perhaps the 3.7 behavior was always wrong, but python let you get away with it, and the thumbscrews have been tightened with 3.8.  In case it is the latter, I will try to find where Theano wraps the integer indices in a numpy array, as it needs to specify an integer array instead of a floating point array at that point, it seems.

Comment 18 Jerry James 2019-08-23 19:20:05 UTC
Okay, I see it.  This line in test_sort.py, in test2:

axis = tensor.scalar()

creates a floating point scalar, but we want an integer scalar there.  I'm going to change that to:

axis = tensor.scalar(dtype='int32')

and try again.  Robert-André, is that what you did?

Comment 19 Robert-André Mauchin 🐧 2019-08-24 08:22:58 UTC
(In reply to Jerry James from comment #18)
> Okay, I see it.  This line in test_sort.py, in test2:
> 
> axis = tensor.scalar()
> 
> creates a floating point scalar, but we want an integer scalar there.  I'm
> going to change that to:
> 
> axis = tensor.scalar(dtype='int32')
> 
> and try again.  Robert-André, is that what you did?

I use int64 because it was a float64 initially. I built it in Rawhide and now f31.

Comment 20 Robert-André Mauchin 🐧 2019-08-25 18:57:00 UTC
(In reply to Miro Hrončok from comment #14)
> This hasn't been rebuild with Python 3.8 and it blocks quite a stack:
> 

I rebuilt the stack except:

python-nineml
https://github.com/INCF/nineml-python/issues/45

python-chaospy -> python-scikit-learn -> need python-joblib
https://github.com/joblib/joblib/issues/927

pythran -> python-beniget -> python-gast
https://github.com/serge-sans-paille/gast/pull/27

python-libpysal
https://github.com/pysal/libpysal/issues/177

python-scikit-image -> python-pywt
https://github.com/PyWavelets/pywt/issues/508

sagemath -> python-fpylll need qd which is retired
Fix qd with:
%ifarch s390 s390x aarch64 ppc64le
%global optflags %{optflags} -ffp-contract=off
%endif

python-biopython: update by sagitter pending in Bodhi
python-ProDy
python-neurosynth

python-scikit-image: TODO
python-ccdproc
python-photutils

Comment 21 Miro Hrončok 2019-09-16 08:52:12 UTC
Thank You.


Note You need to log in before you can comment on or make changes to this bug.