Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1841335 - python-shapely fails to build on s390x, blocking the Python 3.9 rebuild of many packages
Summary: python-shapely fails to build on s390x, blocking the Python 3.9 rebuild of ma...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: geos
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Sandro Mani
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1842150 (view as bug list)
Depends On:
Blocks: ZedoraTracker F32FTBFS F33FTBFS PYTHON39 1838465 1841631 1841684 1841695 1841702 1841712 1841788 1841791 1842049 1842107 1842156 1842312
TreeView+ depends on / blocked
 
Reported: 2020-05-28 21:48 UTC by Miro Hrončok
Modified: 2020-06-17 09:38 UTC (History)
11 users (show)

Fixed In Version: geos-3.8.1-2.fc33, python-shapely-1.7-3b1.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-17 09:38:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Demonstration of geos bug (2.70 KB, text/x-csrc)
2020-05-31 15:53 UTC, Jerry James
no flags Details
C++ version of demonstration (2.89 KB, text/x-csrc)
2020-05-31 18:19 UTC, Jerry James
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github libgeos geos issues 323 0 None open GEOSWithin gives bad results on big endian architectures (s390x) 2020-06-16 12:14:17 UTC

Description Miro Hrončok 2020-05-28 21:48:07 UTC
python-shapely-1.7-3b1.fc33 fails to build on s390x with Python 3.9. Other arches build fine.

See https://koji.fedoraproject.org/koji/buildinfo?buildID=1513447

Multiple failures are linked from https://src.fedoraproject.org/rpms/python-shapely/c/a31a3ba9d1e52574dfe26999c5bd99903129e281?branch=master

The failure is:

======================================================================
FAIL: test_topological_error (tests.test_iterops.IterOpsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Shapely-1.7b1/tests/test_iterops.py", line 69, in test_topological_error
    list(iterops.within(polygon1, [polygon2]))
AssertionError: TopologicalError not raised
======================================================================
FAIL: test_binary_predicate_exceptions (tests.test_predicates.PredicatesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Shapely-1.7b1/tests/test_predicates.py", line 42, in test_binary_predicate_exceptions
    self.assertRaises(TopologicalError, Polygon(p1).within, Polygon(p2))
AssertionError: TopologicalError not raised by within
======================================================================
FAIL: test_split_line_with_line (tests.test_split.TestSplitLine)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Shapely-1.7b1/tests/test_split.py", line 122, in test_split_line_with_line
    self.assertTrue(splitter.touches(self.ls))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 225 tests in 0.633s
FAILED (failures=3, skipped=1)

This blocks the Python 3.9 rebuild of this and all dependent packages:

$ repoquery --repo=rawhide{,-source} --whatrequires python3-shapely --recursive
cura-1:4.6.1-1.fc33.noarch
cura-1:4.6.1-1.fc33.src
cura-fdm-materials-0:4.6.1-1.fc33.noarch
flatcam-0:8.5-17.20180606git46454c293a9b.fc32.noarch
pyosmium-0:3.0.0-1.fc33.src
pyproj-0:2.6.1.post1-1.fc33.src
python-cartopy-0:0.18.0-1.fc33.src
python-descartes-0:1.1.0-14.fc32.src
python-fiona-0:1.8.13-4.fc33.src
python-geopandas-0:0.7.0-1.fc33.src
python-geoplot-0:0.4.0-2.fc33.src
python-libpysal-0:4.2.2-1.fc32.src
python-mapclassify-0:2.2.0-1.fc33.src
python-missingno-0:0.4.2-3.fc33.src
python-tilestache-0:1.51.14-3.fc32.src
python-trimesh-0:3.6.34-1.fc33.src
python-uranium-0:4.6.1-1.fc33.src
python3-cartopy-0:0.18.0-1.fc33.x86_64
python3-geopandas-0:0.7.0-1.fc33.noarch
python3-geoplot-0:0.4.0-2.fc33.noarch
python3-missingno-0:0.4.2-3.fc33.noarch
python3-trimesh-all-0:3.6.34-1.fc33.noarch
python3-trimesh-easy-0:3.6.34-1.fc33.noarch
python3-uranium-0:4.6.1-1.fc33.noarch

This is a big stack, hence setting the severity to high.

If this is fixed, please rebuild the package in the f33-python target. See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/24UMFCUXDERAP5KAU3QMWQRKOVLSBSJR/

Thanks.

Comment 1 Miro Hrončok 2020-05-29 07:11:23 UTC
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.

Comment 2 Dan Horák 2020-05-29 09:07:20 UTC
If I see right, then nothing changed in python-shapely, but the buildroot is different ...

rebuild in F-31 = OK
rebuild in F-32 -> same errors as with the rawhide/python-3.9 builds

Comment 3 Jerry James 2020-05-29 20:12:15 UTC
The python code didn't change, but geos did.  The last successful build of python-shapely was on January 31.  On March 2, geos 3.8.0 was built for F32+, followed later by geos 3.8.1.  There may be an endianness bug in geos 3.8.x.

Comment 4 Jerry James 2020-05-29 20:13:52 UTC
Or something changed bit width, and the shapely cython code needs to be updated.

Comment 5 Petr Viktorin 2020-05-29 21:03:57 UTC
Please always regenerate Cython code on build. I sent a PR for it: https://bugzilla.redhat.com/show_bug.cgi?id=1841335

Sadly, it doesn't help the FTBFS :(

Comment 6 Miro Hrončok 2020-05-29 21:27:15 UTC
The PR link is: https://src.fedoraproject.org/rpms/python-shapely/pull-request/5

Comment 7 Miro Hrončok 2020-05-30 22:41:08 UTC
*** Bug 1842150 has been marked as a duplicate of this bug. ***

Comment 8 Jerry James 2020-05-31 15:53:49 UTC
Created attachment 1693859 [details]
Demonstration of geos bug

The attached file is a C translation of one of the failing python-shapely tests.  It shows that the bug is in geos, not python-shapely.  Build it with "gcc -o test test.c -lgeos_c".  If built and run on a little endian machine, it correctly produces this output:

TopologyException: side location conflict at 459 346
The result is 2

If built and run on s390x, it incorrectly produces this output:

The result is 0

Comment 9 Jerry James 2020-05-31 18:19:13 UTC
Created attachment 1693904 [details]
C++ version of demonstration

This version demonstrates that the bug is in the C++ core of geos, not in the C wrapper that python-shapely uses.  Build it like this: "g++ -o test test.cpp -lgeos".  You will see warnings, but they can be ignored.

Comment 10 Igor Raits 2020-05-31 18:34:49 UTC
Hello,

Please note that this comment was generated automatically. If you feel that this output has mistakes, please contact me via email (ignatenkobrain).

All subpackages of a package agaisnt which this bug was filled are now installable or removed from Fedora 33.

Thanks for taking care of it!

Comment 11 Miro Hrončok 2020-06-01 11:17:26 UTC
Can we temporarily skip the python-shapely 3 failing shapely tests on s390x to unblock handful of packages that might not be affected by this problem?

Comment 12 Jerry James 2020-06-01 14:33:10 UTC
Since geos is computing incorrect results on s390x, then python-shapely is too.  That means that anything sitting on top of python-shapely is likely to fail its test suite on s390x, too, I would think.  We should concentrate our efforts on diagnosing and fixing the geos problem.

In fact, I tried to do just that, but ran into bug 1842325.  If somebody can fix that, I think we should be able to fix the geos bug pretty quickly.

Note that I am not a maintainer for either geos or python-shapely.  I'm just Random Drive-by User who is trying to lend a hand.

Comment 13 Jerry James 2020-06-01 23:10:42 UTC
I've noticed two things:

1) The geos spec file does this:

%ifarch armv7hl aarch64 s390x ppc64le
make test || :
%else
make test
%endif

and indeed, many tests fail on s390x.

2) The reason I can't get the C++ pretty printers to work for me so I can debug this issue is that python support was compiled out of gdb due to bug 1829702.

Comment 14 Miro Hrončok 2020-06-01 23:24:20 UTC
2) Can you debug this on Fedora 32?

Comment 15 Sandro Mani 2020-06-03 13:02:16 UTC
Latest geos git still suffers from test failures [1].

Any chance of getting ssh access to a s390x machine to bisect this?

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=45318460

Comment 16 Dan Horák 2020-06-03 13:08:29 UTC
(In reply to Sandro Mani from comment #15)
> Latest geos git still suffers from test failures [1].
> 
> Any chance of getting ssh access to a s390x machine to bisect this?
> 
> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=45318460

I can arrange access to s390x, send me an email to sharkcz

Comment 17 Sandro Mani 2020-06-10 08:06:35 UTC
@Dan Horák Have you received my PM?

Comment 18 Dan Horák 2020-06-10 08:11:33 UTC
(In reply to Sandro Mani from comment #17)
> @Dan Horák Have you received my PM?

yes, and I have replied the other day, please check your spam box ...

Comment 19 Sandro Mani 2020-06-10 08:14:11 UTC
Hmm don't see it anywhere. Can you please resend?

Comment 20 Dan Horák 2020-06-10 08:26:10 UTC
(In reply to Sandro Mani from comment #19)
> Hmm don't see it anywhere. Can you please resend?

done, sender is dan at danny dot cz

Comment 21 Miro Hrončok 2020-06-12 12:31:43 UTC
FTR: I'm trying to bisect this trough Koji.

Comment 22 Miro Hrončok 2020-06-12 13:53:09 UTC
This is what I got:

10ff0f8d6809c212aa9eb75f710bd068a90ccff0 is the first new commit
commit 10ff0f8d6809c212aa9eb75f710bd068a90ccff0
Author: Paul Ramsey <pramsey>
Date:   Fri Dec 14 09:24:28 2018 -0800

    Remove references to RobustDeterminant
    JTS 2fc9fa215cdfff4210ec55cb8c6cfb79618b4ff2

 src/algorithm/CGAlgorithmsDD.cpp     | 5 +++++
 src/algorithm/RayCrossingCounter.cpp | 3 ++-
 src/algorithm/SIRtreePointInRing.cpp | 3 ++-
 3 files changed, 9 insertions(+), 2 deletions(-)


I am not available to inspect the commit at this point.

Comment 24 Miro Hrončok 2020-06-12 14:05:39 UTC
(the commit does not rever cleanly on 3.8.1)

Comment 25 Sandro Mani 2020-06-12 14:56:24 UTC
I'm also looking into it, the problem is that newly added tests were immediately failing on those arches, so the actual code may have already been broken on those arches way before...

Comment 26 Miro Hrončok 2020-06-15 09:52:53 UTC
Note that the bisected commit is the one where the test.c reproducer from Jerry starts to give incorrect results on s390x. I.e. I was not bisecting based on unit test results, but only on this one case.

Comment 27 Sandro Mani 2020-06-15 21:23:56 UTC
Likely the DD class [1][2] itself being the issue...

[1] https://github.com/libgeos/geos/blob/master/include/geos/math/DD.h
[2] https://github.com/libgeos/geos/blob/master/src/math/DD.cpp

Comment 28 Miro Hrončok 2020-06-16 12:14:18 UTC
I've summarized this to https://github.com/libgeos/geos/issues/323 but that is only a mirror. Upstream uses https://trac.osgeo.org/geos/report/1 where I requested an account (pending approval).

Comment 29 Miro Hrončok 2020-06-16 20:43:40 UTC
(In reply to Sandro Mani from comment #15)
> Latest geos git still suffers from test failures [1].
> 
> Any chance of getting ssh access to a s390x machine to bisect this?
> 
> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=45318460

bed36f15c780057ae9b83eb9cd2e8ef6a9ada498 is the fixer of the C reproducer. Not sure if it fixes shapely.

Latest goes git still fails tests, but might actually fix shapely tests.

Comment 30 Miro Hrončok 2020-06-16 22:51:29 UTC
I've opened https://src.fedoraproject.org/rpms/geos/pull-request/2

Could somebody with access to s390x verify it fixes shapely tests?

Alternatively, Dan, could you e-mail me the access details?

Comment 31 Dan Horák 2020-06-17 09:04:27 UTC
It looks good, I have used F-32, rebuilt geos with the PR #2, installed the rpms, then built shapely, all tests passed.

Comment 32 Sandro Mani 2020-06-17 09:17:41 UTC
Thanks Miro for taking care of this and sorry for not being more responsive, dayjob is keeping me pretty busy at the moment.

Comment 33 Miro Hrončok 2020-06-17 09:38:02 UTC
Thank you all.


Note You need to log in before you can comment on or make changes to this bug.