Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1716540

Summary: shiboken fails to build with Python 3.8
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: shibokenAssignee: Felix Schwarz <fschwarz>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: fschwarz, hobbes1069, jreznik, klember, orion, pviktori, rdieter, than, tvignaud
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: https://copr.fedorainfracloud.org/coprs/g/python/python3.8/package/shiboken/
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-03 09:34:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1750908, 1750909, 1686977, 1773376    

Description Miro Hrončok 2019-06-03 14:41:29 UTC
See https://copr.fedorainfracloud.org/coprs/g/python/python3.8/package/shiboken/ for actual logs. This report is automated and not very verbose, but feel free to ping me for help.

Comment 1 Richard Shaw 2019-06-03 20:10:47 UTC
Looks like it can't find everything it needs... What's changed in Python 3.8 to cause this since it worked with 3.7?

-- Could NOT find Python3Libs (missing: PYTHON3_LIBRARIES PYTHON3_INCLUDE_DIRS) 
-- Found Python3Interp: /usr/bin/python3  
-- Could NOT find Python3InterpDbg (missing: PYTHON3_DBG_EXECUTABLE) 
-- Could NOT find Python3Libs (missing: PYTHON3_LIBRARIES PYTHON3_INCLUDE_DIRS) 
-- Could NOT find Python3InterpDbg (missing: PYTHON3_DBG_EXECUTABLE) 
BUILDSTDERR: PYTHON WITH DEBUG: /usr/bin/python3
-- Found LibXml2: /usr/lib64/libxml2.so (found suitable version "2.9.9", minimum required is "2.6.32") 
-- Found LibXslt: /usr/lib64/libxslt.so (found suitable version "1.1.32", minimum required is "1.1.19") 
BUILDSTDERR: -- sphinx-build - found
-- Tests will be generated using the protected hack!
-- Found PythonInterp: /usr/bin/python3 (found version "3.8") 
BUILDSTDERR: CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
BUILDSTDERR: Please set them or make sure they are set and tested correctly in the CMake files:
BUILDSTDERR: PYTHON3_INCLUDE_DIR (ADVANCED)

The current command to build is:

%cmake .. -DUSE_PYTHON3=yes
%make_build

Comment 2 Miro Hrončok 2019-06-03 21:29:08 UTC
I think this is related to either:

https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Or the removal of "m" from abiglags.

Comment 3 Miro Hrončok 2019-06-03 21:31:50 UTC
> Or the removal of "m" from abiglags.

https://bugs.python.org/issue36707

Comment 4 Richard Shaw 2019-06-04 15:02:31 UTC
Yes, it looks like Shiboken has their own CMake kludge to detect Python and related files.

Comment 5 Richard Shaw 2019-06-05 14:56:37 UTC
Ok, for posterity here's the library names for python 3.8 (using the supplied mock config):

# ll /usr/lib64/libpython*
lrwxrwxrwx. 1 root root      19 Mar 10 18:18 /usr/lib64/libpython2.7.so -> libpython2.7.so.1.0
-rwxr-xr-x. 1 root root 2821528 Mar 10 18:19 /usr/lib64/libpython2.7.so.1.0
lrwxrwxrwx. 1 root root      19 May 29 03:38 /usr/lib64/libpython3.8.so -> libpython3.8.so.1.0
-rwxr-xr-x. 1 root root 3485576 May 29 03:39 /usr/lib64/libpython3.8.so.1.0
-rwxr-xr-x. 1 root root   15320 May 29 03:39 /usr/lib64/libpython3.so

Versus my F29 system:

$ ll /usr/lib64/libpython*
lrwxrwxrwx. 1 root root      19 Oct 15  2018 /usr/lib64/libpython2.7.so -> libpython2.7.so.1.0
-rwxr-xr-x. 1 root root 2490224 Oct 15  2018 /usr/lib64/libpython2.7.so.1.0
lrwxrwxrwx. 1 root root      20 May 10 19:48 /usr/lib64/libpython3.7m.so -> libpython3.7m.so.1.0
-rwxr-xr-x. 1 root root 3504328 May 10 19:49 /usr/lib64/libpython3.7m.so.1.0
-rwxr-xr-x. 1 root root   14480 May 10 19:49 /usr/lib64/libpython3.so

But the cmake config checks for several combinations of d|m|u including "" (none) so why does it find it?

Comment 6 Petr Viktorin (pviktori) 2019-06-07 19:28:54 UTC
It looks like shiboken first finds the Python libraries/includes, and then (independently?) the interpreter.
Usually, this is done the other way around: find the interpreter, and ask it where its supporting files are.

I could get it to work with:

%cmake .. -DUSE_PYTHON3=yes \
    -DPYTHON3_INCLUDE_DIR:PATH=$(%{__python3} -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
    -DPYTHON3_LIBRARY:PATH=$(%{__python3} -c "from distutils.sysconfig import get_config_var; print(get_config_var('LDLIBRARY'))")


That uncovers a Python 3.8 incompatibility, which can fixed by only setting tp_print on Python 2:

--- Shiboken-1.2.4/libshiboken/sbkenum.cpp (original)
+++ Shiboken-1.2.4/libshiboken/sbkenum.cpp (updated)
@@ -529,7 +529,9 @@
     ::memset(type, 0, sizeof(SbkEnumType));
     Py_TYPE(type) = &SbkEnumType_Type;
     type->tp_basicsize = sizeof(SbkEnumObject);
+#ifndef IS_PY3K
     type->tp_print = &SbkEnumObject_print;
+#endif
     type->tp_repr = &SbkEnumObject_repr;
     type->tp_str = &SbkEnumObject_repr;
     type->tp_flags = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_CHECKTYPES;


That uncovers a test failure that I can't wrap my head around now.

Comment 7 Richard Shaw 2019-06-07 19:51:33 UTC
Ack, so no quick fix. Sounds like I need to get PySide2 in Fedora which HOPEFULLY doesn't have any python 3.8 issues.

Comment 8 Miro Hrončok 2019-06-24 17:57:20 UTC
Let's start by contacting upstream?

Comment 9 Richard Shaw 2019-06-25 11:31:09 UTC
I can try when I'm back from vacation, but the answer may be "move to PySide/Shiboken 2"

Comment 10 Ben Cotton 2019-08-13 16:57:29 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 11 Ben Cotton 2019-08-13 19:09:20 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 12 Miro Hrončok 2019-08-14 22:24:50 UTC
The coordinated rebuild of Python 3.8 has started in the `f32-python` side tag.

If you figure out how to rebuild this package, please don't rebuild it in regular rawhide, but use the side tag instead:

    on branch master:
    $ fedpkg build --target=f32-python

To wait for a build to show up in the side tag, do:

    $ koji wait-repo f32-python --build=<nvr>

Where <nvr> is name-version-release of the source package, e.g. python-foo-1.1-2.fc32.

An updated mock config is posted at:
http://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Note that it will take a while before the essential packages are rebuilt, so don't expect all your dependencies to be available right away.

Thanks. Let us know if you need up to date info, or if you have any questions.



PS this message is mass posted to all the bugs that block the PYTHON38 bug. If this is also a Fedora 31 FTBFS bug and you manage to fix it, you can do a f31 build as usual:

    on branch f31:
    $ fedpkg build

Comment 13 Richard Shaw 2019-08-15 01:21:38 UTC
If I can get pyside2/shiboken2 building again I think the best course of action is to move all the consumers of pyside/shiboken to that package, however, I have no idea how easy that will be.

Comment 14 Miro Hrončok 2019-08-21 16:35:36 UTC
The f32-python side tag has been merged. In order to rebuild the package, do it in regular rawhide, but please wait until python3-3.8 is tagged:

  $ koji wait-repo f32-build --build python3-3.8.0~b3-3.fc32


If your built already started in f32-python, after it is finished, please tag it to rawhide with:

  $ koji tag-build f32-pending <nvr>

For example:

  $ koji tag-build f32-pending libreoffice-6.3.0.4-3.fc32

Thanks!

(This comment is mass posted to all bugzillas blocking the PYTHON38 tracking bug.)

Comment 15 Miro Hrončok 2019-08-21 17:29:34 UTC
(Python 3.8 has landed in the rawhide buildroot.)

Comment 16 Richard Shaw 2019-11-18 00:17:43 UTC
I'm not sure if there's anything else relying on it but PySide1 does not support Python3 so this should be retired.

Comment 17 Felix Schwarz 2019-11-18 07:54:25 UTC
(In reply to Richard Shaw from comment #16)
> I'm not sure if there's anything else relying on it but PySide1 does not
> support Python3 so this should be retired.

+1 from me. I think we should retire the whole pyside1 stack: generatorrunner, apiextractor, shiboken and python-pyside.

From a quick glance only "ubertooth" depends on pyside1 and they ship a custom patch to use pyside1 instead of pyside2 so switching should be no problem for that package. I filed bug 1773445 to inform the maintainer.

Comment 18 Felix Schwarz 2019-11-25 20:58:16 UTC
(I sent this text also by mail to the various "*-maintainers" aliases.)

I'd like to propose that we retire the python-pyside(v1) stack in Fedora
rawhide. The stack does not build anymore with Python 3.8:

- shiboken: bug 1716540
- python-pyside: bug 1773376

Also upstream moved to pyside2 several years ago and pyside2 is available in Fedora.

To me the case if pretty clear but of course if there is someone else who wants to spend time in getting the packages in shape again, I'd support keeping the stack.

=> If there are no objections I'll try to retire these packages in 7 days
   (in rawhide).

I would prefer explicit ACKs though.

Comment 19 Felix Schwarz 2019-12-03 09:34:23 UTC
package now retired in rawhide (see also bug 1773376 for vote count)

Comment 20 Miro Hrončok 2019-12-03 10:05:45 UTC
What is the pyside2 replacement?

Comment 21 Miro Hrončok 2019-12-03 10:10:00 UTC
Ignore my previous comment, it was posted to a wrong bugzilla. I was asking about generatorrunner.