Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1705482
Summary: | python 3.8 requires using -embed variant of pkgconf file, causing libguestfs to fail to build Python 3.8 bindings | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Miro Hrončok <mhroncok> | ||||||||
Component: | libguestfs | Assignee: | Richard W.M. Jones <rjones> | ||||||||
Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
Severity: | high | Docs Contact: | |||||||||
Priority: | unspecified | ||||||||||
Version: | rawhide | CC: | praiskup, rjones | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | libguestfs-1.40.2-5.fc31 | Doc Type: | If docs needed, set a value | ||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2019-06-02 20:30:26 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | |||||||||||
Bug Blocks: | 1686977 | ||||||||||
Attachments: |
|
We regularly build libguestfs against Python 3.8, so this is unlikely to have anything to do with Python. What's actually happening here is a failure (of a kind) in Copr. We rely on the downloaded RPMs from the BuildRequires staying around in the cache for the build. However in this case it looks like the earlier cached packages are being removed. I think this is controlled by one of the dnf settings like 'keepcache=True', so likely setting that will fix the problem. If a way to file bugs against Copr exists then we could reassign the bug there. (In reply to Richard W.M. Jones from comment #1) > We regularly build libguestfs against Python 3.8, so this is unlikely to have > anything to do with Python. Most likely not, yet it prevents the 3.8 rebuild. > What's actually happening here is a failure (of a kind) in Copr. We rely on > the downloaded RPMs from the BuildRequires staying around in the cache for > the build. However in this case it looks like the earlier cached packages > are being removed. I think this is controlled by one of the dnf settings > like 'keepcache=True', so likely setting that will fix the problem. Is this a proper thing to rely on? > If a way to file bugs against Copr exists then we could reassign the bug > there. It is. However I'm not confident this is an actual bug. Anyway, as a workaround, what is the best thing to do to skip only the tests that rely on this behavior? In the meantime, i'll try to rebuild in local mock to see if it builds. It builds fine locally in mock, lowering severity. Who's to say what's proper? In any case it is required and you can't build the package at all without it. > you can't build the package at all without it.
So this is not only needed for %check?
It's needed to build the entire package. Here's the request when we enabled keepcache for Koji: https://pagure.io/koji/pull-request/16 Thanks for clarification. Nice Copr people, could you please enable keepcache? See comment #1. Shouldn't this be the default in mock, then? Copr dumps the mock configs used for the build: https://copr-be.cloud.fedoraproject.org/results/%40python/python3.8/fedora-rawhide-x86_64/00901206-libguestfs/configs/fedora-rawhide-x86_64.cfg There seems to be keepcache=1 used already. Yes, and the buildlog: https://copr-be.cloud.fedoraproject.org/results/%40python/python3.8/fedora-rawhide-x86_64/00901206-libguestfs/builder-live.log indicates that the package is there. The culprit is likely different. I don't think so. See the lines: + find /var/cache/dnf /var/cache/yum -type f -name '*.rpm' -print0 + createrepo repo ... In the output of createrepo libtirpc (not -devel) is not listed, indicating that it wasn't present in /var/cache/{dnf,yum}. However if keepcache=1 is really present then that means something else is going on which I don't understand. There is libtirpc-devel > + find /var/cache/dnf /var/cache/yum -type f -name '*.rpm' -print0 > + createrepo repo > 01:20:24: Version: 0.12.2 (Features: DeltaRPM LegacyWeakdeps ) > .... > 01:20:24: Adding pkg: repo/libldm-devel-0.2.4-4.fc30.x86_64.rpm > 01:20:24: Adding pkg: repo/libtirpc-devel-1.1.4-2.rc2.fc30.1.x86_64.rpm And checking your spec you have there: > BuildRequires: libtirpc-devel You do not require libtirpc itself. And you have some other errors during build - but probably not related to this issue: Start: build setup for libguestfs-1.40.2-5.fc31.src.rpm sh: /usr/bin/perl: No such file or directory warning: line 718: Possible unexpanded macro in: Requires: php(zend-abi) = %{php_zend_api} warning: line 719: Possible unexpanded macro in: Requires: php(api) = %{php_core_api} sh: /usr/bin/perl: No such file or directory What's the resolution? How can I move this forward? (In reply to Miroslav Suchý from comment #13) > > BuildRequires: libtirpc-devel > > You do not require libtirpc itself. That's implied because: $ rpm -qR libtirpc-devel /bin/sh /bin/sh /usr/bin/pkg-config libtirpc(x86-64) = 1.1.4-2.rc2.fc30.1 libtirpc.so.3()(64bit) ... $ rpm -q --whatprovides 'libtirpc.so.3()(64bit)' libtirpc-1.1.4-2.rc2.fc30.1.x86_64 However it may be a case that libtirpc is already installed and so doesn't get pulled into /var/cache/yum? It's hard to tell without having access to the buildroot, so I don't think there's any feasible way to debug this further. I've just got: "No package libtirpc.x86_64 available" in local mock. $ fedpkg srpm $ mock -r ~/.config/mock/fedora-rawhide-x86_64-python3.8.cfg clean $ mock -r ~/.config/mock/fedora-rawhide-x86_64-python3.8.cfg --no-clean --no-cleanup-after ./libguestfs-1.40.2-4.fc31.src.rpm (Config from https://copr.fedorainfracloud.org/coprs/g/python/python3.8) It builds for me when I use the standard cfg (ie. /etc/mock/fedora-rawhide-x86_64.cfg). It also works for me with the cfg from https://copr.fedorainfracloud.org/coprs/g/python/python3.8 although the build failed at the end for an unrelated reason: RPM build errors: File not found: /builddir/build/BUILDROOT/libguestfs-1.40.2-4.fc31.x86_64/usr/lib64/python3.8/site-packages/libguestfsmod*.so File not found: /builddir/build/BUILDROOT/libguestfs-1.40.2-4.fc31.x86_64/usr/lib64/python3.8/site-packages/guestfs.py File not found: /builddir/build/BUILDROOT/libguestfs-1.40.2-4.fc31.x86_64/usr/lib64/python3.8/site-packages/__pycache__/guestfs*.py* File not found: /builddir/build/BUILDROOT/libguestfs-1.40.2-4.fc31.x86_64/usr/share/man/man3/guestfs-python.3* Notice that the python3.8 cfg starts by including /etc/mock/fedora-rawhide-x86_64.cfg On my local machine that has the [main] keepcache=1 setting, so maybe that is missing on your machine? My mock config also has: [main] keepcache=1 (it's the default config as provided by mock-core-configs-30.3-1.fc30.noarch) Either way a build failure is what will block the real rebuild in Koji. Would you mind opening a "libguestfs fails to build with Python 3.8" bug and attaching the build log? We routinely build libguestfs in Koji so I doubt there's going to be a problem. The issue seems to be something to do with copr. As for the Python 3.8 failure, it doesn't seem to be related to libguestfs but something in the %{python*} macros. I'll attach the log to this bug once I have recreated them. Created attachment 1575605 [details]
build.log
Created attachment 1575606 [details]
root.log
> We routinely build libguestfs in Koji so I doubt there's going to be a problem. In that case, repurposing this bug to the actual failure. We'd like to start building in koji next week. --- Checking for Python --- checking for python... /usr/bin/python3 checking Python version... 3.8 checking for PYTHON... yes checking Python prefix... /usr checking for Python site-packages path... /usr/lib64/python3.8/site-packages checking for Python extension suffix (PEP-3149)... .cpython-38-x86_64-linux-gnu.so checking for PyCapsule_New in -lc... yes checking for PyString_AsString in -lc... no ... ------------------------------------------------------------ Thank you for downloading libguestfs 1.40.2 This is how we have configured the optional components for you today: Daemon .............................. yes Appliance ........................... yes QEMU ................................ /usr/bin/qemu-kvm guestfish and C-based virt tools .... yes FUSE filesystem ..................... yes Default backend ..................... libvirt GNU gettext for i18n ................ yes virt-p2v ............................ yes OCaml bindings ...................... yes OCaml-based virt tools .............. yes Perl bindings ....................... yes Perl-based virt tools ............... yes Python bindings ..................... no Ruby bindings ....................... yes Java bindings ....................... no Haskell bindings .................... no PHP bindings ........................ yes Erlang bindings ..................... yes Lua bindings ........................ yes Go bindings ......................... no gobject bindings .................... yes gobject introspection ............... yes bash completion ..................... yes If any optional component is configured 'no' when you expected 'yes' then you should check the preceding messages. Please report bugs back to the mailing list: http://www.redhat.com/mailman/listinfo/libguestfs Next you should type 'make' to build the package, then 'make check' to run the tests. Or run 'make help' to list some common targets. ------------------------------------------------------------------------------------------------------------------------ Thank you for downloading libguestfs 1.40.2 This is how we have configured the optional components for you today: Daemon .............................. yes Appliance ........................... yes QEMU ................................ /usr/bin/qemu-kvm guestfish and C-based virt tools .... yes FUSE filesystem ..................... yes Default backend ..................... libvirt GNU gettext for i18n ................ yes virt-p2v ............................ yes OCaml bindings ...................... yes OCaml-based virt tools .............. yes Perl bindings ....................... yes Perl-based virt tools ............... yes Python bindings ..................... no Ruby bindings ....................... yes Java bindings ....................... no Haskell bindings .................... no PHP bindings ........................ yes Erlang bindings ..................... yes Lua bindings ........................ yes Go bindings ......................... no gobject bindings .................... yes gobject introspection ............... yes bash completion ..................... yes If any optional component is configured 'no' when you expected 'yes' then you should check the preceding messages. Please report bugs back to the mailing list: http://www.redhat.com/mailman/listinfo/libguestfs Next you should type 'make' to build the package, then 'make check' to run the tests. Or run 'make help' to list some common targets. ------------------------------------------------------------ > something in the %{python*} macros. Macros all are good, but Python itself has changed. Check out https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build I'm guessing the build system is building something that needs to be linked against libpython, it fails because it isn't and the tooling determines Python is not working. I understand Python but I don't understand the buildsystem of libguestfs. If you help me undersstand what exactly this line is doing, I can help you fix this: Python bindings ..................... no https://github.com/libguestfs/libguestfs/blob/ca8f8afcc5b5b77be69f5b353ed8aef3fae1883d/m4/guestfs-python.m4#L28 The build.log file isn't very helpful, but looking in the generated python/Makefile shows that PYTHON_LIBS is empty. This comes from the /usr/lib64/pkgconfig/python3.pc file, where indeed the libs entry is empty: # See: man pkg-config prefix=/usr exec_prefix=/usr libdir=/usr/lib64 includedir=/usr/include Name: Python Description: Build a C extension for Python Requires: Version: 3.8 Libs.private: -lcrypt -lpthread -ldl -lutil -lm Libs: Cflags: -I${includedir}/python3.8 I notice there is also a python3-embed.pc file which has the expected Libs. This seems to be a new "feature" of Python 3.8 (not in 3.7). It is briefly mentioned in the link you gave. I'm unclear from the description whether or not we should use the -embed variant. What's the difference between "embed Python into an application" and "C extensions must not be linked to libpython"? C extensions are (eventually) linked into an application. C extensions are loaded by Python, so there are not explicitly linked to libpython as they used to be: $ ldd /usr/lib64/python3.7/lib-dynload/array.cpython-37m-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffc8f7cc000) libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (0x00007f775c67b000) libc.so.6 => /lib64/libc.so.6 (0x00007f775c4b5000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f775c494000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f775c48e000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f775c489000) libm.so.6 => /lib64/libm.so.6 (0x00007f775c343000) /lib64/ld-linux-x86-64.so.2 (0x00007f775ca03000) $ ldd /usr/lib64/python3.8/lib-dynload/array.cpython-38-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffff6bc5000) libc.so.6 => /lib64/libc.so.6 (0x00007f743ca69000) /lib64/ld-linux-x86-64.so.2 (0x00007f743cc67000) If you use the thing (eg. an ELF LSB shared object) in any other way than simply "import thing" from Python, than I guess you are "embedding" Python (most likely calling [Py_Initialize] or similar). [Py_Initialize] https://docs.python.org/3/c-api/init.html#c.Py_Initialize Ad copr, we have: config_opts['plugin_conf']['yum_cache_enable'] = False .. which might be related, I'll try to debug. But locally it behaves even differently, since the find /var/cache/dnf says there are _no_ packages. And even then, the build succeeds. (In reply to Pavel Raiskup from comment #26) > Ad copr, we have: > config_opts['plugin_conf']['yum_cache_enable'] = False > .. which might be related, I'll try to debug. But locally it behaves > even differently, since the find /var/cache/dnf says there are _no_ > packages. And even then, the build succeeds. It uses the network if it's available. See what the "ping" command says. This isn't upstream yet but I added the patch to Fedora Rawhide and here's a build against Python 3.7: https://koji.fedoraproject.org/koji/taskinfo?taskID=35169299 Thanks. FTR I can build the package in Copr by enabling internet connection. Copr builds should be fixed by: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9988126185fb4152a18f83bcd7b8a5e551830720 Fixed upstream in c4638739d684609471cfa94c08e7b90c70ea3e83 (libguestfs >= 1.40.1) |
Created attachment 1561561 [details] Full log from Copr libguestfs 1:1.40.2-5.fc31 fails to build with Python 3.8 (the failure might not be Python 3.8 related): Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync DNF version: 4.2.5 cachedir: /builddir/build/BUILD/libguestfs-1.40.2/cachedir Unknown configuration value: failovermethod=priority in /builddir/build/BUILD/libguestfs-1.40.2/yum.conf; Configuration: OptionBinding with id "failovermethod" does not exist repo: downloading from remote: local local 138 MB/s | 732 kB 00:00 local: using metadata from Thu May 2 01:20:27 2019. No module defaults found Completion plugin: Can't write completion cache: unable to open database file No package libtirpc.x86_64 available. Exiting due to strict setting. Error: No package libtirpc.x86_64 available. supermin: /usr/bin/dnf download -v -c '/builddir/build/BUILD/libguestfs-1.40.2/yum.conf' --destdir='/var/tmp/supermine1158f.tmpdir/d704im41' --disableexcludes=all 'iproute.x86_64' 'libtirpc.x86_64' 'nilfs-utils.x86_64' 'openssh-clients.x86_64' 'policycoreutils.x86_64' 'syslinux-extlinux.x86_64' 'systemd.x86_64' 'vim-minimal.x86_64' 'zfs-fuse.x86_64' 'bash.x86_64' 'e2fsprogs.x86_64' 'file.x86_64' 'grep.x86_64' 'lvm2.x86_64' 'mdadm.x86_64' 'rsync.x86_64' 'systemd-udev.x86_64' 'util-linux.x86_64' 'kpartx.x86_64' 'glibc.x86_64' 'libattr.x86_64' 'coreutils-common.x86_64' 'openssl-libs.x86_64' 'libpwquality.x86_64' 'dpkg.x86_64' 'wget.x86_64' 'dracut.x86_64' 'krb5-libs.x86_64' 'libgcrypt.x86_64' 'crypto-policies.noarch' 'openssh.x86_64' 'audit-libs.x86_64' 'libsemanage.x86_64' 'rpm.x86_64' 'pam.x86_64' 'mtools.x86_64' 'shadow-utils.x86_64' 'setup.noarch' 'ca-certificates.noarch' 'openldap.x86_64' 'fedora-release-common.noarch' 'fuse-common.x86_64' 'gnupg2.x86_64' 'fedora-repos.noarch' 'dbus-common.noarch' 'fedora-repos-rawhide.noarch': command failed, see earlier errors make[2]: *** [Makefile:2153: stamp-supermin] Error 1 make[2]: Leaving directory '/builddir/build/BUILD/libguestfs-1.40.2/appliance' make[1]: *** [Makefile:2247: all-recursive] Error 1 make[1]: Leaving directory '/builddir/build/BUILD/libguestfs-1.40.2' make: *** [Makefile:2151: all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.XoPpZw (%build) Full log attached.