Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1718294 - nbdkit fails to build with Python 3.8
Summary: nbdkit fails to build with Python 3.8
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: nbdkit
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-06-07 12:34 UTC by Petr Viktorin
Modified: 2019-06-10 10:42 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-10 08:44:01 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Petr Viktorin 2019-06-07 12:34:32 UTC
nbdkit fails to build with Python 3.8.0b1.

Looking at the build, it seems "nbdkit python --dump-plugin" attempts to load a Python extension module directly as a C shared library. This is not supported; Python modules should be imported from Python code or using C API like PyImport_ImportModule.

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.8/fedora-rawhide-x86_64/00927318-nbdkit/

For all our attempts to build nbdkit with Python 3.8, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/package/nbdkit/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.8:
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Let us know here if you have any questions.

Comment 1 Richard W.M. Jones 2019-06-07 12:46:09 UTC
Your interpretation is wrong, the *.so file isn't a Python module, and we use
PyRun_SimpleFileEx to run the Python code.

The reason it fails to build is some problem in Python 3.8:

> undefined symbol: PyByteArray_Type

I can see a load of people have this problem on-line, but no clear explanation
of why it happens or resolution for it.

Comment 2 Richard W.M. Jones 2019-06-07 16:40:21 UTC
Oh I see, it's because in this case we do need to use python-embed.pc instead
of python.pc.

Comment 3 Richard W.M. Jones 2019-06-09 21:11:28 UTC
Upstream fix:
https://github.com/libguestfs/nbdkit/commit/fb1a74b3d84571bc4c6bb57b26128c92b3368c52

nbdkit >= 1.13.5

Soon to be added to Rawhide.

Comment 4 Miro Hrončok 2019-06-10 07:38:33 UTC
It still fails:

+ echo ./test-dump-plugin.sh: run cleanup hooks: exit code 1
./test-dump-plugin.sh: run cleanup hooks: exit code 1
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 1
FAIL test-dump-plugin.sh (exit status: 1)
SKIP: test-memory-largest.sh
============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -v -P memory-largest.pid -U /tmp/tmp.6wzqEvg2nx -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so 9223372036854775807
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so
nbdkit: debug: registered plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so (name memory)
nbdkit: debug: memory: load
nbdkit: debug: memory: config key=size, value=9223372036854775807
nbdkit: debug: memory: config_complete
nbdkit: debug: bound to unix socket /tmp/tmp.6wzqEvg2nx
nbdkit: debug: forked into background (new pid = 17321)
nbdkit: debug: written pidfile /builddir/build/BUILD/nbdkit-1.13.5/tests/memory-largest.pid
./test-memory-largest.sh: run cleanup hooks: exit code 77
nbdkit: debug: memory: unload
SKIP test-memory-largest.sh (exit status: 77)
SKIP: test-pattern-largest.sh
=============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -v -P pattern-largest.pid -U /tmp/tmp.waDs8y95wc -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so 9223372036854775807
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so
nbdkit: debug: registered plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so (name pattern)
nbdkit: debug: pattern: load
nbdkit: debug: pattern: config key=size, value=9223372036854775807
nbdkit: debug: pattern: config_complete
nbdkit: debug: bound to unix socket /tmp/tmp.waDs8y95wc
nbdkit: debug: forked into background (new pid = 20699)
nbdkit: debug: written pidfile /builddir/build/BUILD/nbdkit-1.13.5/tests/pattern-largest.pid
./test-pattern-largest.sh: run cleanup hooks: exit code 77
nbdkit: debug: pattern: unload
SKIP test-pattern-largest.sh (exit status: 77)
SKIP: test-vddk-real.sh
=======================
+ requires test x '!=' x
+ echo './test-vddk-real.sh: â€test x != x’ failed with error code 1'
./test-vddk-real.sh: â€test x != x’ failed with error code 1
+ echo './test-vddk-real.sh: test prerequisite is missing or not working'
./test-vddk-real.sh: test prerequisite is missing or not working
+ exit 77
+ _run_cleanup_hooks
+ status=77
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo ./test-vddk-real.sh: run cleanup hooks: exit code 77
./test-vddk-real.sh: run cleanup hooks: exit code 77
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 77
SKIP test-vddk-real.sh (exit status: 77)
FAIL: test-python-exception.sh
==============================
+ test '!' -d .
+ '[' '' = 1 ']'
+ output=test-python-exception.out
+ rm -f test-python-exception.out
+ cleanup_fn rm -f test-python-exception.out
+ _cleanup_hook[${#_cleanup_hook[@]}]='rm -f test-python-exception.out'
+ nbdkit -f -v python ./python-exception.py test=simple
+ :
+ cat test-python-exception.out
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -f -v -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so ./python-exception.py test=simple
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: error: cannot open plugin '/builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so': /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so: undefined symbol: PyByteArray_Type
Use 'nbdkit --help' or read the nbdkit(1) manual page for documentation.
+ grep 'this is the test string' test-python-exception.out
++ _run_cleanup_hooks
++ status=1
++ set +e
++ trap '' INT QUIT TERM EXIT ERR
++ echo ./test-python-exception.sh: run cleanup hooks: exit code 1
./test-python-exception.sh: run cleanup hooks: exit code 1
++ (( i = 0 ))
++ (( i < 1 ))
++ rm -f test-python-exception.out
++ (( ++i  ))
++ (( i < 1 ))
++ exit 1
FAIL test-python-exception.sh (exit status: 1)
FAIL: test-shebang-python.sh
============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -P shebang-python.pid -U /tmp/tmp.CWpqXvwwgD -f -v -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so ./shebang.py
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: error: cannot open plugin '/builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so': /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so: undefined symbol: PyByteArray_Type
Use 'nbdkit --help' or read the nbdkit(1) manual page for documentation.
./test-shebang-python.sh: PID file was not created
./test-shebang-python.sh: run cleanup hooks: exit code 1
FAIL test-shebang-python.sh (exit status: 1)
SKIP: test-fua.sh
=================
++ mktemp -d
+ sockdir=/tmp/tmp.2UBabllkkp
+ files='fua.img
       fua1.log fua1.pid
       fua2.log fua2.pid
       fua3.log fua3.pid
       fua4.log fua4.pid'
+ rm -f fua.img fua1.log fua1.pid fua2.log fua2.pid fua3.log fua3.pid fua4.log fua4.pid
+ truncate -s 1M fua.img
+ qemu-io -f raw -t none -c flush -c 'w -f -z 0 64k' fua.img
file system may not support O_DIRECT
can't open device fua.img: Could not open 'fua.img': Invalid argument
+ echo './test-fua.sh: missing or broken qemu-io'
./test-fua.sh: missing or broken qemu-io
+ rm fua.img
+ exit 77
+ _run_cleanup_hooks
+ status=77
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo ./test-fua.sh: run cleanup hooks: exit code 77
./test-fua.sh: run cleanup hooks: exit code 77
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 77
SKIP test-fua.sh (exit status: 77)
RPM build errors:
+ exit 1
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.U2FYct (%check)
BUILDSTDERR:     Bad exit status from /var/tmp/rpm-tmp.U2FYct (%check)
Child return code was: 1

Comment 5 Richard W.M. Jones 2019-06-10 08:12:46 UTC
Please attach or link to the complete build log.

Comment 7 Richard W.M. Jones 2019-06-10 08:33:48 UTC
checking for PYTHON... no
checking for PYTHON... no
checking for PYTHON... yes

Comparing this to the source:

https://github.com/libguestfs/nbdkit/blob/46323648adb79f338d25f6ee8806249f017d3e38/configure.ac#L476-L508

indicates that the python-embed.pc files were not found.  I suppose it's called python-<VERSION>-embed
not python-embed-<VERSION>.

Comment 8 Richard W.M. Jones 2019-06-10 08:44:01 UTC
My attempts to build this locally in mock always fail with:

[MIRROR] glusterfs-6.2-1.fc31.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 84 but expected size is: 668104
[FAILED] glusterfs-6.2-1.fc31.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

which seems to be a server side problem.  Anyway I have pushed another patch
blindly which may fix this, in nbdkit-1.13.5-2.fc31.

Comment 9 Miro Hrončok 2019-06-10 10:42:44 UTC
For the server problem, I recommend scrubbing mock dnf cache,  with mock -r ... --scrub=dnf-cache.

Anyway, running another build.


Note You need to log in before you can comment on or make changes to this bug.