Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1632220 - nbdkit fails to build: TLS handshake failed: The TLS connection was non-properly terminated.
Summary: nbdkit fails to build: TLS handshake failed: The TLS connection was non-prope...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nbdkit
Version: 7.6
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1632789
TreeView+ depends on / blocked
 
Reported: 2018-09-24 11:48 UTC by mxie@redhat.com
Modified: 2019-08-06 12:02 UTC (History)
10 users (show)

Fixed In Version: nbdkit-1.2.6-1.el7.2
Doc Type: Bug Fix
Doc Text:
Previously, the nbdkit package was configured with an incorrect priority for the TLS protocol. As a consequence, TLS connections connections to the NBD server did not work. This update fixes the TLS priority in nbdkit, which ensures proper functionality of TLS connections.
Clone Of:
: 1632231 1632789 (view as bug list)
Environment:
Last Closed: 2019-08-06 12:02:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
rpmbuild-src-nbdkit.log (173.08 KB, text/plain)
2018-09-24 11:48 UTC, mxie@redhat.com 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2015 0 None None None 2019-08-06 12:02:17 UTC

Description mxie@redhat.com 2018-09-24 11:48:54 UTC 
Created attachment 1486379 [details]
rpmbuild-src-nbdkit.log

Description of problem:
Failed to rebuild ndbkit from src RPM package

Version-Release number of selected component (if applicable):
libvirt-4.5.0-10.el7.x86_64
qemu-kvm-rhev-2.12.0-18.el7.x86_64
rpm-build-4.11.3-35.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Download nbdkit src.rpm package to local server and copy to /home
2.log into server with regular user and create .rpmmacros file containing:

$ cat .rpmmacros 
%_topdir  %(echo $HOME)/rpmbuild
%_smp_mflags -j5

3.Resolve the dependence problem before rebuilding,then do below to build, but the result is failed

$rpmbuild --rebuild nbdkit-1.2.6-1.el7.src.rpm 

Actual results:
As above description

Expected results:


Additional info:
Comment 2 Richard W.M. Jones 2018-09-24 12:00:39 UTC 
I can reproduce this with
qemu-kvm-rhev-2.12.0-8.el7.x86_64
nbdkit-1.2.6-1.el7.src.rpm
Comment 3 Richard W.M. Jones 2018-09-24 12:01:38 UTC 
BTW I could *also* reproduce the TPS failures too ...
Comment 4 Richard W.M. Jones 2018-09-24 12:14:31 UTC 
TPS failures are addressed in separate bug 1632231.  In this bug
I will concentrate only on the TLS failure.
Comment 5 Richard W.M. Jones 2018-09-24 12:33:09 UTC 
The actual error is from nbdkit:

  nbdkit: example1[1]: error: failed to set TLS session priority to @NBDKIT,SYSTEM: The request is invalid.

(https://github.com/libguestfs/nbdkit/blob/c1a91988e466b3e19168a9d6ec55e8ae1eae254c/src/crypto.c#L472 )

When compiling nbdkit for RHEL we use:

  ./configure --with-tls-priority=@NBDKIT,SYSTEM [...]

As explained in the nbdkit manual (nbdkit-tls(1) in later versions):

   Choice of TLS algorithms
       TLS has a bewildering choice of algorithms that can be used.  To enable
       you to choose a default set of algorithms, there is a configure setting
       "--with-tls-priority".  This defaults to "NORMAL" which, to quote the
       GnuTLS documentation:

           ""NORMAL" means all "secure" ciphersuites.  The 256-bit ciphers are
           included as a fallback only.  The ciphers are sorted by security
           margin."

       You could also set the TLS priority so that it can be configured from a
       file at runtime:

        ./configure --with-tls-priority=@SYSTEM

       means use the policy from /etc/crypto-policies/config.

        ./configure --with-tls-priority=@NBDKIT,SYSTEM

       means use the policy from /etc/crypto-policies/local.d/nbdkit.config
       and fall back to /etc/crypto-policies/config if the first file does not
       exist.

       More information can be found in gnutls_priority_init(3).

I believe from reading the RHEL 7 gnutls_priority_init(3) man page that this
old version of gnutls doesn't support the @... syntax.

What's more worrying is why this didn't fail in brew.  In the build.log I can
see that the test gets skipped for some reason:

SKIP: test-tls.sh
Comment 6 Richard W.M. Jones 2018-09-24 13:18:48 UTC 
Nominating this bug for RHEL 7.6 z-stream since it breaks TLS support.

We don't require TLS support for virt-v2v so it's not urgent enough
to stop RHEL 7.6 GA.
Comment 7 Richard W.M. Jones 2018-09-24 13:44:11 UTC 
As far as I can tell this version of GnuTLS does not support the @...
syntax for priorities.  It was added in
https://gitlab.com/gnutls/gnutls/commit/6b6d9dd44e056d12dccbb543122e96d6e4df8489
which is present in gnutls >= 3.5.1.  RHEL 7 has an older version so
my suggestion is that we do this:

  -%configure --disable-static --with-tls-priority=@NBDKIT,SYSTEM \
  +%configure --disable-static --with-tls-priority=NORMAL \

(Note this is consistent with what libvirt.spec does on RHEL 7.6)

The second issue is skipping the test in brew.  The test requires the
'ss' program which is present in RHEL 7 in the 'iproute' package, so we
need to add:

  +BuildRequires: iproute
Comment 8 Richard W.M. Jones 2018-09-24 13:54:10 UTC 
(In reply to Richard W.M. Jones from comment #7)
> The second issue is skipping the test in brew.  The test requires the
> 'ss' program which is present in RHEL 7 in the 'iproute' package, so we
> need to add:
> 
>   +BuildRequires: iproute

Sorry this part is wrong as we already have:

  BuildRequires:  /usr/sbin/ss

In fact the problem was that /usr/bin/qemu-img (ie. qemu 1.5.3) does
not support the --object option, which unfortunately means the
test cannot run.  I don't believe it's possible to pull in qemu-img-rhev
into the buildroot.
Comment 11 mxie@redhat.com 2019-04-16 08:13:48 UTC 
Verify the bug with below builds:
nbdkit-1.8.0-1.el7.src.rpm
libguestfs-devel-1.40.2-3.el7.x86_64
libvirt-4.5.0-12.el7.x86_64
qemu-kvm-rhev-2.12.0-26.el7.x86_64
gnutls-3.3.29-9.el7_6.x86_64

Steps:
1.Download nbdkit src.rpm package to local server and copy to /home

2.log into server with regular user and create .rpmmacros file containing:
$ cat .rpmmacros 
%_topdir  %(echo $HOME)/rpmbuild
%_smp_mflags -j5

3.Resolve the dependence problem before rebuilding,then build nbdkit package from src package 
$rpmbuild --rebuild nbdkit-1.8.0-1.el7.src.rpm 
....
PASS: test-help.sh
PASS: test-version.sh
PASS: test-dump-config.sh
PASS: test-help-plugin.sh
PASS: test-version-plugin.sh
PASS: test-version-filter.sh
PASS: test-dump-plugin.sh
PASS: test-start.sh
PASS: test-single.sh
PASS: test-captive.sh
PASS: test-random-sock.sh
PASS: test-tls.sh
SKIP: test-tls-psk.sh
PASS: test-ip.sh
PASS: test-socket-activation
PASS: test-foreground.sh
PASS: test-debug-flags.sh
PASS: test-exit-with-parent
PASS: test-parallel-file.sh
PASS: test-parallel-nbd.sh
PASS: test-data-7E.sh
SKIP: test-data-base64.sh
PASS: test-data-file.sh
PASS: test-data-raw.sh
PASS: test-floppy.sh
PASS: test-iso.sh
SKIP: test-memory-largest.sh
PASS: test-memory-largest-for-qemu.sh
PASS: test-partitioning1.sh
PASS: test-partitioning4.sh
PASS: test-partitioning2.sh
PASS: test-partitioning3.sh
PASS: test-pattern.sh
SKIP: test-pattern-largest.sh
PASS: test-pattern-largest-for-qemu.sh
PASS: test-vddk.sh
PASS: test-zero.sh
PASS: test-python-exception.sh
PASS: test-shebang-python.sh
PASS: test-layers.sh
PASS: test-layers
PASS: test-blocksize.sh
PASS: test-cache.sh
PASS: test-cow.sh
PASS: test-fua.sh
PASS: test-log.sh
PASS: test-nozero.sh
PASS: test-offset2.sh
PASS: test-truncate1.sh
PASS: test-truncate2.sh
PASS: test-truncate3.sh
make[4]: Entering directory `/home/mm/rpmbuild/BUILD/nbdkit-1.8.0/tests'
make[4]: Nothing to be done for `all'.
make[4]: Leaving directory `/home/mm/rpmbuild/BUILD/nbdkit-1.8.0/tests'
============================================================================
Testsuite summary for nbdkit 1.8.0
============================================================================
# TOTAL: 51
# PASS:  47
# SKIP:  4
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
...

4.Find below nbdkit packages in path /home/mm/rpmbuild/RPMS/x86_64
$ ls /home/mm/rpmbuild/RPMS/x86_64/
nbdkit-1.8.0-1.el7.x86_64.rpm
nbdkit-basic-plugins-1.8.0-1.el7.x86_64.rpm
nbdkit-debuginfo-1.8.0-1.el7.x86_64.rpm
nbdkit-devel-1.8.0-1.el7.x86_64.rpm
nbdkit-example-plugins-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-python2-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-python-common-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-vddk-1.8.0-1.el7.x86_64.rpm

5.Install the nbdkit packages in path /home/mm/rpmbuild/RPMS/x86_64 to replace the installed nbdkit package and verify the function of nbdkit by virt-v2v, and the conversion can be finished successfully
virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt esx6.7-win2019-x86_64 -oo rhv-cluster=nfs



Also verify the bug with below builds:
nbdkit-1.8.0-1.el7.src.rpm
libguestfs-devel-1.40.2-3.el7.x86_64
libvirt-4.5.0-12.el7.x86_64
qemu-kvm-1.5.3-164.el7.x86_64
gnutls-3.3.29-9.el7_6.x86_64


Steps:
1.Remove related qemu-kvm-rhev packages and install related qemu-kvm packages, restart libvirtd service successfully

2.Delete the nbdkit packages in /home/mm/rpmbuild/RPMS/x86_64/ and build nbdkit package from src package 
$rpmbuild --rebuild nbdkit-1.8.0-1.el7.src.rpm 
....
PASS: test-help.sh
PASS: test-version.sh
PASS: test-dump-config.sh
PASS: test-help-plugin.sh
PASS: test-version-plugin.sh
PASS: test-version-filter.sh
PASS: test-dump-plugin.sh
PASS: test-start.sh
SKIP: test-single.sh
PASS: test-captive.sh
PASS: test-random-sock.sh
SKIP: test-tls.sh
SKIP: test-tls-psk.sh
SKIP: test-ip.sh
PASS: test-socket-activation
PASS: test-foreground.sh
PASS: test-debug-flags.sh
PASS: test-exit-with-parent
PASS: test-parallel-file.sh
PASS: test-parallel-nbd.sh
PASS: test-data-7E.sh
SKIP: test-data-base64.sh
PASS: test-data-file.sh
PASS: test-data-raw.sh
PASS: test-floppy.sh
PASS: test-iso.sh
SKIP: test-memory-largest.sh
PASS: test-memory-largest-for-qemu.sh
SKIP: test-partitioning1.sh
SKIP: test-partitioning4.sh
PASS: test-partitioning2.sh
PASS: test-partitioning3.sh
PASS: test-pattern.sh
SKIP: test-pattern-largest.sh
PASS: test-pattern-largest-for-qemu.sh
PASS: test-vddk.sh
SKIP: test-zero.sh
PASS: test-python-exception.sh
PASS: test-shebang-python.sh
PASS: test-layers.sh
PASS: test-layers
SKIP: test-blocksize.sh
PASS: test-cache.sh
PASS: test-cow.sh
PASS: test-fua.sh
PASS: test-log.sh
SKIP: test-nozero.sh
PASS: test-offset2.sh
PASS: test-truncate1.sh
PASS: test-truncate2.sh
SKIP: test-truncate3.sh
make[4]: Entering directory `/home/mm/rpmbuild/BUILD/nbdkit-1.8.0/tests'
make[4]: Nothing to be done for `all'.
make[4]: Leaving directory `/home/mm/rpmbuild/BUILD/nbdkit-1.8.0/tests'
============================================================================
Testsuite summary for nbdkit 1.8.0
============================================================================
# TOTAL: 51
# PASS:  38
# SKIP:  13
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

....

3.Find below nbdkit packages in path /home/mm/rpmbuild/RPMS/x86_64
$ ls /home/mm/rpmbuild/RPMS/x86_64
nbdkit-1.8.0-1.el7.x86_64.rpm
nbdkit-basic-plugins-1.8.0-1.el7.x86_64.rpm
nbdkit-debuginfo-1.8.0-1.el7.x86_64.rpm
nbdkit-devel-1.8.0-1.el7.x86_64.rpm
nbdkit-example-plugins-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-python2-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-python-common-1.8.0-1.el7.x86_64.rpm
nbdkit-plugin-vddk-1.8.0-1.el7.x86_64.rpm

4.Install the nbdkit packages in path /home/mm/rpmbuild/RPMS/x86_64 to replace the installed nbdkit package and verify the function of nbdkit by virt-v2v, and the conversion can be finished successfully
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  esx6.7-win2019-x86_64


Hi rjones,
   
   When build nbdkit from src package by qemu-kvm-rhev, test-tls.sh is passed but there are 4 tests are skipped, when build nbdkit from src package by qemu-kvm, test-tls.sh is skipped and the other 12 tests also are skipped, is it normal?
Comment 12 Pino Toscano 2019-04-16 08:23:59 UTC 
(In reply to mxie from comment #11)
>    When build nbdkit from src package by qemu-kvm-rhev, test-tls.sh is
> passed but there are 4 tests are skipped, when build nbdkit from src package
> by qemu-kvm, test-tls.sh is skipped and the other 12 tests also are skipped,
> is it normal?

Please make sure that the version of qemu-kvm(-rhev) is the same as qemu-kvm(-rhev).
At least some of the tests use features in qemu-io not available in qemu 1.5.
Comment 13 Richard W.M. Jones 2019-04-16 09:21:20 UTC 
The test is skipped under various circumstances as can be seen here:

https://github.com/libguestfs/nbdkit/blob/4444c7f5367209c7f235b3d0b4c0959e62df03f9/tests/test-tls.sh#L38

However I can't tell why it skipped.  To see why it skipped, have a look
at the file tests/test-tls.sh.log in the build directory.  It should contain
one of those messages.
Comment 14 mxie@redhat.com 2019-04-24 05:44:25 UTC 
Hi rjones,

   Got test-tls.sh.log in /home/mm/rpmbuild/BUILD/nbdkit-1.8.0/tests via executing "$rpmbuild -bb /home/mm/rpmbuild/SPECS/nbdkit.spec",pls check!

$ cat test-tls.sh.log 
+ ss --version
ss utility, iproute2-ss170501
+ command -v qemu-img
+ qemu-img --help
+ grep -- --object
+ echo './test-tls.sh: '\''qemu-img'\'' command does not have the --object option'
./test-tls.sh: 'qemu-img' command does not have the --object option
+ exit 77
+ _run_cleanup_hooks
+ status=77
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo ./test-tls.sh: run cleanup hooks: exit code 77
./test-tls.sh: run cleanup hooks: exit code 77
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 77
SKIP test-tls.sh (exit status: 77)
Comment 15 Richard W.M. Jones 2019-04-24 11:51:17 UTC 
So the skip reason is:

  ./test-tls.sh: 'qemu-img' command does not have the --object option

The test for this is:

  https://github.com/libguestfs/nbdkit/blob/4444c7f5367209c7f235b3d0b4c0959e62df03f9/tests/test-tls.sh#L47

As you can see we require qemu-img new enough to support the --object option
because that option is used later to configure TLS certificates:

  https://github.com/libguestfs/nbdkit/blob/4444c7f5367209c7f235b3d0b4c0959e62df03f9/tests/test-tls.sh#L81

I suspect what's happening here is you're using qemu-img instead of qemu-img-rhev.
Comment 16 mxie@redhat.com 2019-04-24 13:05:53 UTC 
According to comment15, test-tls.sh is skipped when build nbdkit from src package with qemu-kvm is a expected result. As test-tls.sh is passed when build nbdkit from src package with qemu-kvm-rhev, move the bug from ON_QA to VERIFIED
Comment 18 errata-xmlrpc 2019-08-06 12:02:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2015
Note You need to log in before you can comment on or make changes to this bug.