Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1709211 - Suppress error message printed by nbdkit-vddk-plugin when VDDK extents support is broken
Summary: Suppress error message printed by nbdkit-vddk-plugin when VDDK extents suppor...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: nbdkit
Version: 8.1
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard: V2V
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2019-05-13 08:07 UTC by mxie@redhat.com
Modified: 2021-05-25 06:41 UTC (History)
8 users (show)

Fixed In Version: nbdkit-1.24.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-25 06:41:20 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
vddk6.7-nfc-error-rhel8.1.log (2.36 MB, text/plain)
2019-05-13 08:07 UTC, mxie@redhat.com 		no flags Details
esx5.5-vddk6.7.3.log (347.01 KB, text/plain)
2019-11-19 09:23 UTC, mxie@redhat.com 		no flags Details
esx5.5-vddk6.7.3-nbdkit-1.24.0-1.log (1.51 MB, text/plain)
2021-01-19 15:29 UTC, mxie@redhat.com 		no flags Details
View All

Description mxie@redhat.com 2019-05-13 08:07:14 UTC 
Created attachment 1567813 [details]
vddk6.7-nfc-error-rhel8.1.log

Description of problem:
There is NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train

Version-Release number of selected component (if applicable):
virt-v2v-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64
libguestfs-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64
libvirt-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64
qemu-kvm-4.0.0-0.module+el8.1.0+3169+3c501422.x86_64
nbdkit-1.12.1-1.module+el8.1.0+3164+94495c71.x86_64
kernel-4.18.0-83.el8.x86_64
VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server
# virt-v2v -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64 
[   0.1] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.2] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  40.8] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  70.1] Inspecting the overlay
[  77.8] Checking for sufficient free disk space in the guest
[  77.8] Estimating space required on target for each disk
[  77.8] Converting Windows 8.1 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  87.0] Mapping filesystem data to avoid copying unused and blank areas
[  87.7] Closing the overlay
[  87.9] Assigning disks to buses
[  87.9] Checking if the guest needs BIOS or UEFI to boot
[  87.9] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[  88.2] Copying disk 1/1 to /tmp/v2v.j8tJoW/c980b640-cb59-47f7-a0fe-90312d33deae/images/c8eecdd8-8906-4cfd-8cd2-f018515e868b/6de49971-6544-43d4-8acf-3dd13063f9d5 (raw)
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
^C  (73.69/100%)

2.Check virt-v2v-input-vmware man page about NFC error 
2.1 #man virt-v2v-inpurt-vmware
 VDDK: ESXi NFC service memory limits
       In the verbose log you may see errors like:

        nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg:
        received NFC error 5 from server: Failed to allocate the
        requested 2097176 bytes

       This seems especially common when there are multiple parallel
       connections open to the VMware server.

       These can be caused by resource limits set on the VMware server.  You
       can increase the limit for the NFC service by editing
       /etc/vmware/hostd/config.xml and adjusting the "<maxMemory>" setting:

        <nfcsvc>
          <path>libnfcsvc.so</path>
          <enabled>true</enabled>
          <maxMemory>50331648</maxMemory>
          <maxStreamMemory>10485760</maxStreamMemory>
        </nfcsvc>

       and restarting the "hostd" service:

        # /etc/init.d/hostd restart

       For more information see https://bugzilla.redhat.com/1614276.

3.Set esxi5.5 host according to step2
3.1 ~ # cat  /etc/vmware/hostd/config.xml  |grep nfcsvc -A 4
      <nfcsvc>
         <path>libnfcsvc.so</path>
         <enabled>true</enabled>
         <maxMemory>50331648</maxMemory>
         <maxStreamMemory>10485760</maxStreamMemory>
      </nfcsvc>  
3.2 ~ #  /etc/init.d/hostd restart
watchdog-hostd: Terminating watchdog process with PID 34023
hostd stopped.
Ramdisk 'hostd' with estimated size of 413MB already exists
hostd started.

4.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server again
# virt-v2v -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64
[   0.1] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.0] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  91.2] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[ 174.5] Inspecting the overlay
[ 181.2] Checking for sufficient free disk space in the guest
[ 181.2] Estimating space required on target for each disk
[ 181.2] Converting Windows 8.1 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[ 188.9] Mapping filesystem data to avoid copying unused and blank areas
[ 189.6] Closing the overlay
[ 189.8] Assigning disks to buses
[ 189.8] Checking if the guest needs BIOS or UEFI to boot
[ 189.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[ 190.0] Copying disk 1/1 to /tmp/v2v.TgtI0U/c980b640-cb59-47f7-a0fe-90312d33deae/images/c78df6f2-9272-4af9-bc62-7fabc909f740/2ca1cb92-6559-4af3-83dc-ea951d2673f5 (raw)
^C  (0.00/100%)

Actual results:
As above description

Expected results:
Suppress NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train

Additional info:
1.Can't reproduce the problem when convert a guest from ESXi5.5 host via vddk6.5 on rhel8.1 v2v conversion server
2.Can't reproduce the problem when convert a guest from ESXi6.0/6.5/6.7 host via vddk6.7 on rhel8.1 v2v conversion server
Comment 1 Richard W.M. Jones 2019-05-13 08:15:04 UTC 
https://github.com/libguestfs/nbdkit/blob/master/plugins/vddk/vddk.c#L711

emits an error message while testing if VDDK extents support is broken on the server.
We could suppress this error by temporarily disabling the error handler.
Comment 2 mxie@redhat.com 2019-11-19 09:21:00 UTC 
Virt-v2v can't convert guest from esx5.5 with vddk VMware-vix-disklib-6.7.3-14389676.x86_64 tar.gz now

Reproduce the bug with below builds:
virt-v2v-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
libguestfs-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64
libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64

Steps to reproduce:
1. Convert a guest from esx5.5 with vddk 6.7.3 by virt-v2v
#  virt-v2v  -i libvirt  -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib  -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd 
[   0.5] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  12.4] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE
[  26.1] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX
nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
......
......
[  44.8] Inspecting the overlay
.......
.......
.......
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
virt-v2v: error: inspection could not detect the source guest (or physical 
machine).

Assuming that you are running virt-v2v/virt-p2v on a source which is 
supported (and not, for example, a blank disk), then this should not 
happen.

No root device found in this operating system image.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
nbdkit: vddk[2]: error: SSL: Unknown SSL Error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: SSL: Unknown SSL Error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR

Result1:
   A lot of NFC ERROR shows during converting and the conversion is failed due to "nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection"


2.Convert a guest from esx5.5 with vddk 6.7.0 (VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz) by virt-v2v
#  virt-v2v  -i libvirt  -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib  -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd 
[   0.5] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.5] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  41.5] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  71.4] Inspecting the overlay
[  89.2] Checking for sufficient free disk space in the guest
[  89.2] Estimating space required on target for each disk
[  89.2] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el6’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[ 225.7] Mapping filesystem data to avoid copying unused and blank areas
[ 226.5] Closing the overlay
[ 226.6] Assigning disks to buses
[ 226.6] Checking if the guest needs BIOS or UEFI to boot
[ 226.6] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 228.0] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.TpBYUG/nbdkit0.sock", "file.export": "/" } (raw)
^C  (0.00/100%)

Result2:
   V2V conversion can be finished although there is NFS error showing during converting.
Comment 3 mxie@redhat.com 2019-11-19 09:23:54 UTC 
Created attachment 1637578 [details]
esx5.5-vddk6.7.3.log
Comment 7 Richard W.M. Jones 2020-02-13 09:14:26 UTC 
It's nbdkit.  In this function:

https://github.com/libguestfs/nbdkit/blob/4dd31a48a9db61c0a76d7f1c876822b594947057/plugins/vddk/vddk.c#L670

If the server is ESXi 5.5 (which doesn't support extents) then the function
we run to test for extents support will print the following harmless cosmetic
errors:

nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server

This bug is about requesting a way to suppress these errors from being visible.

In addition there's a further bug that latest VDDK doesn't support
access to ESXi 5.5 at all.  It's unclear if this was deliberate or VMware
simply stopped testing it (since ESXi 5.5 is long out of support).  In any
case, that's a VDDK bug that we cannot do anything about, and therefore there
is no separate bug to track that.
Comment 8 John Ferlan 2020-04-20 13:37:13 UTC 
Eric - I leave this for you to handle as I see some upstream changes you posted earlier this month related to vddk support.
Comment 9 Richard W.M. Jones 2020-04-20 16:12:59 UTC 
I don't believe this is fixed yet.  See comment 1, comment 7.
Comment 10 Richard W.M. Jones 2020-05-07 11:24:25 UTC 
https://www.redhat.com/archives/libguestfs/2020-May/msg00023.html

TBH I think the cure here may be worse than the disease and we
may need to close this bug as WONTFIX.  There's an ongoing internal
discussion about whether to keep supporting VMware ESXi 5.5 which
is the only version that this bug affects.
Comment 11 Richard W.M. Jones 2020-05-07 16:11:15 UTC 
https://github.com/libguestfs/nbdkit/commit/47bd6b9b95f9e64489d8b7c84da477a06208b883

I'm going to assign this to AV 8.3.0 so it has plenty of time to "mature"
[in the sense of cheese!] upstream.  It should be a trivial backport on
top of the rebase.
Comment 16 mxie@redhat.com 2021-01-19 14:56:04 UTC 
Verify the bug with builds:
virt-v2v-1.42.0-7.module+el8.4.0+9398+f376ac33.x86_64
libguestfs-1.44.0-1.module+el8.4.0+9398+f376ac33.x86_64
libvirt-libs-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64
qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.15-0.el8.noarch


Steps:
1.Use v2v to convert a guest from ESXi5.5 host to rhv4.4 via vddk6.7.3

# virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk6.7.3 -io  vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd  -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -os nfs_data -n ovirtmgmt  esx5.5-rhel7.9-x86_64
[   0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.9-x86_64 -it vddk  -io vddk-libdir=/home/vddk6.7.3 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   2.4] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE
[   5.6] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX
nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: SSL: Unknown SSL Error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
[  17.5] Inspecting the overlay
[  37.3] Checking for sufficient free disk space in the guest
[  37.3] Estimating space required on target for each disk
[  37.3] Converting Red Hat Enterprise Linux Server 7.9 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 156.7] Mapping filesystem data to avoid copying unused and blank areas
[ 157.4] Closing the overlay
[ 157.6] Assigning disks to buses
[ 157.6] Checking if the guest needs BIOS or UEFI to boot
[ 157.6] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 158.9] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.dnMe63/nbdkit4.sock", "file.export": "/" } (qcow2)
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX
nbdkit: vddk[3]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[3]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[3]: error: SSL: Unknown SSL Error
nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[3]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
    (100.00/100%)
[ 738.8] Creating output metadata
[ 740.7] Finishing off


2.Use v2v to convert a guest from ESXi5.5 host to rhv4.4 via vddk7.0

# virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd  -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -os nfs_data -n ovirtmgmt  esx5.5-win2019-x86_64
[   0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   2.1] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR]NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE
[   5.7] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR]NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_IO_EX
nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR]NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
[  18.3] Inspecting the overlay
[  22.5] Checking for sufficient free disk space in the guest
[  22.5] Estimating space required on target for each disk
[  22.5] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  32.1] Mapping filesystem data to avoid copying unused and blank areas
[  32.5] Closing the overlay
[  32.8] Assigning disks to buses
[  32.8] Checking if the guest needs BIOS or UEFI to boot
[  32.8] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  34.1] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.0D8Ayl/nbdkit4.sock", "file.export": "/" } (qcow2)
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_IO_EX
nbdkit: vddk[3]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe
nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[3]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe
nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[3]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
    (100.00/100%)
[ 804.9] Creating output metadata
[ 806.7] Finishing off


Additional info:

1.There is no NFC error when convert a guest from ESXi5.5 host via vddk6.7.0

# virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk6.7.0 -io  vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd  -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -os nfs_data -n ovirtmgmt  esx5.5-win2019-x86_64
[   0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk  -io vddk-libdir=/home/vddk6.7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   2.1] Creating an overlay to protect the source from being modified
[   5.7] Opening the overlay
[  12.9] Inspecting the overlay
[  17.6] Checking for sufficient free disk space in the guest
[  17.6] Estimating space required on target for each disk
[  17.6] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  27.0] Mapping filesystem data to avoid copying unused and blank areas
[  27.4] Closing the overlay
[  27.7] Assigning disks to buses
[  27.7] Checking if the guest needs BIOS or UEFI to boot
[  27.7] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  29.0] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.kAEcsD/nbdkit4.sock", "file.export": "/" } (qcow2)
^C  (3.01/100%)


2. There is no NFC error when convert a guest from ESXi5.5 host via vddk6.5

# virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk6.5 -io  vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd  -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -os nfs_data -n ovirtmgmt  esx5.5-win2019-x86_64
[   0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk  -io vddk-libdir=/home/vddk6.5 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   2.2] Creating an overlay to protect the source from being modified
[   5.3] Opening the overlay
[  12.2] Inspecting the overlay
[  18.2] Checking for sufficient free disk space in the guest
[  18.2] Estimating space required on target for each disk
[  18.2] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  28.3] Mapping filesystem data to avoid copying unused and blank areas
[  28.8] Closing the overlay
[  29.1] Assigning disks to buses
[  29.1] Checking if the guest needs BIOS or UEFI to boot
[  29.1] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  30.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.6Kbpxt/nbdkit4.sock", "file.export": "/" } (qcow2)
^C  (38.22/100%)


Hi rjones,

    Please check step1 and step2, seems the bug hasn't been fixed yet because NFC error info will still show if convert guest from ESXi5.5 host with vddk>=6.7.3, but good news is v2v can convert guest from ESXi5.5 with with vddk>=6.7.3 now (the problem of comment got fixed)
Comment 17 mxie@redhat.com 2021-01-19 15:29:06 UTC 
Created attachment 1748745 [details]
esx5.5-vddk6.7.3-nbdkit-1.24.0-1.log
Comment 18 Richard W.M. Jones 2021-01-19 15:30:52 UTC 
I think what's happening is we have successfully suppressed the
original error ("Illegal message during fssrvr session").  In that
limited sense the bug is fixed.

However VDDK 6.7.3 and 7.0 have other problems with the ESXi 5.5 server,
although they are just warnings.  These are essentially new issues,
but they don't affect the conversion from finishing successfully.

And coincidentally VDDK 6.7.3 / 7.0 conversions have been fixed
to ESXi 5.5.  They is most likely because of other changes that
we have made upstream (in the rebase) and are nothing to do with this
particular bug fix.

I'd also like to add that VMware ESXi 5.5 support has ended (in 2018!)

Let me ask Martin Tessun if we should continue testing against ESXi 5.5.
Comment 20 mxie@redhat.com 2021-01-20 06:33:02 UTC 
According to comment18，v2v has successfully suppressed the original error "Illegal message during fssrvr session", in this sense, the bug has been fixed. Besides, we got confirmation from PM Martin by mail, for ESXi 5.5, v2v will drop testing and support, so move the bug from ON_QA to VERIFIED.
Comment 22 errata-xmlrpc 2021-05-25 06:41:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098
Note You need to log in before you can comment on or make changes to this bug.