Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1430287 - kernel XFS driver hits 2000 sec timeout during XFS log recovery in virt-v2v conversion of RHEL 7.2 guest
Summary: kernel XFS driver hits 2000 sec timeout during XFS log recovery in virt-v2v c...
Keywords:
Status: CLOSED DUPLICATE of bug 1425700
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.4
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-08 09:49 UTC by kuwei@redhat.com
Modified: 2017-03-10 12:15 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-10 12:15:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
v2v.log (108.97 KB, text/plain)
2017-03-08 09:49 UTC, kuwei@redhat.com
no flags Details
v2v-windows (131.24 KB, text/plain)
2017-03-08 10:44 UTC, kuwei@redhat.com
no flags Details

Description kuwei@redhat.com 2017-03-08 09:49:22 UTC
Created attachment 1261163 [details]
v2v.log

Description of problem:
Convert a guest from vmware to kvm by virt-v2v can't be finished in 2 hours

Version-Release number of selected component (if applicable):
virt-v2v-1.36.1-1.el7.x86_64
libguestfs-1.36.1-1.el7.x86_64
libvirt-3.1.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-5.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1:Convert a rhel7.2 guest from vmware to kvm.
# #  virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64 --password-file /tmp/passwd -on test1
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64
[   1.2] Creating an overlay to protect the source from being modified
[   1.8] Initializing the target -o libvirt -os default
[   1.8] Opening the overlay
[  35.0] Inspecting the overlay
^c
 
Result:  the above conversion  can't be finished in 2 hours.
For more information you can see the attached  v2v.log.

2.Convert a windows guest from vmware to kvm.
# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008-x86_64 --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008-x86_64
[   1.4] Creating an overlay to protect the source from being modified
[   2.4] Initializing the target -o libvirt -os default
[   2.4] Opening the overlay
[  24.4] Inspecting the overlay
[ 109.0] Checking for sufficient free disk space in the guest
[ 109.0] Estimating space required on target for each disk
[ 109.0] Converting Windows Server (R) 2008 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.


^C
Result:  the above conversion can't be finished in 3 hours.

Actual results:
As above description

Expected results:
It should be convert a guest from vmware to kvm successfully in 1 hours.


Addition info:
Guest can be convert from vmware to kvm with below builds:
virt-v2v-1.36.1-1.el7.x86_64
libguestfs-1.36.1-1.el7.x86_64
libvirt-3.1.0-1.el7.x86_64
qemu-kvm-1.5.3-132.el7.x86_64

Comment 2 Pino Toscano 2017-03-08 10:14:58 UTC
(In reply to kuwei from comment #0)
> 1:Convert a rhel7.2 guest from vmware to kvm.
> # #  virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1
> esx5.5-rhel7.2-x86_64 --password-file /tmp/passwd -on test1
> [   0.0] Opening the source -i libvirt -ic
> vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64
> [   1.2] Creating an overlay to protect the source from being modified
> [   1.8] Initializing the target -o libvirt -os default
> [   1.8] Opening the overlay
> [  35.0] Inspecting the overlay
> ^c

In this case, there I/O errors when reading the disks, and most probably that causes the kernel to throw error traces. My guess is that it is an issue either in QEMU (the curl/https driver), or in the kernel.

Can you please paste the result of `rpm -qa | grep kernel | sort -V`, on the host where virt-v2v was run?
I see you are using qemu-kvm-rhev, could you please try with the plain qemu-kvm?

> 2.Convert a windows guest from vmware to kvm.

Can you please provide the logs for this case?

Comment 3 kuwei@redhat.com 2017-03-08 10:42:16 UTC
(In reply to Pino Toscano from comment #2)

> In this case, there I/O errors when reading the disks, and most probably
> that causes the kernel to throw error traces. My guess is that it is an
> issue either in QEMU (the curl/https driver), or in the kernel.
> 
> Can you please paste the result of `rpm -qa | grep kernel | sort -V`, on the
> host where virt-v2v was run?
# rpm -qa | grep kernel | sort -V
kernel-3.10.0-567.el7.x86_64
kernel-devel-3.10.0-567.el7.x86_64
kernel-headers-3.10.0-567.el7.x86_64
kernel-tools-3.10.0-567.el7.x86_64
kernel-tools-libs-3.10.0-567.el7.x86_64

> I see you are using qemu-kvm-rhev, could you please try with the plain
> qemu-kvm?

Using qemu-kvm can convert guest from vmware to kvm successfully,you can see my addition info comment 0. But we can't start the guest on qemu-kvm packages,it is a bug https://bugzilla.redhat.com/show_bug.cgi?id=1430214.I think the bug maybe some help.

> > 2.Convert a windows guest from vmware to kvm.
> 
> Can you please provide the logs for this case?

Comment 4 kuwei@redhat.com 2017-03-08 10:44:28 UTC
Created attachment 1261197 [details]
v2v-windows

Comment 5 Pino Toscano 2017-03-08 10:59:27 UTC
(In reply to kuwei from comment #3)
> > I see you are using qemu-kvm-rhev, could you please try with the plain
> > qemu-kvm?
> 
> Using qemu-kvm can convert guest from vmware to kvm successfully,you can see
> my addition info comment 0.

Oh you are right, sorry.

> But we can't start the guest on qemu-kvm
> packages,it is a bug https://bugzilla.redhat.com/show_bug.cgi?id=1430214.I
> think the bug maybe some help.

This is a libvirt 3.1.0 regression, see bug 1430258.

You should be able to work around it by explicitly using the direct backend:
export LIBGUESTFS_BACKEND=direct

(Remember to drop this workaround once the libvirt bug is fixed.)

> > > 2.Convert a windows guest from vmware to kvm.
> > 
> > Can you please provide the logs for this case?

Hmm... does it work if by chance you install libguestfs-rescue together with virt-v2v? It looks like lsof might needed by the base library, because of a local workaround.

Comment 6 Richard W.M. Jones 2017-03-08 11:42:42 UTC
The timeout happens after 2000 seconds, which happens to be the
timeout that we set in the qemu curl driver:

https://github.com/libguestfs/libguestfs/blob/0b793550d430cd8135c177f74dec13341c731b6d/v2v/vCenter.ml#L224

In qemu commit

http://git.qemu-project.org/?p=qemu.git;a=commit;h=796a060bc0fab40953997976a2e30d9d6235bc7b

I changed the curl driver so it's now supposed to print the underlying
error if there is one.  Since that didn't happen in this case, I
conclude that the curl driver really is hitting the timeout, and
there is not any other error.

While we could increase the timeout again (it was previously
increased from 600 - see bug 1146007), it's not clear to me
that we should be hitting the timeout at all.  That would
indicate that the XFS driver in the kernel isn't reading or
writing any data over the connection for 2000 seconds, but is
instead either hung or doing some kind of processing.

Comment 9 Richard W.M. Jones 2017-03-08 20:57:34 UTC
I obtained the source image from the source server.

However I was able to open the disk locally just fine, both with
Fedora 26 and RHEL 7.4-preview, so I don't really know what's
going on here.

On RHEL 7.4 I am using:

kernel-3.10.0-567.el7.x86_64
10:qemu-kvm-rhev-2.8.0-5.el7.x86_64

I also tried qemu-kvm-1.5.3-126.el7.x86_64

Comment 16 kuwei@redhat.com 2017-03-10 09:38:52 UTC
Hi, rjones, 
i try some different qemu-kvm-rhev versions and virt-v2v to convert guest from vmware to kvm, below list is the result.

Scenario 1: 
when i using   qemu-kvm-rhev-2.8.0-5.el7.x86_64,   
virt-v2v-1.36.1-1.el7.x86_64, --- failed 
virt-v2v-1.36.2-1.el7.x86_64 , ---failed
virt-v2v-1.32.7-3.el7.x86_64  --- failed.

Scenario 2:
Using    qemu-kvm-rhev-2.8.0-6.el7.x86_64,  
virt-v2v-1.36.1-1.el7.x86_64,--- passed  
virt-v2v-1.36.2-1.el7.x86_64 --- passed.

Scenario 3: 
Using qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64,
virt-v2v-1.36.2-1.el7.x86_64 , ---passed


So, I think qemu-kvm-rhev-2.8.0-5.el7.x86_64 maybe has some issues.

Comment 17 Richard W.M. Jones 2017-03-10 09:50:33 UTC
qemu-kvm-rhev 2.8.0-6 fixes bug 1425700, something to do with
the handling of virtio-scsi queues.  It is plausible that this
change may have fixed this bug.

I am going to test the -6 package myself using the reproducer
in comment 10.  Let's keep this bug open to see if the bug
appears again.

Comment 18 Richard W.M. Jones 2017-03-10 12:15:18 UTC
Can confirm that this appears to be fixed by
qemu-kvm-rhev-2.8.0-6.el7.

On that basis I will mark it as a duplicate of bug 1425700.

*** This bug has been marked as a duplicate of bug 1425700 ***


Note You need to log in before you can comment on or make changes to this bug.