Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1598440
Summary: | virt-v2v will hang at opening the overlay during conversion with libvirt-4.5.0-1 | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | mxie <mxie> | ||||||||||
Component: | libvirt | Assignee: | Daniel Berrangé <berrange> | ||||||||||
Status: | CLOSED ERRATA | QA Contact: | mxie <mxie> | ||||||||||
Severity: | urgent | Docs Contact: | |||||||||||
Priority: | urgent | ||||||||||||
Version: | 7.6 | CC: | berrange, jdenemar, juzhou, mzhan, ptoscano, rjones, tzheng, xiaodwan, xuzhang | ||||||||||
Target Milestone: | rc | Keywords: | Regression | ||||||||||
Target Release: | --- | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | libvirt-4.5.0-2.el7 | Doc Type: | If docs needed, set a value | ||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2018-10-30 09:57:31 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 910269 | ||||||||||||
Attachments: |
|
Add virt-v2v developer Richard for further check. I've noticed something similar happening in Fedora Rawhide at the moment. I didn't have time to look into it yet. Try running: libguestfs-test-tool It looks like a libvirt, qemu or kernel bug of some kind. (In reply to Richard W.M. Jones from comment #4) > I've noticed something similar happening in Fedora Rawhide > at the moment. I didn't have time to look into it yet. > > Try running: > libguestfs-test-tool > > It looks like a libvirt, qemu or kernel bug of some kind. libguestfs-test-tool hangs at the same part when trying to launch libvirt guest. # libguestfs-test-tool ************************************************************ * IMPORTANT NOTICE * * When reporting bugs, include the COMPLETE, UNEDITED * output below in your bug report. * ************************************************************ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin XDG_RUNTIME_DIR=/run/user/0 SELinux: Enforcing guestfs_get_append: (null) guestfs_get_autosync: 1 guestfs_get_backend: libvirt guestfs_get_backend_settings: [] guestfs_get_cachedir: /var/tmp guestfs_get_hv: /usr/libexec/qemu-kvm guestfs_get_memsize: 500 guestfs_get_network: 0 guestfs_get_path: /usr/lib64/guestfs guestfs_get_pgroup: 0 guestfs_get_program: libguestfs-test-tool guestfs_get_recovery_proc: 1 guestfs_get_smp: 1 guestfs_get_sockdir: /tmp guestfs_get_tmpdir: /tmp guestfs_get_trace: 0 guestfs_get_verbose: 1 host_cpu: x86_64 Launching appliance, timeout set to 600 seconds. libguestfs: launch: program=libguestfs-test-tool libguestfs: launch: version=1.38.2rhel=7,release=6.el7,libvirt libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=libvirt libguestfs: launch: tmpdir=/tmp/libguestfsg4bVYg libguestfs: launch: umask=0022 libguestfs: launch: euid=0 libguestfs: libvirt version = 4005000 (4.5.0) libguestfs: guest random name = guestfs-d6zxdsa4pzpkvi7p libguestfs: connect to libvirt libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0 libguestfs: successfully opened libvirt handle: conn = 0x55a56abf7100 libguestfs: qemu version (reported by libvirt) = 2012000 (2.12.0) libguestfs: get libvirt capabilities libguestfs: parsing capabilities XML libguestfs: build appliance libguestfs: begin building supermin appliance libguestfs: run supermin libguestfs: command: run: /usr/bin/supermin5 libguestfs: command: run: \ --build libguestfs: command: run: \ --verbose libguestfs: command: run: \ --if-newer libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock libguestfs: command: run: \ --copy-kernel libguestfs: command: run: \ -f ext2 libguestfs: command: run: \ --host-cpu x86_64 libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d supermin: version: 5.1.19 supermin: rpm: detected RPM version 4.11 supermin: package handler: fedora/rpm supermin: acquiring lock on /var/tmp/.guestfs-0/lock supermin: build: /usr/lib64/guestfs/supermin.d supermin: reading the supermin appliance supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-winsupport.tar.gz type gzip base image (tar) supermin: mapping package names to installed packages supermin: resolving full list of package dependencies supermin: build: 192 packages, including dependencies supermin: build: 31629 files supermin: build: 7572 files, after matching excludefiles supermin: build: 7579 files, after adding hostfiles supermin: build: 7573 files, after removing unreadable files supermin: build: 7597 files, after munging supermin: kernel: looking for kernel using environment variables ... supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ... supermin: kernel: looking for kernels in /boot ... supermin: kernel: kernel version of /boot/vmlinuz-3.10.0-907.el7.x86_64 = 3.10.0-907.el7.x86_64 (from content) supermin: kernel: picked modules path /lib/modules/3.10.0-907.el7.x86_64 supermin: kernel: kernel version of /boot/vmlinuz-0-rescue-039312dd01ca49a38c625180d4dc0610 = 3.10.0-907.el7.x86_64 (from content) supermin: kernel: picked modules path /lib/modules/3.10.0-907.el7.x86_64 supermin: kernel: picked vmlinuz /boot/vmlinuz-3.10.0-907.el7.x86_64 supermin: kernel: kernel_version 3.10.0-907.el7.x86_64 supermin: kernel: modpath /lib/modules/3.10.0-907.el7.x86_64 supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.ueoqupei/root' supermin: ext2: populating from base image supermin: ext2: copying files from host filesystem supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.ueoqupei/initrd' supermin: ext2: wrote 31 modules to minimal initrd supermin: renaming /var/tmp/.guestfs-0/appliance.d.ueoqupei to /var/tmp/.guestfs-0/appliance.d libguestfs: finished building supermin appliance libguestfs: command: run: qemu-img libguestfs: command: run: \ create libguestfs: command: run: \ -f qcow2 libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw libguestfs: command: run: \ /tmp/libguestfsg4bVYg/overlay2.qcow2 Formatting '/tmp/libguestfsg4bVYg/overlay2.qcow2', fmt=qcow2 size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16 libguestfs: create libvirt XML libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n <name>guestfs-d6zxdsa4pzpkvi7p</name>\n <memory unit="MiB">500</memory>\n <currentMemory unit="MiB">500</currentMemory>\n <cpu mode="host-passthrough">\n <model fallback="allow"/>\n </cpu>\n <vcpu>1</vcpu>\n <clock offset="utc">\n <timer name="rtc" tickpolicy="catchup"/>\n <timer name="pit" tickpolicy="delay"/>\n <timer name="hpet" present="no"/>\n </clock>\n <os>\n <type>hvm</type>\n <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm</cmdline>\n <bios useserial="yes"/>\n </os>\n <on_reboot>destroy</on_reboot>\n <devices>\n <rng model="virtio">\n <backend model="random">/dev/urandom</backend>\n </rng>\n <controller type="scsi" index="0" model="virtio-scsi"/>\n <disk device="disk" type="file">\n <source file="/tmp/libguestfsg4bVYg/scratch1.img"/>\n <target dev="sda" bus="scsi"/>\n <driver name="qemu" type="raw" cache="unsafe"/>\n <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n </disk>\n <disk type="file" device="disk">\n <source file="/tmp/libguestfsg4bVYg/overlay2.qcow2"/>\n <target dev="sdb" bus="scsi"/>\n <driver name="qemu" type="qcow2" cache="unsafe"/>\n <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n </disk>\n <serial type="unix">\n <source mode="connect" path="/tmp/libguestfscd6xDK/console.sock"/>\n <target port="0"/>\n </serial>\n <channel type="unix">\n <source mode="connect" path="/tmp/libguestfscd6xDK/guestfsd.sock"/>\n <target type="virtio" name="org.libguestfs.channel.0"/>\n </channel>\n <controller type="usb" model="none"/>\n <memballoon model="none"/>\n </devices>\n <qemu:commandline>\n <qemu:env name="TMPDIR" value="/var/tmp"/>\n </qemu:commandline>\n</domain>\n libguestfs: command: run: ls libguestfs: command: run: \ -a libguestfs: command: run: \ -l libguestfs: command: run: \ -R libguestfs: command: run: \ -Z /var/tmp/.guestfs-0 libguestfs: /var/tmp/.guestfs-0: libguestfs: drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 . libguestfs: drwxrwxrwt. root root system_u:object_r:tmp_t:s0 .. libguestfs: drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 appliance.d libguestfs: -rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 lock libguestfs: libguestfs: /var/tmp/.guestfs-0/appliance.d: libguestfs: drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 . libguestfs: drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 .. libguestfs: -rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 initrd libguestfs: -rwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 kernel libguestfs: -rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 root libguestfs: command: run: ls libguestfs: command: run: \ -a libguestfs: command: run: \ -l libguestfs: command: run: \ -Z /tmp/libguestfscd6xDK libguestfs: drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 . libguestfs: drwxrwxrwt. root root system_u:object_r:tmp_t:s0 .. libguestfs: srw-rw----. root qemu unconfined_u:object_r:user_tmp_t:s0 console.sock libguestfs: srw-rw----. root qemu unconfined_u:object_r:user_tmp_t:s0 guestfsd.sock libguestfs: launch libvirt guest I can reproduce this just by updating libvirt to 4.5.0-1.el7 (leaving everything else unchanged) so it is a libvirt bug. It looks like the same thing I was seeing in Rawhide. Created attachment 1456906 [details]
libvirt.log
libvirt.log (client side) during hang.
Created attachment 1456907 [details]
libvirtd.log
libvirtd (server side) log during the same hang.
I can't reproduce it with F28 + libvirt 4.5.0, but not tried rawhide yet. Could you provide the /var/log/libvirt/qemu/$GUESTNAME.log as it might have something useful. Also test if selinux permissive helps or not - if so is there an AVC Created attachment 1456924 [details]
guestfs-3ah8l6e1k43fmk2c.log
qemu log for guest
SELinux is set to Enforcing. Setting it to Permissive does not appear to help, so I suppose it's not an SELinux problem. Downgrading to qemu-kvm-rhev-2.10.0-21.el7_5.4 fixes the problem, indicating that the common factor might be libvirt 4.5.0 + qemu 2.12. I screwed up the chardev FD passing code and passed in a UNIX listener socket even for client mode chardevs, and libguestfs uses client mode. This patch ought to fix it https://www.redhat.com/archives/libvir-list/2018-July/msg00341.html Verify the bug with builds: virt-v2v-1.38.2-6.el7.x86_64 libguestfs-1.38.2-6.el7.x86_64 libvirt-4.5.0-2.el7.x86_64 qemu-kvm-rhev-2.12.0-7.el7.x86_64 virtio-win-1.9.4-2.el7.noarch Steps: 1.Convert a guest from vmware to rhv4.2's data domain by virt-v2v # virt-v2v -ic vpx://vsphere.local%5cAdministrator.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2016-x86_64 -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data -op /tmp/rhvpasswd -oo rhv-cafile=/home/ca.pem -oo rhv-direct -of raw --password-file /tmp/passwd -b ovirtmgmt [ 0.3] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2016-x86_64 [ 2.0] Creating an overlay to protect the source from being modified [ 3.0] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd -os nfs_data [ 4.6] Opening the overlay [ 25.6] Inspecting the overlay [ 136.2] Checking for sufficient free disk space in the guest [ 136.2] Estimating space required on target for each disk [ 136.2] Converting Windows Server 2016 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 166.6] Mapping filesystem data to avoid copying unused and blank areas [ 168.7] Closing the overlay [ 169.5] Checking if the guest needs BIOS or UEFI to boot [ 169.5] Assigning disks to buses [ 169.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.dW7QbF/nbdkit1.sock", "file.export": "/" } (raw) (100.00/100%) [2118.2] Creating output metadata [2139.4] Finishing off 2.Power on guest on rhv4.2 and checkpoints of guest are passed 3.Convert a guest from Xen to rhv4.2's export domain by virt-v2v # virt-v2v -ic xen+ssh://root.3.21 xen-hvm-rhel7.5-x86_64 -o rhv -os 10.66.144.40:/home/nfs_export -of qcow2 -b ovirtmgmt [ 0.0] Opening the source -i libvirt -ic xen+ssh://root.3.21 xen-hvm-rhel7.5-x86_64 [ 0.6] Creating an overlay to protect the source from being modified [ 5.3] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export [ 8.3] Opening the overlay [ 66.9] Inspecting the overlay [ 117.5] Checking for sufficient free disk space in the guest [ 117.5] Estimating space required on target for each disk [ 117.5] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 367.3] Mapping filesystem data to avoid copying unused and blank areas [ 370.6] Closing the overlay [ 373.6] Checking if the guest needs BIOS or UEFI to boot [ 373.6] Assigning disks to buses [ 373.6] Copying disk 1/1 to /tmp/v2v.T0IcSJ/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/fc7ce3c2-a4e9-45bf-96a1-bda41ef05f24/a47c2e18-d198-48dd-94e2-e3f7341139a4 (qcow2) (100.00/100%) [ 724.6] Creating output metadata [ 724.8] Finishing off 4.Import guest from export domain to data domain on rhv4.2, power on guest and checkpoints of guest are passed Result: Virt-v2v conversion could be finished successfully with libvirt-4.5.0-2, so move the bug from ON_QA to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3113 |
Created attachment 1456767 [details] v2v-hang-opening-overlay.log Description of problem: virt-v2v will hang at opening the overlay during conversion with libvirt-4.5.0-1 Version-Release number of selected component (if applicable): virt-v2v-1.38.2-6.el7.x86_64 libguestfs-1.38.2-6.el7.x86_64 libvirt-4.5.0-1.el7.x86_64 qemu-kvm-rhev-2.12.0-7.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Convert a guest by virt-v2v but the conversion stay at opening overlay for several hours # virt-v2v avocado-vt-vm1 -o null [ 0.0] Opening the source -i libvirt avocado-vt-vm1 [ 0.0] Creating an overlay to protect the source from being modified [ 0.1] Initializing the target -o null [ 0.1] Opening the overlay ^C Actual results: As above description Expected results: Virt-v2v conversion could be finished successfully Additional info: 1.Can't reproduce the problem if downgrade libvirt to 4.4.0-1, so it is a regression bug on libvirt libvirt-4.4.0-1.el7.x86_64 virt-v2v-1.38.2-6.el7.x86_64 libguestfs-1.38.2-6.el7.x86_64 qemu-kvm-rhev-2.12.0-7.el7.x86_64