Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1144490

Summary: QEMU crashes on source host during migration
Product: [Fedora] Fedora Reporter: Daniel BerrangĂ© <berrange>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: amit.shah, berrange, cfergeau, crobinso, dwmw2, gkurz, itamar, mst, pbonzini, rjones, scottt.tw, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-2.1.2-4.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-11 07:02:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Berrangé 2014-09-19 14:12:14 UTC
Description of problem:
Testing qemu 2.1.1 migration with openstack I see the source QEMU crashing during migration. This is a regression from 2.1.0 version

2014-09-19 12:46:05.760+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-00000006 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu Conroe -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ef67b53c-92ea-4bd3-a22a-b27052fc6f2a -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2014.2.0,serial=f09d05e7-68d8-4cab-affb-69a368983b1b,uuid=ef67b53c-92ea-4bd3-a22a-b27052fc6f2a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:4e:01:19,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/26 (label charserial1)
qemu-system-x86_64: /home/berrange/src/fedora/qemu/qemu-2.1.1/hw/net/virtio-net.c:1348: virtio_net_save: Assertion `!n->vhost_started' failed.
2014-09-19 12:51:20.423+0000: shutting down


Version-Release number of selected component (if applicable):
qemu-2.1.1-1.fc21.x86_64

How reproducible:
Always (with openstack VM config at least)

Steps to Reproduce:
1. Boot guest with a virtio NIC
2. Trigger migration
3.

Actual results:
Source QEMU crashes with an assert

Expected results:
Migration works !

Additional info:

Comment 1 Cole Robinson 2014-09-19 18:14:54 UTC
I know there are some patches that are already accumulating for qemu 2.1.2 that I think are migration related. mst, does that assertion `!n->vhost_started' sound familiar?

Comment 2 Michael S. Tsirkin 2014-09-21 08:44:12 UTC
yes, a known bug, will be fixed in 2.1.2

Comment 3 Fedora Update System 2014-09-21 19:46:45 UTC
qemu-2.1.1-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.1-2.fc21

Comment 4 Fedora Update System 2014-09-24 15:45:37 UTC
Package qemu-2.1.1-2.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qemu-2.1.1-2.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-11180/qemu-2.1.1-2.fc21
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2014-09-26 19:07:24 UTC
qemu-2.1.2-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.2-2.fc21

Comment 6 Fedora Update System 2014-09-29 04:03:07 UTC
qemu-2.1.2-2.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Greg Kurz 2014-10-06 15:11:16 UTC
Because of that BZ, the latest QEMU packages for fedora21 are broken with pseries booting from a virtio disk.

When looking at the source RPM, I have spotted the following patch file:

0006-virtio-pci-enable-bus-master-for-old-guests.patch

From a9862ff2c205aa289b62abcb7ddd03c7630f5c7a Mon Sep 17 00:00:00 2001
From: "Michael S. Tsirkin" <mst>
Date: Thu, 11 Sep 2014 18:45:33 +0200
Subject: [PATCH] virtio-pci: enable bus master for old guests

commit cc943c36faa192cd4b32af8fe5edb31894017d35
    pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't
enable bus mastering for virtio PCI devices. For the same reason,
rhel6.[56] ppc64 guests cannot boot on a virtio-blk disk anymore.

Old guests forgot to enable bus mastering, enable it automatically on
DRIVER (guests use some devices before DRIVER_OK).

Reported-by: Greg Kurz <gkurz.ibm.com>
Reviewed-by: Greg Kurz <gkurz.ibm.com>
Tested-by: Greg Kurz <gkurz.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst>
(cherry picked from commit e43c0b2ea5574efb0bedebf6a7d05916eefeba52)

This commit has been reviewed as toxic and reverted from the QEMU master branch upstream:

http://git.qemu-project.org/?p=qemu.git;a=commit;h=45363e46aeebfc99753389649eac7c7fc22bfe52

If fedora21 is supposed to act as an hypervisor for pseries guest, you should consider reverting
this commit as well.

Comment 8 Greg Kurz 2014-10-06 15:25:34 UTC
(In reply to Greg Kurz from comment #7)
> Because of that BZ, the latest QEMU packages for fedora21 are broken with
> pseries booting from a virtio disk.
> 
> When looking at the source RPM, I have spotted the following patch file:
> 
> 0006-virtio-pci-enable-bus-master-for-old-guests.patch
> 

Oops my bad, wrong patch file... I meant:

0007-virtio-pci-fix-migration-for-pci-bus-master.patch

From 2c8d6826ebaf1bf846edd213f8fe8f95c0c300cb Mon Sep 17 00:00:00 2001
From: "Michael S. Tsirkin" <mst>
Date: Thu, 11 Sep 2014 18:34:29 +0300
Subject: [PATCH] virtio-pci: fix migration for pci bus master

...

Cc: Jason Wang <jasowang>
Cc: Greg Kurz <gkurz.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst>
(cherry picked from commit 4d43d3f3c8147ade184df9a1e9e82826edd39e19)

Comment 9 Fedora Update System 2014-10-06 18:44:12 UTC
qemu-2.1.2-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.2-4.fc21

Comment 10 Fedora Update System 2014-10-08 19:10:17 UTC
Package qemu-2.1.2-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qemu-2.1.2-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-12404/qemu-2.1.2-4.fc21
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-10-11 07:02:41 UTC
qemu-2.1.2-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.