Can not reproduce by my side directly by qemu, both test with qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 and qemu-kvm-1.5.3-60.el7.x86_64

cli, boot with 5 queues, 10 vectors and -smp 1 
# /usr/libexec/qemu-kvm -name r7_n -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614 -no-user-config -nodefaults -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/rhel7cp1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,queues=5,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -spice port=5930,disable-ticketing  -device usb-tablet,id=input0  -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -monitor stdio

then inside guest, enable mq:
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

# ethtool -L eth0 combined 5
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	5


guest # uname -r
3.10.0-123.el7.x86_64

Hi, I can not reproduce it, could you have a look my steps and tell me any difference, does this issue 100% reproduce in your enviroment?

Could you have a look comment2 and add your comment?

Best Regards,
Junyi

(In reply to Hu Jianwei from comment #0)
> Description of problem:
> qemu-kvm process crashed after enabling the multi-queue support in one
> network interface in guest OS.
> 
> Version-Release number of selected component (if applicable):
> libvirt-1.1.1-29.el7.x86_64
> qemu-kvm-rhev-1.5.3-60.el7ev.x86_64
> kernel-3.10.0-121.el7.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. [root@localhost ~]# virsh dumpxml r7_n | grep interface -A8
>     <interface type='network'>
>       <mac address='52:54:00:e5:99:99'/>
>       <source network='default_001'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <driver name='vhost' queues='5'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
> ...
> 
> [root@localhost ~]# virsh start r7_n
> Domain r7_n started
> 
> 2. In guest OS:
> kernel version
> kernel-3.10.0-123.el7.x86_64
> 
> [root@oolong ~]# ethtool -l eth0
> Channel parameters for eth0:
> Pre-set maximums:
> RX:		0
> TX:		0
> Other:		0
> Combined:	5
> Current hardware settings:
> RX:		0
> TX:		0
> Other:		0
> Combined:	1
> 
> [root@oolong ~]# ethtool -L eth0 combined 5
> 
> [root@localhost network-scripts]#   <=====lost connection from guest.
> 
> 
> 
> Actual results:
> As shown above, after running "ethtool -L eth0 combined 5", the qemu-kvm
> process crashed.
> 
> 
> [root@localhost ~]# tailf /var/log/libvirt/qemu/r7_n.log 
> 2014-05-08 08:07:24.758+0000: 4875: debug : virFileClose:90 : Closed fd 37
> 2014-05-08 08:07:24.758+0000: 4875: debug : virCommandHandshakeChild:391 :
> Handshake with parent is done
> char device redirected to /dev/pts/4 (label charserial0)
> main_channel_link: add main channel client
> main_channel_handle_parsed: net test: latency 0.144000 ms, bitrate
> 36571428571 bps (34877.232142 Mbps)
> inputs_connect: inputs channel client create
> red_dispatcher_set_cursor_peer: 
> qemu-kvm: could not enable queue
> qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407:
> virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed.
> 2014-05-08 08:09:06.428+0000: shutting down
> 

This generally mean there's a mis configuration of host network. Could you pls show me how the network setup in host? (tap bridge or macvtap)?
> ^C
> 
> Expected results:
> The multi-queue can be enabled on that interface, and the qemu-kvm process
> should keep running, no crash occured.
> 
> 
> Additional info:
> qemu-kvm command line(generated by libvirt):
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name r7_n -S -machine
> pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp
> 1,sockets=1,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614
> -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_n.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
> -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> file=/var/lib/libvirt/images/r7_n.img,if=none,id=drive-virtio-disk0,
> format=raw,cache=none -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> id=virtio-disk0,bootindex=1 -netdev
> tap,fds=25:26:27:28,id=hostnet0,vhost=on,vhostfds=29:30:31:32 -device

Looks like a bug of libvirt?

queues = 5 is specified but only 4 tap file descriptors and vhost file descriptors were pass to qemu?
> virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:
> 99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,
> name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice
> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl
> -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

Could not reproduce with the following qemu-kvm cli:

/usr/libexec/qemu-kvm -netdev tap,vhost=on,id=hn0,fds=3:4:5:6:7,vhostfds=8:9:10:11:12: /home/kvm_autotest_root/images/RHEL-Server-7.0-64-virtio.qcow2 -m 2G -device virtio-net-pci,netdev=hn0,vectors=32,mac=00:00:05:00:00:06,mq=on -enable-kvm -smp 5 -vnc :10

btw, is it a regression?

> This generally mean there's a mis configuration of host network. Could you
> pls show me how the network setup in host? (tap bridge or macvtap)?

The virtual network is like below:

[root@localhost ~]# virsh net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>5de056c8-568f-4a90-8862-0048150aa230</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:b5:c0:46'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

[root@localhost ~]# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400b5c046	yes		virbr0-nic
							vnet0
virbr5		8000.52540025cf91	yes		virbr5-nic
[root@localhost ~]# ifconfig virbr0
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:b5:c0:46  txqueuelen 0  (Ethernet)
        RX packets 964  bytes 114030 (111.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 515  bytes 1106097 (1.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@localhost ~]# ifconfig vnet0
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:fee5:9999  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:e5:99:99  txqueuelen 500  (Ethernet)
        RX packets 90  bytes 9235 (9.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183  bytes 10640 (10.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


> Looks like a bug of libvirt?
> 
> queues = 5 is specified but only 4 tap file descriptors and vhost file
> descriptors were pass to qemu?

Sorry for putting another qemu command line in Additional info part in the bug, libvirt can pass a correct value to qemu(2N+2), please refer to below:

[root@localhost ~]# virsh dumpxml r7_n | grep queue -A4
      <driver name='vhost' queues='6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
[root@localhost ~]# virsh start r7_n
Domain r7_n started

[root@localhost ~]# ps aux | grep qemu-kvm |grep -v grep| sed 's/-device/\n-device/g' | grep "virtio-net-pci"
-device virtio-net-pci,mq=on,vectors=14,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 
[root@localhost ~]#

(In reply to Hu Jianwei from comment #6)
> > This generally mean there's a mis configuration of host network. Could you
> > pls show me how the network setup in host? (tap bridge or macvtap)?
> 
> The virtual network is like below:
> 
> [root@localhost ~]# virsh net-dumpxml default
> <network connections='1'>
>   <name>default</name>
>   <uuid>5de056c8-568f-4a90-8862-0048150aa230</uuid>
>   <forward mode='nat'>
>     <nat>
>       <port start='1024' end='65535'/>
>     </nat>
>   </forward>
>   <bridge name='virbr0' stp='on' delay='0' />
>   <mac address='52:54:00:b5:c0:46'/>
>   <ip address='192.168.122.1' netmask='255.255.255.0'>
>     <dhcp>
>       <range start='192.168.122.2' end='192.168.122.254' />
>     </dhcp>
>   </ip>
> </network>
> 
> [root@localhost ~]# brctl show
> bridge name	bridge id		STP enabled	interfaces
> virbr0		8000.525400b5c046	yes		virbr0-nic
> 							vnet0
> virbr5		8000.52540025cf91	yes		virbr5-nic
> [root@localhost ~]# ifconfig virbr0
> virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
>         ether 52:54:00:b5:c0:46  txqueuelen 0  (Ethernet)
>         RX packets 964  bytes 114030 (111.3 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 515  bytes 1106097 (1.0 MiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> [root@localhost ~]# ifconfig vnet0
> vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::fc54:ff:fee5:9999  prefixlen 64  scopeid 0x20<link>
>         ether fe:54:00:e5:99:99  txqueuelen 500  (Ethernet)
>         RX packets 90  bytes 9235 (9.0 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 183  bytes 10640 (10.3 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> 
> > Looks like a bug of libvirt?
> > 
> > queues = 5 is specified but only 4 tap file descriptors and vhost file
> > descriptors were pass to qemu?
> 
> Sorry for putting another qemu command line in Additional info part in the
> bug, libvirt can pass a correct value to qemu(2N+2), please refer to below:
> 
> [root@localhost ~]# virsh dumpxml r7_n | grep queue -A4
>       <driver name='vhost' queues='6'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
> [root@localhost ~]# virsh start r7_n
> Domain r7_n started
> 
> [root@localhost ~]# ps aux | grep qemu-kvm |grep -v grep| sed
> 's/-device/\n-device/g' | grep "virtio-net-pci"
> -device
> virtio-net-pci,mq=on,vectors=14,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:
> 99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 
> [root@localhost ~]#

Confused, several questions:

- Did you reproduce the issue with queues=5 or queues=6?
- What the full qemu-kvm command line when you reproduce this.
- Is it a regression?

Thanks

> Confused, several questions:
> 
> - Did you reproduce the issue with queues=5 or queues=6?
> - What the full qemu-kvm command line when you reproduce this.
> - Is it a regression?
> 
Q1, yes, I can reproduce it with queues=5 and 6, exactly the reproduced value is from 2 to 8. (the valid value of queues is from 1 to 8.)

Q2, full command line(e.g. queues=8):
/usr/libexec/qemu-kvm -name r7_n -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 5,sockets=5,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_n.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/r7_n.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=25:26:27:28:29:30:31:32,id=hostnet0,vhost=on,vhostfds=33:34:35:36:37:38:39:40 -device virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

Q3, I just tested it using the latest versions, if necessary, I'll do the regression testing for it using old versions from libvirt side.

Thanks.

> - Is it a regression?
> 
> Thanks

Maybe this is not a regression issue.

I tried below combined versions, the bug reprodcued on all, please refer to the following:

1.libvirt-1.1.1-26.el7.x86_64 (Fixed bug 1071888, vectors parameter is 2N+2.)
  qemu-kvm-1.5.3-45.el7.x86_64

cat /var/log/libvirt/qemu/r7_n.log
...
qemu-kvm: could not enable queue
qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed.

2. libvirt-1.1.1-26.el7.x86_64
   qemu-kvm-1.5.3-2.el7.x86_64

cat /var/log/libvirt/qemu/r7_n.log
...
qemu-kvm: could not enable queue
qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:305: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed.
2014-05-09 07:05:46.181+0000: shutting down

3. libvirt-1.1.1-15.el7.x86_64
   qemu-kvm-1.5.3-2.el7.x86_64

cat /var/log/libvirt/qemu/r7_n.log
...
qemu-kvm: could not enable queue
qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:305: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed.

Thanks.

Thanks for the testing

what does you /var/log/audit/audit.log looks like just after the crash? Can you reproduce if you disable the selinux?

Thanks

(In reply to jason wang from comment #10)
> Thanks for the testing
> 
> what does you /var/log/audit/audit.log looks like just after the crash? Can
> you reproduce if you disable the selinux?
> 
> Thanks

[root@sriov2 ~]# getenforce 
Enforcing
[root@sriov2 ~]# virsh dumpxml r7 | grep queue -A4
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
[root@sriov2 ~]# virsh start r7
Domain r7 started

[root@sriov2 ~]# virsh console r7
Connected to domain r7
Escape character is ^]

Red Hat Enterprise Linux Server 7.0 (Maipo)
Kernel 3.10.0-123.el7.x86_64 on an x86_64

oolong login: root
Password: 
Last login: Fri May  9 15:54:20 on ttyS0
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	2
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[root@oolong ~]# ethtool -L eth0 combined 2

[root@sriov2 ~]# tailf /var/log/audit/audit.log 
type=AVC msg=audit(1399622478.790:893): avc:  denied  { attach_queue } for  pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket
type=SYSCALL msg=audit(1399622478.790:893): arch=c000003e syscall=16 success=no exit=-13 a0=18 a1=400454d9 a2=7ff7f1b5e970 a3=0 items=0 ppid=1 pid=7585 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c638,c877 key=(null)
type=ANOM_ABEND msg=audit(1399622478.790:894): auid=4294967295 uid=107 gid=107 ses=4294967295 subj=system_u:system_r:svirt_t:s0:c638,c877 pid=7585 comm="qemu-kvm" reason="memory violation" sig=6
type=ANOM_PROMISCUOUS msg=audit(1399622482.570:895): dev=vnet0 prom=0 old_prom=256 auid=4294967295 uid=107 gid=107 ses=4294967295
type=VIRT_CONTROL msg=audit(1399622482.692:896): pid=2409 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=stop reason=failed vm="r7" uuid=353713ab-40d9-4523-8fc1-26282dfe88e9 vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'

Disabled selinux on host, the bug was disappeared.
[root@sriov2 ~]# setenforce 0
[root@sriov2 ~]# getenforce 
Permissive
[root@sriov2 ~]# virsh start r7
Domain r7 started

[root@sriov2 ~]# virsh console r7
Connected to domain r7
Escape character is ^]

Red Hat Enterprise Linux Server 7.0 (Maipo)
Kernel 3.10.0-123.el7.x86_64 on an x86_64

oolong login: root
Password: 
Last login: Fri May  9 16:01:02 on ttyS0
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	2
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[root@oolong ~]# ethtool -L eth0 combined 2
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	2
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	2

Thanks.

Looks like SELinux prevent qemu to attach the a new tun queue. We need allow this operation otherwise the multiqueue function won't be functional.

type=AVC msg=audit(1399622478.790:893): avc:  denied  { attach_queue } for  pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket

Reset the component to selinux-policy.

Feel free to reassign this bug if I was wrong.

I think this is a libvirt bug. 

tun_socket should be relabeled to svirt_t to have

allow svirt_t self:tun_socket attach_queue;

Reproduced this bug with following components and Jason's script:
# uname -r
3.10.0-123.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.5.3-60.el7_0.2.x86_64
# rpm -qa |grep selinux
selinux-policy-3.12.1-153.el7.noarch
libselinux-utils-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
selinux-policy-targeted-3.12.1-153.el7.noarch
libselinux-2.2.2-6.el7.x86_64

Steps:
1.Change security context of the image to test:
# chcon -u system_u -r system_r -t svirt_t /rhel7-0409.qcow2 

2.Confirm the SElinux is enabled with mode enforcing:
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28


3.Launch qemu via user 'qemu' :

# ps auZ |grep kvm
unconfined_u:system_r:svirt_t:s0-s0:c0.c1023 qemu 4949 103  0.3 2734940 27964 pts/0 Sl+ 15:56   0:03 /usr/libexec/qemu-kvm -netdev tap,vhost=on,id=hn0,fds=3:4:5:6,vhostfds=7:8:9:10 /rhel7-0409.qcow2 -m 2G -device virtio-net-pci,netdev=hn0,vectors=32,mac=00:00:05:00:00:06,mq=on -enable-kvm -smp 4 -vnc :10


4.Inside guest, try to enable mq:
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	4
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

# ethtool -L eth0 combined 4

Result, qemu quit like this:
qemu-kvm: could not enable queue
qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed.

From log:
...
type=AVC msg=audit(1399967917.807:630): avc:  denied  { attach_queue } for  pid=4966 comm="qemu-kvm" scontext=unconfined_u:system_r:svirt_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=tun_socket
....

So according to above, this bug is reproduced.

# chcon -u system_u -r system_r -t svirt_t /rhel7-0409.qcow2  

Is the wrong thing to do svirt_t is a process type not a file type. If you did this on an enforcing system it would be blocked.

https://danwalsh.livejournal.com/54803.html

Miroslav was reporting that libvirt should tell SELinux the label on the tun_socket should be svirt_t before creating it. This would allow us to make sure only the tun_socket intended for the VM can be used by the VM.

If we allow svirt_t to use virtd_t tun_socket, then it can use any tun_socket that got leaked to it.  Including ones intended for other VMs

(In reply to Miroslav Grepl from comment #19)
> I think this is a libvirt bug. 
> 
> tun_socket should be relabeled to svirt_t to have
> 
> allow svirt_t self:tun_socket attach_queue;

But the /dev/net/tun has tun_tap_device_t label by default. So what you are suggesting is to behave differently in case of multiqueue (MQ) and in case of a single queue? That is, if the TAP is MQ label the /dev/net/tun with svirt_t, if it's a single queue label it with tun_tap_device_t? It's not gonna fly really because if libvirt changes the label on the /dev/net/tun other process (possibly other libvirtd) may hop in and open the device with label it should not have. Even locks within libvirt won't help as the problem is at interprocess level.

Well you get

type=AVC msg=audit(1399622478.790:893): avc:  denied  { attach_queue } for  pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket

I've changed the libvirt code, so that tun FD is labeled as svirt_t but I still see AVC denial:

kernel: type=1400 audit(1408450864.055:32): avc:  denied  { relabelto } for  pid=32311 comm="lt-libvirtd" name="tun" dev="devtmpfs" ino=15650 scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:svirt_t:s0:c258,c379 tclass=chr_file

Aparently, libvirt can't relabel chr_file.

Well you have libvirtd running as init_t.

ls -Z /usr/sbin/libvirtd

and

ps -eZ |grep libvirt

Okay, I've proposed patch upstream:

https://www.redhat.com/archives/libvir-list/2014-August/msg00801.html

I've just pushed the patch upstream:

commit cf976d9dcf4e592261b14f03572bb519531ebbce
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Aug 13 16:08:03 2014 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Aug 20 09:42:24 2014 +0200

    qemu: Label all TAP FDs
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1095636
    
    When starting up the domain the domain's NICs are allocated. As of
    1f24f682 (v1.0.6) we are able to use multiqueue feature on virtio
    NICs. It breaks network processing into multiple queues which can be
    processed in parallel by different host CPUs. The queues are, however,
    created by opening /dev/net/tun several times. Unfortunately, only the
    first FD in the row is labelled so when turning the multiqueue feature
    on in the guest, qemu will get AVC denial. Make sure we label all the
    FDs needed.
    
    Moreover, the default label of /dev/net/tun doesn't allow
    attaching a queue:
    
        type=AVC msg=audit(1399622478.790:893): avc:  denied  { attach_queue }
        for  pid=7585 comm="qemu-kvm"
        scontext=system_u:system_r:svirt_t:s0:c638,c877
        tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023
        tclass=tun_socket
    
    And as suggested by SELinux maintainers, the tun FD should be labeled
    as svirt_t. Therefore, we don't need to adjust any range (as done
    previously by Guannan in ae368ebf) rather set the seclabel of the
    domain directly.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v1.2.7-179-gcf976d9

Going through the upstream git log, I've noticed this commit which we may need to backport too:

commit a4431931393aeb1ac5893f121151fa3df4fde612
Author:     Martin Kletzander <mkletzan>
AuthorDate: Mon Sep 1 15:27:00 2014 +0200
Commit:     Martin Kletzander <mkletzan>
CommitDate: Mon Sep 1 15:36:23 2014 +0200

    selinux: properly label tap FDs with imagelabel
    
    The cleanup in commit cf976d9d used secdef->label to label the tap
    FDs, but that is not possible since it's process-only label (svirt_t)
    and not a object label (e.g. svirt_image_t).  Starting a domain failed
    with EPERM, but simply using secdef->imagelabel instead of
    secdef->label fixes it.
    
    Signed-off-by: Martin Kletzander <mkletzan>

I could reproduce this issue with libvirt-1.1.1-29.el7.x86_64


Steps to Reproduce:
1. [root@localhost ~]# virsh dumpxml rhel7 | grep interface -A8
    <interface type='network'>
      <mac address='52:54:00:e5:99:11'/>
      <source network='default_001'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='5'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
...

[root@localhost ~]# virsh start rhel7
Domain rhel7 started

2. In guest OS:
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[root@oolong ~]# ethtool -L eth0 combined 5

[root@localhost /]#   <=====lost connection from guest.

After running "ethtool -L eth0 combined 5", the qemu-kvm process crashed.






With libvirt-1.2.8-1.el7.x86_64, this issue still exists:

Here are steps:
1. [root@localhost ~]# virsh dumpxml rhel7 | grep interface -A8
    <interface type='network'>
      <mac address='52:54:00:e5:99:11'/>
      <source network='default_001'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='5'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
...

[root@localhost ~]# virsh start rhel7
Domain rhel7 started

2. In guest OS:
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[root@oolong ~]# ethtool -L eth0 combined 5

[root@localhost /]#   <=====lost connection from guest.

After running "ethtool -L eth0 combined 5", the qemu-kvm process crashed.


Could you help to check or confirm above?

(In reply to Michal Privoznik from comment #34)
> Going through the upstream git log, I've noticed this commit which we may
> need to backport too:
> 
> commit a4431931393aeb1ac5893f121151fa3df4fde612
> Author:     Martin Kletzander <mkletzan>
> AuthorDate: Mon Sep 1 15:27:00 2014 +0200
> Commit:     Martin Kletzander <mkletzan>
> CommitDate: Mon Sep 1 15:36:23 2014 +0200
> 
>     selinux: properly label tap FDs with imagelabel
>     
>     The cleanup in commit cf976d9d used secdef->label to label the tap
>     FDs, but that is not possible since it's process-only label (svirt_t)
>     and not a object label (e.g. svirt_image_t).  Starting a domain failed
>     with EPERM, but simply using secdef->imagelabel instead of
>     secdef->label fixes it.
>     
>     Signed-off-by: Martin Kletzander <mkletzan>

Move back to ASSIGNED.

Not sure if it is possible but could we change virtd_t to virtd_tun_t if FD is passed and allow svirt_t to access virtd_tun_t tun_socket?

system_u:system_r:virtd_tun_t:s0

So we would have

 type=AVC msg=audit(1399622478.790:893): avc:  denied  { attach_queue }
        for  pid=7585 comm="qemu-kvm"
        scontext=system_u:system_r:svirt_t:s0:c638,c877
        tcontext=system_u:system_r:virtd_tun_t:s0-s0:c0.c1023
        tclass=tun_socket

Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2014-September/msg01163.html

Miroslav, do you want me to clone this to selinux component so you can push the fix?

So I've just pushed the patch upstream:

commit ba7468dbb13f552a9177d01ea8bad155f9877bc3
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Sep 18 15:17:29 2014 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Sep 18 19:02:47 2014 +0200

    virSecuritySELinuxSetTapFDLabel: Temporarily revert to old behavior
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1141879
    
    A long time ago I've implemented support for so called multiqueue
    net.  The idea was to let guest network traffic be processed by
    multiple host CPUs and thus increasing performance. However, this
    behavior is enabled by QEMU via special ioctl() iterated over the
    all tap FDs passed in by libvirt. Unfortunately, SELinux comes in
    and disallows the ioctl() call because the /dev/net/tun has label
    system_u:object_r:tun_tap_device_t:s0 and 'attach_queue' ioctl()
    is not allowed on tun_tap_device_t type. So after discussion with
    a SELinux developer we've decided that the FDs passed to the QEMU
    should be labelled with svirt_t type and SELinux policy will
    allow the ioctl(). Therefore I've made a patch
    (cf976d9dcf4e592261b14f03572) that does exactly this. The patch
    was fixed then by a4431931393aeb1ac5893f121151fa3df4fde612 and
    b635b7a1af0e64754016d758376f382470bc11e7. However, things are not
    that easy - even though the API to label FD is called
    (fsetfilecon_raw) the underlying file is labelled too! So
    effectively we are mangling /dev/net/tun label. Yes, that broke
    dozen of other application from openvpn, or boxes, to qemu
    running other domains.
    
    The best solution would be if SELinux provides a way to label an
    FD only, which could be then labeled when passed to the qemu.
    However that's a long path to go and we should fix this
    regression AQAP. So I went to talk to the SELinux developer again
    and we agreed on temporary solution that:
    
    1) All the three patches are reverted
    2) SELinux temporarily allows 'attach_queue' on the
    tun_tap_device_t
    
    Signed-off-by: Michal Privoznik <mprivozn>

v1.2.8-204-gba7468d

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-October/msg00134.html

Hi Michal,

I still found the issue with libvirt-1.2.8-5.el7 we talked before:

1.prepare two guest with NIC
# virsh dumpxml test34|grep interface -2
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00c:158'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>


# virsh dumpxml test3|grep interface -2
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='00:22:44:66:88:aa'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <smartcard mode='passthrough' type='spicevmc'>
      <address type='ccid' controller='0' slot='0'/>

2.start 2 guest
# virsh start test3
Domain test3 started
# virsh start test34
Domain test34 started

3.seliunx will report a lot of avc
time->Tue Oct 14 11:58:12 2014
type=SYSCALL msg=audit(1413259092.550:943913): arch=c000003e syscall=0 success=no exit=-13 a0=1a a1=7f50d74a16bc a2=11000 a3=3 items=0 ppid=1 pid=727 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c621,c731 key=(null)
type=AVC msg=audit(1413259092.550:943913): avc:  denied  { read } for  pid=727 comm="qemu-kvm" path="/dev/net/tun" dev="devtmpfs" ino=8321 scontext=system_u:system_r:svirt_t:s0:c621,c731 tcontext=system_u:object_r:tun_tap_device_t:s0:c569,c714 tclass=chr_file
----
time->Tue Oct 14 11:58:12 2014
type=SYSCALL msg=audit(1413259092.550:943914): arch=c000003e syscall=0 success=no exit=-13 a0=1a a1=7f50d74a16bc a2=11000 a3=3 items=0 ppid=1 pid=727 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c621,c731 key=(null)
^C

4.# ll -Z /dev/net/tun
crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0:c279,c536 /dev/net/tun


This issue just like bug 1147057, and the result just like bug 1147057 comment 12 said:

one vm gets access to the network and all others don't.

Would you please tell me if you will fix this issue in this bug? or open another bug for it? 


Thanks,
Luyao Huang

(In reply to Luyao Huang from comment #41)
> Hi Michal,
> 
> I still found the issue with libvirt-1.2.8-5.el7 we talked before:

> 
> one vm gets access to the network and all others don't.
> 
> Would you please tell me if you will fix this issue in this bug? or open
> another bug for it? 

It seems like we need yet another upstream patch:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-October/msg00375.html


Michal

*** Bug 1149664 has been marked as a duplicate of this bug. ***

I can not reproduce it using libvirt-1.2.8-6.el7.x86_64

Version:
libvirt-1.2.8-6.el7.x86_64
qemu-kvm-rhev-2.1.2-6.el7.x86_64
qemu-kvm-1.5.3-77.el7.x86_64

[root@localhost ~]# getenforce 
Enforcing

[root@localhost ~]# ll -Z /dev/net/tun 
crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun

[root@localhost ~]# virsh dumpxml r7 | grep "/interface" -B6
    <interface type='network'>
      <mac address='52:54:00:e5:99:99'/>
      <source network='default'/>
      <model type='virtio'/>
      <driver name='vhost' queues='5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

Before starting the domain, in another terminal, check the output of audit.
[root@localhost ~]# tailf /var/log/audit/audit.log | grep denied

[root@localhost ~]# virsh start r7
Domain r7 started

[root@localhost ~]# virsh console r7
Connected to domain r7
Escape character is ^]

Red Hat Enterprise Linux Server 7.0 (Maipo)
Kernel 3.10.0-123.el7.x86_64 on an x86_64

oolong login: root
Password: 
Last login: Thu Nov  6 15:44:41 on ttyS0
[root@oolong ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.58  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::5054:ff:fee5:9999  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:e5:99:99  txqueuelen 1000  (Ethernet)
        RX packets 37  bytes 3278 (3.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 42  bytes 5711 (5.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[root@oolong ~]# ethtool -L eth0 combined 5
[root@oolong ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	5
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	5

[root@localhost ~]# ll -Z /dev/net/tun 
crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun
[root@localhost ~]# virsh destroy r7
Domain r7 destroyed

[root@localhost ~]# ll -Z /dev/net/tun 
crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun

During the doamin life cycle, no any avc denied message printed.
[root@localhost ~]# tailf /var/log/audit/audit.log | grep denie
        <=== nothing

I found the patch series is temporary, could I verify it according to above results?

If we have better solution, need we re-open and re-test it again? Thanks.

(In reply to Hu Jianwei from comment #45)
> I can not reproduce it using libvirt-1.2.8-6.el7.x86_64
> 
> Version:
> libvirt-1.2.8-6.el7.x86_64
> qemu-kvm-rhev-2.1.2-6.el7.x86_64
> qemu-kvm-1.5.3-77.el7.x86_64
> 

> I found the patch series is temporary, could I verify it according to above
> results?
> 
> If we have better solution, need we re-open and re-test it again? Thanks.

This is enough to verify the bug. The temporary means, until the time SELinux provides a better API to label resources, the patch fixes the issue. Once SELinux does provide API and libvirt learns it, there shouldn't be any change in behaviour. So I'd say move this bug to VERIFIED and trouble with new approach when it gets real (which may take ages).

Thanks for Michal's reply.

According to comment 45 and 46, we can get expected results, so change to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html