Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1095636
Summary: | SELinux prevent qemu from attaching tuntap queues | |||
---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Hu Jianwei <jiahu> | |
Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 7.0 | CC: | dwalsh, dyuan, hhuang, honzhang, jasowang, jiahu, juzhang, knoel, lhuang, mgrepl, mprivozn, mst, mzhan, pmoore, qiguo, rbalakri, tdosek, virt-bugs, virt-maint, ydu, zhwang | |
Target Milestone: | rc | Keywords: | Upstream, ZStream | |
Target Release: | --- | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | libvirt-1.2.8-6.el7 | Doc Type: | Bug Fix | |
Doc Text: |
Cause:
For better network performance both libvirt and qemu allow to split network traffic into multiple queues which can then be processed in parallel on multiple host CPUs. One queue per CPU. The queues are represented by a file descriptors that are passed to qemu. However, this didn't play nicely with sVirt. The FDs created by libvirt was not labelled, while qemu process running the guest was (it's sVirt after all).
Consequence:
The qemu process got permission denied when trying to utilize the multiqueue feature and died subsequently.
Fix:
Libvirt sets label on all FDs passed to qemu now.
Result:
Qemu can fully utilize the multiqueue feature.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1149664 1171125 (view as bug list) | Environment: | ||
Last Closed: | 2015-03-05 07:35:08 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1149664 | |||
Bug Blocks: | 1171125 |
Description
Hu Jianwei
2014-05-08 09:04:11 UTC
Can not reproduce by my side directly by qemu, both test with qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 and qemu-kvm-1.5.3-60.el7.x86_64 cli, boot with 5 queues, 10 vectors and -smp 1 # /usr/libexec/qemu-kvm -name r7_n -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614 -no-user-config -nodefaults -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/rhel7cp1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,queues=5,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -spice port=5930,disable-ticketing -device usb-tablet,id=input0 -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -monitor stdio then inside guest, enable mq: # ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 # ethtool -L eth0 combined 5 # ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 5 guest # uname -r 3.10.0-123.el7.x86_64 Hi, I can not reproduce it, could you have a look my steps and tell me any difference, does this issue 100% reproduce in your enviroment? Could you have a look comment2 and add your comment? Best Regards, Junyi (In reply to Hu Jianwei from comment #0) > Description of problem: > qemu-kvm process crashed after enabling the multi-queue support in one > network interface in guest OS. > > Version-Release number of selected component (if applicable): > libvirt-1.1.1-29.el7.x86_64 > qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 > kernel-3.10.0-121.el7.x86_64 > > How reproducible: > 100% > > Steps to Reproduce: > 1. [root@localhost ~]# virsh dumpxml r7_n | grep interface -A8 > <interface type='network'> > <mac address='52:54:00:e5:99:99'/> > <source network='default_001'/> > <target dev='vnet0'/> > <model type='virtio'/> > <driver name='vhost' queues='5'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > ... > > [root@localhost ~]# virsh start r7_n > Domain r7_n started > > 2. In guest OS: > kernel version > kernel-3.10.0-123.el7.x86_64 > > [root@oolong ~]# ethtool -l eth0 > Channel parameters for eth0: > Pre-set maximums: > RX: 0 > TX: 0 > Other: 0 > Combined: 5 > Current hardware settings: > RX: 0 > TX: 0 > Other: 0 > Combined: 1 > > [root@oolong ~]# ethtool -L eth0 combined 5 > > [root@localhost network-scripts]# <=====lost connection from guest. > > > > Actual results: > As shown above, after running "ethtool -L eth0 combined 5", the qemu-kvm > process crashed. > > > [root@localhost ~]# tailf /var/log/libvirt/qemu/r7_n.log > 2014-05-08 08:07:24.758+0000: 4875: debug : virFileClose:90 : Closed fd 37 > 2014-05-08 08:07:24.758+0000: 4875: debug : virCommandHandshakeChild:391 : > Handshake with parent is done > char device redirected to /dev/pts/4 (label charserial0) > main_channel_link: add main channel client > main_channel_handle_parsed: net test: latency 0.144000 ms, bitrate > 36571428571 bps (34877.232142 Mbps) > inputs_connect: inputs channel client create > red_dispatcher_set_cursor_peer: > qemu-kvm: could not enable queue > qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407: > virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed. > 2014-05-08 08:09:06.428+0000: shutting down > This generally mean there's a mis configuration of host network. Could you pls show me how the network setup in host? (tap bridge or macvtap)? > ^C > > Expected results: > The multi-queue can be enabled on that interface, and the qemu-kvm process > should keep running, no crash occured. > > > Additional info: > qemu-kvm command line(generated by libvirt): > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name r7_n -S -machine > pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp > 1,sockets=1,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614 > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_n.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive > file=/var/lib/libvirt/images/r7_n.img,if=none,id=drive-virtio-disk0, > format=raw,cache=none -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0, > id=virtio-disk0,bootindex=1 -netdev > tap,fds=25:26:27:28,id=hostnet0,vhost=on,vhostfds=29:30:31:32 -device Looks like a bug of libvirt? queues = 5 is specified but only 4 tap file descriptors and vhost file descriptors were pass to qemu? > virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:e5:99: > 99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -chardev > spicevmc,id=charchannel0,name=vdagent -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0, > name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice > port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl > -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device > intel-hda,id=sound0,bus=pci.0,addr=0x4 -device > hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 Could not reproduce with the following qemu-kvm cli: /usr/libexec/qemu-kvm -netdev tap,vhost=on,id=hn0,fds=3:4:5:6:7,vhostfds=8:9:10:11:12: /home/kvm_autotest_root/images/RHEL-Server-7.0-64-virtio.qcow2 -m 2G -device virtio-net-pci,netdev=hn0,vectors=32,mac=00:00:05:00:00:06,mq=on -enable-kvm -smp 5 -vnc :10 btw, is it a regression? > This generally mean there's a mis configuration of host network. Could you > pls show me how the network setup in host? (tap bridge or macvtap)? The virtual network is like below: [root@localhost ~]# virsh net-dumpxml default <network connections='1'> <name>default</name> <uuid>5de056c8-568f-4a90-8862-0048150aa230</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0' /> <mac address='52:54:00:b5:c0:46'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254' /> </dhcp> </ip> </network> [root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.525400b5c046 yes virbr0-nic vnet0 virbr5 8000.52540025cf91 yes virbr5-nic [root@localhost ~]# ifconfig virbr0 virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ether 52:54:00:b5:c0:46 txqueuelen 0 (Ethernet) RX packets 964 bytes 114030 (111.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 515 bytes 1106097 (1.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost ~]# ifconfig vnet0 vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::fc54:ff:fee5:9999 prefixlen 64 scopeid 0x20<link> ether fe:54:00:e5:99:99 txqueuelen 500 (Ethernet) RX packets 90 bytes 9235 (9.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 183 bytes 10640 (10.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > Looks like a bug of libvirt? > > queues = 5 is specified but only 4 tap file descriptors and vhost file > descriptors were pass to qemu? Sorry for putting another qemu command line in Additional info part in the bug, libvirt can pass a correct value to qemu(2N+2), please refer to below: [root@localhost ~]# virsh dumpxml r7_n | grep queue -A4 <driver name='vhost' queues='6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> [root@localhost ~]# virsh start r7_n Domain r7_n started [root@localhost ~]# ps aux | grep qemu-kvm |grep -v grep| sed 's/-device/\n-device/g' | grep "virtio-net-pci" -device virtio-net-pci,mq=on,vectors=14,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 [root@localhost ~]# (In reply to Hu Jianwei from comment #6) > > This generally mean there's a mis configuration of host network. Could you > > pls show me how the network setup in host? (tap bridge or macvtap)? > > The virtual network is like below: > > [root@localhost ~]# virsh net-dumpxml default > <network connections='1'> > <name>default</name> > <uuid>5de056c8-568f-4a90-8862-0048150aa230</uuid> > <forward mode='nat'> > <nat> > <port start='1024' end='65535'/> > </nat> > </forward> > <bridge name='virbr0' stp='on' delay='0' /> > <mac address='52:54:00:b5:c0:46'/> > <ip address='192.168.122.1' netmask='255.255.255.0'> > <dhcp> > <range start='192.168.122.2' end='192.168.122.254' /> > </dhcp> > </ip> > </network> > > [root@localhost ~]# brctl show > bridge name bridge id STP enabled interfaces > virbr0 8000.525400b5c046 yes virbr0-nic > vnet0 > virbr5 8000.52540025cf91 yes virbr5-nic > [root@localhost ~]# ifconfig virbr0 > virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 > ether 52:54:00:b5:c0:46 txqueuelen 0 (Ethernet) > RX packets 964 bytes 114030 (111.3 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 515 bytes 1106097 (1.0 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > [root@localhost ~]# ifconfig vnet0 > vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::fc54:ff:fee5:9999 prefixlen 64 scopeid 0x20<link> > ether fe:54:00:e5:99:99 txqueuelen 500 (Ethernet) > RX packets 90 bytes 9235 (9.0 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 183 bytes 10640 (10.3 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > Looks like a bug of libvirt? > > > > queues = 5 is specified but only 4 tap file descriptors and vhost file > > descriptors were pass to qemu? > > Sorry for putting another qemu command line in Additional info part in the > bug, libvirt can pass a correct value to qemu(2N+2), please refer to below: > > [root@localhost ~]# virsh dumpxml r7_n | grep queue -A4 > <driver name='vhost' queues='6'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > <serial type='pty'> > <target port='0'/> > [root@localhost ~]# virsh start r7_n > Domain r7_n started > > [root@localhost ~]# ps aux | grep qemu-kvm |grep -v grep| sed > 's/-device/\n-device/g' | grep "virtio-net-pci" > -device > virtio-net-pci,mq=on,vectors=14,netdev=hostnet0,id=net0,mac=52:54:00:e5:99: > 99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 > [root@localhost ~]# Confused, several questions: - Did you reproduce the issue with queues=5 or queues=6? - What the full qemu-kvm command line when you reproduce this. - Is it a regression? Thanks > Confused, several questions:
>
> - Did you reproduce the issue with queues=5 or queues=6?
> - What the full qemu-kvm command line when you reproduce this.
> - Is it a regression?
>
Q1, yes, I can reproduce it with queues=5 and 6, exactly the reproduced value is from 2 to 8. (the valid value of queues is from 1 to 8.)
Q2, full command line(e.g. queues=8):
/usr/libexec/qemu-kvm -name r7_n -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 5,sockets=5,cores=1,threads=1 -uuid 8577cccf-58ca-4a90-9558-a321e9655614 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_n.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/r7_n.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=25:26:27:28:29:30:31:32,id=hostnet0,vhost=on,vhostfds=33:34:35:36:37:38:39:40 -device virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=52:54:00:e5:99:99,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
Q3, I just tested it using the latest versions, if necessary, I'll do the regression testing for it using old versions from libvirt side.
Thanks.
> - Is it a regression? > > Thanks Maybe this is not a regression issue. I tried below combined versions, the bug reprodcued on all, please refer to the following: 1.libvirt-1.1.1-26.el7.x86_64 (Fixed bug 1071888, vectors parameter is 2N+2.) qemu-kvm-1.5.3-45.el7.x86_64 cat /var/log/libvirt/qemu/r7_n.log ... qemu-kvm: could not enable queue qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed. 2. libvirt-1.1.1-26.el7.x86_64 qemu-kvm-1.5.3-2.el7.x86_64 cat /var/log/libvirt/qemu/r7_n.log ... qemu-kvm: could not enable queue qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:305: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed. 2014-05-09 07:05:46.181+0000: shutting down 3. libvirt-1.1.1-15.el7.x86_64 qemu-kvm-1.5.3-2.el7.x86_64 cat /var/log/libvirt/qemu/r7_n.log ... qemu-kvm: could not enable queue qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:305: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed. Thanks. Thanks for the testing what does you /var/log/audit/audit.log looks like just after the crash? Can you reproduce if you disable the selinux? Thanks (In reply to jason wang from comment #10) > Thanks for the testing > > what does you /var/log/audit/audit.log looks like just after the crash? Can > you reproduce if you disable the selinux? > > Thanks [root@sriov2 ~]# getenforce Enforcing [root@sriov2 ~]# virsh dumpxml r7 | grep queue -A4 <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> [root@sriov2 ~]# virsh start r7 Domain r7 started [root@sriov2 ~]# virsh console r7 Connected to domain r7 Escape character is ^] Red Hat Enterprise Linux Server 7.0 (Maipo) Kernel 3.10.0-123.el7.x86_64 on an x86_64 oolong login: root Password: Last login: Fri May 9 15:54:20 on ttyS0 [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 2 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 [root@oolong ~]# ethtool -L eth0 combined 2 [root@sriov2 ~]# tailf /var/log/audit/audit.log type=AVC msg=audit(1399622478.790:893): avc: denied { attach_queue } for pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket type=SYSCALL msg=audit(1399622478.790:893): arch=c000003e syscall=16 success=no exit=-13 a0=18 a1=400454d9 a2=7ff7f1b5e970 a3=0 items=0 ppid=1 pid=7585 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c638,c877 key=(null) type=ANOM_ABEND msg=audit(1399622478.790:894): auid=4294967295 uid=107 gid=107 ses=4294967295 subj=system_u:system_r:svirt_t:s0:c638,c877 pid=7585 comm="qemu-kvm" reason="memory violation" sig=6 type=ANOM_PROMISCUOUS msg=audit(1399622482.570:895): dev=vnet0 prom=0 old_prom=256 auid=4294967295 uid=107 gid=107 ses=4294967295 type=VIRT_CONTROL msg=audit(1399622482.692:896): pid=2409 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=stop reason=failed vm="r7" uuid=353713ab-40d9-4523-8fc1-26282dfe88e9 vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' Disabled selinux on host, the bug was disappeared. [root@sriov2 ~]# setenforce 0 [root@sriov2 ~]# getenforce Permissive [root@sriov2 ~]# virsh start r7 Domain r7 started [root@sriov2 ~]# virsh console r7 Connected to domain r7 Escape character is ^] Red Hat Enterprise Linux Server 7.0 (Maipo) Kernel 3.10.0-123.el7.x86_64 on an x86_64 oolong login: root Password: Last login: Fri May 9 16:01:02 on ttyS0 [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 2 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 [root@oolong ~]# ethtool -L eth0 combined 2 [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 2 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 2 Thanks. Looks like SELinux prevent qemu to attach the a new tun queue. We need allow this operation otherwise the multiqueue function won't be functional. type=AVC msg=audit(1399622478.790:893): avc: denied { attach_queue } for pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket Reset the component to selinux-policy. Feel free to reassign this bug if I was wrong. I think this is a libvirt bug. tun_socket should be relabeled to svirt_t to have allow svirt_t self:tun_socket attach_queue; Reproduced this bug with following components and Jason's script: # uname -r 3.10.0-123.el7.x86_64 # rpm -q qemu-kvm qemu-kvm-1.5.3-60.el7_0.2.x86_64 # rpm -qa |grep selinux selinux-policy-3.12.1-153.el7.noarch libselinux-utils-2.2.2-6.el7.x86_64 libselinux-python-2.2.2-6.el7.x86_64 selinux-policy-targeted-3.12.1-153.el7.noarch libselinux-2.2.2-6.el7.x86_64 Steps: 1.Change security context of the image to test: # chcon -u system_u -r system_r -t svirt_t /rhel7-0409.qcow2 2.Confirm the SElinux is enabled with mode enforcing: # sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 3.Launch qemu via user 'qemu' : # ps auZ |grep kvm unconfined_u:system_r:svirt_t:s0-s0:c0.c1023 qemu 4949 103 0.3 2734940 27964 pts/0 Sl+ 15:56 0:03 /usr/libexec/qemu-kvm -netdev tap,vhost=on,id=hn0,fds=3:4:5:6,vhostfds=7:8:9:10 /rhel7-0409.qcow2 -m 2G -device virtio-net-pci,netdev=hn0,vectors=32,mac=00:00:05:00:00:06,mq=on -enable-kvm -smp 4 -vnc :10 4.Inside guest, try to enable mq: # ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 4 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 # ethtool -L eth0 combined 4 Result, qemu quit like this: qemu-kvm: could not enable queue qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/hw/net/virtio-net.c:407: virtio_net_set_queues: Assertion `!peer_attach(n, i)' failed. From log: ... type=AVC msg=audit(1399967917.807:630): avc: denied { attach_queue } for pid=4966 comm="qemu-kvm" scontext=unconfined_u:system_r:svirt_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=tun_socket .... So according to above, this bug is reproduced. # chcon -u system_u -r system_r -t svirt_t /rhel7-0409.qcow2 Is the wrong thing to do svirt_t is a process type not a file type. If you did this on an enforcing system it would be blocked. https://danwalsh.livejournal.com/54803.html Miroslav was reporting that libvirt should tell SELinux the label on the tun_socket should be svirt_t before creating it. This would allow us to make sure only the tun_socket intended for the VM can be used by the VM. If we allow svirt_t to use virtd_t tun_socket, then it can use any tun_socket that got leaked to it. Including ones intended for other VMs (In reply to Miroslav Grepl from comment #19) > I think this is a libvirt bug. > > tun_socket should be relabeled to svirt_t to have > > allow svirt_t self:tun_socket attach_queue; But the /dev/net/tun has tun_tap_device_t label by default. So what you are suggesting is to behave differently in case of multiqueue (MQ) and in case of a single queue? That is, if the TAP is MQ label the /dev/net/tun with svirt_t, if it's a single queue label it with tun_tap_device_t? It's not gonna fly really because if libvirt changes the label on the /dev/net/tun other process (possibly other libvirtd) may hop in and open the device with label it should not have. Even locks within libvirt won't help as the problem is at interprocess level. Well you get type=AVC msg=audit(1399622478.790:893): avc: denied { attach_queue } for pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket I've changed the libvirt code, so that tun FD is labeled as svirt_t but I still see AVC denial: kernel: type=1400 audit(1408450864.055:32): avc: denied { relabelto } for pid=32311 comm="lt-libvirtd" name="tun" dev="devtmpfs" ino=15650 scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:svirt_t:s0:c258,c379 tclass=chr_file Aparently, libvirt can't relabel chr_file. Well you have libvirtd running as init_t. ls -Z /usr/sbin/libvirtd and ps -eZ |grep libvirt Okay, I've proposed patch upstream: https://www.redhat.com/archives/libvir-list/2014-August/msg00801.html I've just pushed the patch upstream: commit cf976d9dcf4e592261b14f03572bb519531ebbce Author: Michal Privoznik <mprivozn> AuthorDate: Wed Aug 13 16:08:03 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Wed Aug 20 09:42:24 2014 +0200 qemu: Label all TAP FDs https://bugzilla.redhat.com/show_bug.cgi?id=1095636 When starting up the domain the domain's NICs are allocated. As of 1f24f682 (v1.0.6) we are able to use multiqueue feature on virtio NICs. It breaks network processing into multiple queues which can be processed in parallel by different host CPUs. The queues are, however, created by opening /dev/net/tun several times. Unfortunately, only the first FD in the row is labelled so when turning the multiqueue feature on in the guest, qemu will get AVC denial. Make sure we label all the FDs needed. Moreover, the default label of /dev/net/tun doesn't allow attaching a queue: type=AVC msg=audit(1399622478.790:893): avc: denied { attach_queue } for pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tclass=tun_socket And as suggested by SELinux maintainers, the tun FD should be labeled as svirt_t. Therefore, we don't need to adjust any range (as done previously by Guannan in ae368ebf) rather set the seclabel of the domain directly. Signed-off-by: Michal Privoznik <mprivozn> v1.2.7-179-gcf976d9 Going through the upstream git log, I've noticed this commit which we may need to backport too: commit a4431931393aeb1ac5893f121151fa3df4fde612 Author: Martin Kletzander <mkletzan> AuthorDate: Mon Sep 1 15:27:00 2014 +0200 Commit: Martin Kletzander <mkletzan> CommitDate: Mon Sep 1 15:36:23 2014 +0200 selinux: properly label tap FDs with imagelabel The cleanup in commit cf976d9d used secdef->label to label the tap FDs, but that is not possible since it's process-only label (svirt_t) and not a object label (e.g. svirt_image_t). Starting a domain failed with EPERM, but simply using secdef->imagelabel instead of secdef->label fixes it. Signed-off-by: Martin Kletzander <mkletzan> I could reproduce this issue with libvirt-1.1.1-29.el7.x86_64 Steps to Reproduce: 1. [root@localhost ~]# virsh dumpxml rhel7 | grep interface -A8 <interface type='network'> <mac address='52:54:00:e5:99:11'/> <source network='default_001'/> <target dev='vnet0'/> <model type='virtio'/> <driver name='vhost' queues='5'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> ... [root@localhost ~]# virsh start rhel7 Domain rhel7 started 2. In guest OS: [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 [root@oolong ~]# ethtool -L eth0 combined 5 [root@localhost /]# <=====lost connection from guest. After running "ethtool -L eth0 combined 5", the qemu-kvm process crashed. With libvirt-1.2.8-1.el7.x86_64, this issue still exists: Here are steps: 1. [root@localhost ~]# virsh dumpxml rhel7 | grep interface -A8 <interface type='network'> <mac address='52:54:00:e5:99:11'/> <source network='default_001'/> <target dev='vnet0'/> <model type='virtio'/> <driver name='vhost' queues='5'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> ... [root@localhost ~]# virsh start rhel7 Domain rhel7 started 2. In guest OS: [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 [root@oolong ~]# ethtool -L eth0 combined 5 [root@localhost /]# <=====lost connection from guest. After running "ethtool -L eth0 combined 5", the qemu-kvm process crashed. Could you help to check or confirm above? (In reply to Michal Privoznik from comment #34) > Going through the upstream git log, I've noticed this commit which we may > need to backport too: > > commit a4431931393aeb1ac5893f121151fa3df4fde612 > Author: Martin Kletzander <mkletzan> > AuthorDate: Mon Sep 1 15:27:00 2014 +0200 > Commit: Martin Kletzander <mkletzan> > CommitDate: Mon Sep 1 15:36:23 2014 +0200 > > selinux: properly label tap FDs with imagelabel > > The cleanup in commit cf976d9d used secdef->label to label the tap > FDs, but that is not possible since it's process-only label (svirt_t) > and not a object label (e.g. svirt_image_t). Starting a domain failed > with EPERM, but simply using secdef->imagelabel instead of > secdef->label fixes it. > > Signed-off-by: Martin Kletzander <mkletzan> Move back to ASSIGNED. Not sure if it is possible but could we change virtd_t to virtd_tun_t if FD is passed and allow svirt_t to access virtd_tun_t tun_socket? system_u:system_r:virtd_tun_t:s0 So we would have type=AVC msg=audit(1399622478.790:893): avc: denied { attach_queue } for pid=7585 comm="qemu-kvm" scontext=system_u:system_r:svirt_t:s0:c638,c877 tcontext=system_u:system_r:virtd_tun_t:s0-s0:c0.c1023 tclass=tun_socket Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2014-September/msg01163.html Miroslav, do you want me to clone this to selinux component so you can push the fix? So I've just pushed the patch upstream: commit ba7468dbb13f552a9177d01ea8bad155f9877bc3 Author: Michal Privoznik <mprivozn> AuthorDate: Thu Sep 18 15:17:29 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Sep 18 19:02:47 2014 +0200 virSecuritySELinuxSetTapFDLabel: Temporarily revert to old behavior https://bugzilla.redhat.com/show_bug.cgi?id=1141879 A long time ago I've implemented support for so called multiqueue net. The idea was to let guest network traffic be processed by multiple host CPUs and thus increasing performance. However, this behavior is enabled by QEMU via special ioctl() iterated over the all tap FDs passed in by libvirt. Unfortunately, SELinux comes in and disallows the ioctl() call because the /dev/net/tun has label system_u:object_r:tun_tap_device_t:s0 and 'attach_queue' ioctl() is not allowed on tun_tap_device_t type. So after discussion with a SELinux developer we've decided that the FDs passed to the QEMU should be labelled with svirt_t type and SELinux policy will allow the ioctl(). Therefore I've made a patch (cf976d9dcf4e592261b14f03572) that does exactly this. The patch was fixed then by a4431931393aeb1ac5893f121151fa3df4fde612 and b635b7a1af0e64754016d758376f382470bc11e7. However, things are not that easy - even though the API to label FD is called (fsetfilecon_raw) the underlying file is labelled too! So effectively we are mangling /dev/net/tun label. Yes, that broke dozen of other application from openvpn, or boxes, to qemu running other domains. The best solution would be if SELinux provides a way to label an FD only, which could be then labeled when passed to the qemu. However that's a long path to go and we should fix this regression AQAP. So I went to talk to the SELinux developer again and we agreed on temporary solution that: 1) All the three patches are reverted 2) SELinux temporarily allows 'attach_queue' on the tun_tap_device_t Signed-off-by: Michal Privoznik <mprivozn> v1.2.8-204-gba7468d Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-October/msg00134.html Hi Michal, I still found the issue with libvirt-1.2.8-5.el7 we talked before: 1.prepare two guest with NIC # virsh dumpxml test34|grep interface -2 <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00c:158'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> # virsh dumpxml test3|grep interface -2 <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='00:22:44:66:88:aa'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <smartcard mode='passthrough' type='spicevmc'> <address type='ccid' controller='0' slot='0'/> 2.start 2 guest # virsh start test3 Domain test3 started # virsh start test34 Domain test34 started 3.seliunx will report a lot of avc time->Tue Oct 14 11:58:12 2014 type=SYSCALL msg=audit(1413259092.550:943913): arch=c000003e syscall=0 success=no exit=-13 a0=1a a1=7f50d74a16bc a2=11000 a3=3 items=0 ppid=1 pid=727 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c621,c731 key=(null) type=AVC msg=audit(1413259092.550:943913): avc: denied { read } for pid=727 comm="qemu-kvm" path="/dev/net/tun" dev="devtmpfs" ino=8321 scontext=system_u:system_r:svirt_t:s0:c621,c731 tcontext=system_u:object_r:tun_tap_device_t:s0:c569,c714 tclass=chr_file ---- time->Tue Oct 14 11:58:12 2014 type=SYSCALL msg=audit(1413259092.550:943914): arch=c000003e syscall=0 success=no exit=-13 a0=1a a1=7f50d74a16bc a2=11000 a3=3 items=0 ppid=1 pid=727 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c621,c731 key=(null) ^C 4.# ll -Z /dev/net/tun crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0:c279,c536 /dev/net/tun This issue just like bug 1147057, and the result just like bug 1147057 comment 12 said: one vm gets access to the network and all others don't. Would you please tell me if you will fix this issue in this bug? or open another bug for it? Thanks, Luyao Huang (In reply to Luyao Huang from comment #41) > Hi Michal, > > I still found the issue with libvirt-1.2.8-5.el7 we talked before: > > one vm gets access to the network and all others don't. > > Would you please tell me if you will fix this issue in this bug? or open > another bug for it? It seems like we need yet another upstream patch: http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-October/msg00375.html Michal *** Bug 1149664 has been marked as a duplicate of this bug. *** I can not reproduce it using libvirt-1.2.8-6.el7.x86_64 Version: libvirt-1.2.8-6.el7.x86_64 qemu-kvm-rhev-2.1.2-6.el7.x86_64 qemu-kvm-1.5.3-77.el7.x86_64 [root@localhost ~]# getenforce Enforcing [root@localhost ~]# ll -Z /dev/net/tun crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun [root@localhost ~]# virsh dumpxml r7 | grep "/interface" -B6 <interface type='network'> <mac address='52:54:00:e5:99:99'/> <source network='default'/> <model type='virtio'/> <driver name='vhost' queues='5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> Before starting the domain, in another terminal, check the output of audit. [root@localhost ~]# tailf /var/log/audit/audit.log | grep denied [root@localhost ~]# virsh start r7 Domain r7 started [root@localhost ~]# virsh console r7 Connected to domain r7 Escape character is ^] Red Hat Enterprise Linux Server 7.0 (Maipo) Kernel 3.10.0-123.el7.x86_64 on an x86_64 oolong login: root Password: Last login: Thu Nov 6 15:44:41 on ttyS0 [root@oolong ~]# ifconfig eth0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.122.58 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::5054:ff:fee5:9999 prefixlen 64 scopeid 0x20<link> ether 52:54:00:e5:99:99 txqueuelen 1000 (Ethernet) RX packets 37 bytes 3278 (3.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 42 bytes 5711 (5.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1 [root@oolong ~]# ethtool -L eth0 combined 5 [root@oolong ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 5 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 5 [root@localhost ~]# ll -Z /dev/net/tun crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun [root@localhost ~]# virsh destroy r7 Domain r7 destroyed [root@localhost ~]# ll -Z /dev/net/tun crw-rw-rw-. root root system_u:object_r:tun_tap_device_t:s0 /dev/net/tun During the doamin life cycle, no any avc denied message printed. [root@localhost ~]# tailf /var/log/audit/audit.log | grep denie <=== nothing I found the patch series is temporary, could I verify it according to above results? If we have better solution, need we re-open and re-test it again? Thanks. (In reply to Hu Jianwei from comment #45) > I can not reproduce it using libvirt-1.2.8-6.el7.x86_64 > > Version: > libvirt-1.2.8-6.el7.x86_64 > qemu-kvm-rhev-2.1.2-6.el7.x86_64 > qemu-kvm-1.5.3-77.el7.x86_64 > > I found the patch series is temporary, could I verify it according to above > results? > > If we have better solution, need we re-open and re-test it again? Thanks. This is enough to verify the bug. The temporary means, until the time SELinux provides a better API to label resources, the patch fixes the issue. Once SELinux does provide API and libvirt learns it, there shouldn't be any change in behaviour. So I'd say move this bug to VERIFIED and trouble with new approach when it gets real (which may take ages). Thanks for Michal's reply. According to comment 45 and 46, we can get expected results, so change to Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |