Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1772943
Summary: | Support embedding the QEMU driver in client apps directly (TechPreview) | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Ademar Reis <areis> | ||||
Component: | libvirt | Assignee: | Daniel Berrangé <berrange> | ||||
Status: | CLOSED ERRATA | QA Contact: | yafu <yafu> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 8.0 | CC: | berrange, chhu, jdenemar, lmen, rjones, xuzhang, ymankad | ||||
Target Milestone: | rc | Keywords: | FutureFeature, TechPreview, Triaged | ||||
Target Release: | 8.2 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-6.2.0-1.el8 | Doc Type: | Enhancement | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2020-11-17 17:45:34 UTC | Type: | Feature Request | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Ademar Reis
2019-11-15 16:38:09 UTC
Initial impl is merged upstream with commit 217b564a77041dd5877527caf86bd5bb570d6f44 Author: Daniel P. Berrangé <berrange> Date: Wed Jan 29 11:17:54 2020 +0000 docs: add warning about virt-qemu-run being experimental commit d6006672788ec0f0290d35c76ceb9672476d1ea8 Author: Daniel P. Berrangé <berrange> Date: Fri May 17 13:01:59 2019 +0100 qemu: introduce a new "virt-qemu-run" program commit 3e9076e777aff2f4b08330ed17e559fcfb6b3529 Author: Daniel P. Berrangé <berrange> Date: Fri May 17 12:35:57 2019 +0100 secrets: add support for running secret driver in embedded mode commit 068efae5b1a9efeea4a9c3bc0ae80747da5024fb Author: Daniel P. Berrangé <berrange> Date: Fri May 17 12:35:57 2019 +0100 qemu: add support for running QEMU driver in embedded mode commit 88446e07b2050c30ed07a2a7d26507674c8abe01 Author: Daniel P. Berrangé <berrange> Date: Fri May 17 12:42:04 2019 +0100 libvirt: support an "embed" URI path selector for opening drivers Hi Daniel, Failed to connect virtlogd when trying to start guest with 'virt-qemu-run'. Could you help to check whether it's a bug of selinux-policy or not please? Thanks. Test pkg: libvirt-daemon-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64 1.# virt-qemu-run vm1.xml virt-qemu-run: cannot start VM: can't connect to virtlogd: Cannot recv data: Connection reset by peer 2.Check the audit log: # ausearch -m AVC -ts boot | grep -i virtlogd type=PROCTITLE msg=audit(1591695650.402:285): proctitle="/usr/sbin/virtlogd" type=SYSCALL msg=audit(1591695650.402:285): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=55d680a238e0 a2=0 a3=0 items=0 ppid=1 pid=37358 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="virtlogd" exe="/usr/sbin/virtlogd" subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1591695650.402:285): avc: denied { search } for pid=37358 comm="virtlogd" name="37328" dev="proc" ino=52634 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dir permissive=0 type=PROCTITLE msg=audit(1591695975.266:291): proctitle="/usr/sbin/virtlogd" This is a tricky policy question. Historically QEMU driver ran inside libvirtd, and thus we have a policy rule designed around a "virtd_t" type. When we're embedding QEMU driver into application code, those applications need to have equivalent permissions to virtd_t, but we don't want to run the app as virtd_t in general. Each app that needs to use embedded QEMU will need its own SELinux policy that inherits much, but not all, of virtd_t type. For this, we're really goiing to want to restructure libvirt selinux policy to separate it out for the modular deamons (virtqemud, replacing libvirtd). I think we'll have to consider SELinux enforcement out of scope in the near term, as I can't see us solving this for 8.3 Hi Daniel, I changed the setting of qemu.conf and it seems qemu.conf does not work for embedding QEMU driver. Is that expected behavior? The embedded QEMU driver is intended to be *completely* isolated from the main libvirtd QEMU driver. So yes, the /etc/libvirt/qemu.conf will have no effect on the embedded driver. The embedded driver has its own private root directory with its own location, $EMBEDDED-ROOT/etc/qemu.conf file https://libvirt.org/drvqemu.html#embedTree (In reply to Daniel Berrangé from comment #9) > The embedded QEMU driver is intended to be *completely* isolated from the > main libvirtd QEMU driver. So yes, the /etc/libvirt/qemu.conf will have no > effect on the embedded driver. The embedded driver has its own private root > directory with its own location, $EMBEDDED-ROOT/etc/qemu.conf file > https://libvirt.org/drvqemu.html#embedTree Thanks for your quickly reply. I have another question about starting guest with luks image. I setup the env as the following steps: 1. prepare a qcow2 luks image: #qemu-img create --object secret,id=sec0,data=123456,for‐ -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G 2.Check the image: # qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2 No errors were found on the image. Image end offset: 2359296 3.Prepare secret xml and secret value file: #cat /xml/secret.xml <secret ephemeral='no' private='yes'> <description>LUKS Sample Secret</description> <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid> <usage type='volume'> <volume>/var/lib/libvirt/images/luks.img</volume> </usage> </secret> #cat /xml/secret-value 123456 4.Prepare a guest xml with the luks image(the same uuid with the secret xml): #cat /tmp/vm1.xml ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/> <source file='/var/lib/libvirt/images/luks.qcow2' index='1'> <encryption format='luks'> <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/> </encryption> </source> ... </disk> ... 5.Start guest: # virt-qemu-run -v -d /tmp/vm1.xml -s /xml/secret.xml,/xml/secret-value virt-qemu-run: 77: initializing libvirt virt-qemu-run: 1165: initializing signal handlers virt-qemu-run: 1339: preparing event loop thread virt-qemu-run: 1393: opening secret:///embed?root=/tmp/virt-qemu-run-CKP3N0 virt-qemu-run: 2259: loading secret /xml/secret.xml and /xml/secret-value virt-qemu-run: 2848: opening qemu:///embed?root=/tmp/virt-qemu-run-CKP3N0 virt-qemu-run: 30921: starting guest /tmp/vm1.xml virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-07-21T12:38:19.039565Z qemu-kvm: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-1-format-encryption-secret0"},"file":"libvirt-1-storage","backing":null}: Invalid password, cannot unlock any keyslot virt-qemu-run: 3125617: cleaned up, exiting It seems the raw value of secret is wrong. Did i skip any steps or was it anything wrong with the secret value file? Do you have a trailing newline in your secret-value file ? eg use "echo -n 123456 > secret-value", not a plain "echo 123456" (In reply to Daniel Berrangé from comment #11) > Do you have a trailing newline in your secret-value file ? eg use "echo > -n 123456 > secret-value", not a plain "echo 123456" Use "echo -n 123456" > secret-value" works. Thanks a lot. Test with libvirt-daemon-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64. S1.Start guest without specifying root dir: 1.#setenforce 0 2.#virt-qemu-run -d -v /tmp/vm1.xml virt-qemu-run: 17: initializing libvirt virt-qemu-run: 2230: initializing signal handlers virt-qemu-run: 2533: preparing event loop thread virt-qemu-run: 2681: opening qemu:///embed?root=/tmp/virt-qemu-run-2K72Q0 virt-qemu-run: 36029: starting guest /tmp/vm1.xml virt-qemu-run: 1963123: guest running, Ctrl-C to stop now virt-qemu-run: XML: <domain type='kvm' id='1'> <name>avocado-vt-vm1</name> <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid> <metadata> ... 3.Check the root directory created by libvirt in another terminal: #tree -f . ├── ./cache │ └── ./cache/qemu │ └── ./cache/qemu/capabilities │ └── ./cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml ├── ./lib │ └── ./lib/qemu │ ├── ./lib/qemu/channel │ │ └── ./lib/qemu/channel/target │ │ └── ./lib/qemu/channel/target/domain-1-avocado-vt-vm1 │ ├── ./lib/qemu/checkpoint │ ├── ./lib/qemu/domain-1-avocado-vt-vm1 │ │ ├── ./lib/qemu/domain-1-avocado-vt-vm1/master-key.aes │ │ └── ./lib/qemu/domain-1-avocado-vt-vm1/monitor.sock │ ├── ./lib/qemu/dump │ ├── ./lib/qemu/nvram │ ├── ./lib/qemu/ram │ ├── ./lib/qemu/save │ └── ./lib/qemu/snapshot ├── ./log │ └── ./log/qemu │ └── ./log/qemu/avocado-vt-vm1.log └── ./run └── ./run/qemu ├── ./run/qemu/autostarted ├── ./run/qemu/avocado-vt-vm1.pid ├── ./run/qemu/avocado-vt-vm1.xml ├── ./run/qemu/dbus ├── ./run/qemu/driver.pid └── ./run/qemu/slirp 21 directories, 8 files 4.Check the qemu proceess: #ps aux | grep -i qemu # ps aux | grep -i qemu root 59773 0.0 0.0 437812 27684 pts/0 Sl+ 03:25 0:00 virt-qemu-run -d -v /tmp/vm1.xml qemu 59817 3.1 1.9 3076440 1242760 ? Sl 03:25 0:40 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/tmp/virt-qemu-run-2K72Q0/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes ... 5.Enter 'Ctrl-C' to terminate 'virt-qemu-run' in step 2: ^Cvirt-qemu-run: 1425433879: cleaned up, exiting 6.Check the root dir: # ls /tmp/virt-qemu-run-2K72Q0/ ls: cannot access '/tmp/virt-qemu-run-2K72Q0/': No such file or directory 7.Check the qemu process: # pgrep qemu no output S2.Start guest with specifying root dir: 1.#setenfore 0 2.# virt-qemu-run -d -v /tmp/vm1.xml --root=/var/tmp/vm1 virt-qemu-run: 66: initializing libvirt virt-qemu-run: 2278: initializing signal handlers virt-qemu-run: 2298: preparing event loop thread virt-qemu-run: 2450: opening qemu:///embed?root=/var/tmp/vm1 virt-qemu-run: 40390: starting guest /tmp/vm1.xml virt-qemu-run: 2452882: guest running, Ctrl-C to stop now virt-qemu-run: XML: <domain type='kvm' id='1'> <name>avocado-vt-vm1</name> <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid> <metadata> ... 3.Check the root directory: # tree -f /var/tmp/vm1/ /var/tmp/vm1 ├── /var/tmp/vm1/cache │ └── /var/tmp/vm1/cache/qemu │ └── /var/tmp/vm1/cache/qemu/capabilities │ └── /var/tmp/vm1/cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml ├── /var/tmp/vm1/lib │ └── /var/tmp/vm1/lib/qemu │ ├── /var/tmp/vm1/lib/qemu/channel │ │ └── /var/tmp/vm1/lib/qemu/channel/target │ │ └── /var/tmp/vm1/lib/qemu/channel/target/domain-1-avocado-vt-vm1 │ ├── /var/tmp/vm1/lib/qemu/checkpoint │ ├── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1 │ │ ├── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes │ │ └── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/monitor.sock │ ├── /var/tmp/vm1/lib/qemu/dump │ ├── /var/tmp/vm1/lib/qemu/nvram │ ├── /var/tmp/vm1/lib/qemu/ram │ ├── /var/tmp/vm1/lib/qemu/save │ └── /var/tmp/vm1/lib/qemu/snapshot ├── /var/tmp/vm1/log │ └── /var/tmp/vm1/log/qemu │ └── /var/tmp/vm1/log/qemu/avocado-vt-vm1.log └── /var/tmp/vm1/run └── /var/tmp/vm1/run/qemu ├── /var/tmp/vm1/run/qemu/autostarted ├── /var/tmp/vm1/run/qemu/avocado-vt-vm1.pid ├── /var/tmp/vm1/run/qemu/avocado-vt-vm1.xml ├── /var/tmp/vm1/run/qemu/dbus ├── /var/tmp/vm1/run/qemu/driver.pid └── /var/tmp/vm1/run/qemu/slirp 21 directories, 8 files 4.Check qemu process: # ps aux | grep -i qemu root 61100 0.1 0.0 437684 27612 pts/0 Sl+ 05:05 0:00 virt-qemu-run -d -v /tmp/vm1.xml --root=/var/tmp/vm1 qemu 61218 32.1 1.9 3136856 1242860 ? Sl 05:05 0:30 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes... 5.Enter 'Ctrl-C' to terminate 'virt-qemu-run' in step 2: ^Cvirt-qemu-run: 130260216: cleaned up, exiting 6.Check the root dir, the root dir still exists: # ls /var/tmp/vm1/ cache lib log run 7.Check the qemu process: # pgrep qemu no output S3: Start guest with luks image: 1. prepare a qcow2 luks image: #qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G 2.Check the image: # qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2 No errors were found on the image. Image end offset: 2359296 3.Prepare secret xml and secret value file: #cat /xml/secret.xml <secret ephemeral='no' private='yes'> <description>LUKS Sample Secret</description> <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid> <usage type='volume'> <volume>/var/lib/libvirt/images/luks.img</volume> </usage> </secret> #echo -n 123456 > /xml/secret-value 4.Prepare a guest xml with the luks image(the same uuid with the secret xml): #cat /tmp/vm1.xml ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/> <source file='/var/lib/libvirt/images/luks.qcow2' index='1'> <encryption format='luks'> <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/> </encryption> </source> ... </disk> ... 5.#setenfore 0 6.Start guest: # virt-qemu-run -v -d /tmp/vm1.xml -s /xml/secret.xml,/xml/secret-value virt-qemu-run: 68: initializing libvirt virt-qemu-run: 2048: initializing signal handlers virt-qemu-run: 2362: preparing event loop thread virt-qemu-run: 2491: opening secret:///embed?root=/tmp/virt-qemu-run-G224Q0 virt-qemu-run: 3852: loading secret /xml/secret.xml and /xml/secret-value virt-qemu-run: 4852: opening qemu:///embed?root=/tmp/virt-qemu-run-G224Q0 virt-qemu-run: 34810: starting guest /tmp/vm1.xml virt-qemu-run: 2230569: guest running, Ctrl-C to stop now virt-qemu-run: XML: <domain type='kvm' id='1'> <name>avocado-vt-vm1</name> <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid> <metadata> ... 7.Check qemu process: #ps aux | grep -i secret /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0...-object secret,id=libvirt-1-format-encryption-secret0,data=3ZOmoPbAVVWKBW2pkY6Ivg==,keyid=masterKey0,iv=WVqjT/Q9WIun1FwkkrqUvw==,format=base64 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/luks.qcow2","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},... 8.Check the root dir, could see the secret info: # tree -f /tmp/virt-qemu-run-KT56Q0 /tmp/virt-qemu-run-KT56Q0 ├── /tmp/virt-qemu-run-KT56Q0/cache │ └── /tmp/virt-qemu-run-KT56Q0/cache/qemu │ └── /tmp/virt-qemu-run-KT56Q0/cache/qemu/capabilities │ └── /tmp/virt-qemu-run-KT56Q0/cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml ├── /tmp/virt-qemu-run-KT56Q0/etc │ └── /tmp/virt-qemu-run-KT56Q0/etc/secrets │ ├── /tmp/virt-qemu-run-KT56Q0/etc/secrets/f981dd17-143f-45bc-88e6-ed1fe20ce9da.base64 *** │ └── /tmp/virt-qemu-run-KT56Q0/etc/secrets/f981dd17-143f-45bc-88e6-ed1fe20ce9da.xml *** ├── /tmp/virt-qemu-run-KT56Q0/lib │ └── /tmp/virt-qemu-run-KT56Q0/lib/qemu │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel │ │ └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel/target │ │ └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel/target/domain-1-avocado-vt-vm1 │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/checkpoint │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1 │ │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes │ │ └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1/monitor.sock │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/dump │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/nvram │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/ram │ ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/save │ └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/snapshot ├── /tmp/virt-qemu-run-KT56Q0/log │ └── /tmp/virt-qemu-run-KT56Q0/log/qemu │ └── /tmp/virt-qemu-run-KT56Q0/log/qemu/avocado-vt-vm1.log └── /tmp/virt-qemu-run-KT56Q0/run ├── /tmp/virt-qemu-run-KT56Q0/run/qemu │ ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/autostarted │ ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/avocado-vt-vm1.pid │ ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/avocado-vt-vm1.xml │ ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/dbus │ ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/driver.pid │ └── /tmp/virt-qemu-run-KT56Q0/run/qemu/slirp └── /tmp/virt-qemu-run-KT56Q0/run/secrets └── /tmp/virt-qemu-run-KT56Q0/run/secrets/driver.pid 24 directories, 11 files S4:Modify config in $EMBEDDED-ROOT/etc/qemu.conf 1.Create a dir: #mkdir -p /var/tmp/vm1-root-dir/etc 2.Enable vnc_tls in qemu.conf: #vim /var/tmp/vm1-root-dir/etc/qemu.conf vnc_tls_x509_cert_dir = "/etc/pki/libvirt-vnc" vnc_tls = 1 3.#setenfore 0 4.Prepare guest xml with vnc graphica device: #vim /xml/vm1.xml ... <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> ... 5.Start guest, could see the conifg in qemu.conf works: # virt-qemu-run -v -d /tmp/vm1.xml -r /var/tmp/vm1-root-dir/ virt-qemu-run: 67: initializing libvirt virt-qemu-run: 1409: initializing signal handlers virt-qemu-run: 1421: preparing event loop thread virt-qemu-run: 1544: opening qemu:///embed?root=/var/tmp/vm1-root-dir/ virt-qemu-run: 51822: starting guest /tmp/vm1.xml virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-09-25T04:03:07.832404Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=no: Unable to access credentials /etc/pki/libvirt-vnc/ca-cert.pem: No such file or directory virt-qemu-run: 1668310: cleaned up, exiting S5.Negative test: 1.Start guest with non-existing guest xml: # virt-qemu-run /xml/secret -d -v virt-qemu-run: 21: initializing libvirt virt-qemu-run: 1406: initializing signal handlers virt-qemu-run: 1563: preparing event loop thread virt-qemu-run: 1694: opening qemu:///embed?root=/tmp/virt-qemu-run-T0KAR0 virt-qemu-run: 52945: starting guest /xml/secret virt-qemu-run: cannot read /xml/secret: Failed to open file “/xml/secret”: No such file or directory virt-qemu-run: 53656: cleaned up, exiting 2.Start guest with existing but not xml format file: # virt-qemu-run /xml/secret-value -d -v virt-qemu-run: 18: initializing libvirt virt-qemu-run: 1415: initializing signal handlers virt-qemu-run: 1777: preparing event loop thread virt-qemu-run: 1908: opening qemu:///embed?root=/tmp/virt-qemu-run-TX8ZQ0 virt-qemu-run: 51943: starting guest /xml/secret-value virt-qemu-run: cannot start VM: (domain_definition):1: Start tag expected, '<' not found 123456 ^ 3.Start 2 guests using the same root dir: # virt-qemu-run /xml/vm2.xml -d -v -r /xml/xml/xml/ virt-qemu-run: 78: initializing libvirt virt-qemu-run: 1464: initializing signal handlers virt-qemu-run: 1477: preparing event loop thread virt-qemu-run: 1610: opening qemu:///embed?root=/xml/xml/xml/ 2020-09-27 02:02:30.575+0000: 106863: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, ) 2020-09-27 02:02:30.575+0000: 106863: info : hostname: dell*.com 2020-09-27 02:02:30.575+0000: 106863: error : virStateInitialize:662 : Initialisation of QEMU state driver failed: Failed to acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable virt-qemu-run: cannot open qemu:///embed?root=/xml/xml/xml/: Failed to acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable virt-qemu-run: 39222: cleaned up, exiting 4.Start guest with luks disk but just provide secret xml: # virt-qemu-run /xml/vm1.xml -d -v -s /xml/vm1.xml virt-qemu-run: 75: initializing libvirt virt-qemu-run: 1406: initializing signal handlers virt-qemu-run: 1672: preparing event loop thread virt-qemu-run: 1792: opening secret:///embed?root=/tmp/virt-qemu-run-RXD8Q0 virt-qemu-run: expected a pair of filenames for --secret argument virt-qemu-run: 3120: cleaned up, exiting 5.Start guest with relative root dir: # virt-qemu-run /xml/vm2.xml -d -v -r xml/ virt-qemu-run: 76: initializing libvirt virt-qemu-run: 1510: initializing signal handlers virt-qemu-run: 1523: preparing event loop thread virt-qemu-run: 1647: opening qemu:///embed?root=xml/ 2020-09-27 02:14:45.121+0000: 107850: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, ) 2020-09-27 02:14:45.121+0000: 107850: info : hostname: dell*.com 2020-09-27 02:14:45.121+0000: 107850: error : virStateInitialize:662 : Initialisation of QEMU state driver failed: Failed to create state dir xml//run/qemu: Invalid argument virt-qemu-run: cannot open qemu:///embed?root=xml/: Failed to create state dir xml//run/qemu: Invalid argument virt-qemu-run: 38563: cleaned up, exiting 6.Start guest using virtual network, but virtnetworkd.socket is inactive: # virt-qemu-run /xml/vm2.xml -d -v -r /xml virt-qemu-run: 66: initializing libvirt virt-qemu-run: 2117: initializing signal handlers virt-qemu-run: 2136: preparing event loop thread virt-qemu-run: 2261: opening qemu:///embed?root=/xml virt-qemu-run: 53686: starting guest /xml/vm2.xml 2020-09-27 02:15:54.798+0000: 108161: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, ) 2020-09-27 02:15:54.798+0000: 108161: info : hostname: dell*.com 2020-09-27 02:15:54.798+0000: 108161: warning : qemuProcessStop:7592 : Unable to release network device '<null>' virt-qemu-run: cannot start VM: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': Connection refused virt-qemu-run: 116110: cleaned up, exiting (In reply to yafu from comment #17) > S5.Negative test: > 1.Start guest with non-existing guest xml: > # virt-qemu-run /xml/secret -d -v > virt-qemu-run: 21: initializing libvirt > virt-qemu-run: 1406: initializing signal handlers > virt-qemu-run: 1563: preparing event loop thread > virt-qemu-run: 1694: opening qemu:///embed?root=/tmp/virt-qemu-run-T0KAR0 > virt-qemu-run: 52945: starting guest /xml/secret > virt-qemu-run: cannot read /xml/secret: Failed to open file “/xml/secret”: > No such file or directory > virt-qemu-run: 53656: cleaned up, exiting > > 2.Start guest with existing but not xml format file: > # virt-qemu-run /xml/secret-value -d -v > virt-qemu-run: 18: initializing libvirt > virt-qemu-run: 1415: initializing signal handlers > virt-qemu-run: 1777: preparing event loop thread > virt-qemu-run: 1908: opening qemu:///embed?root=/tmp/virt-qemu-run-TX8ZQ0 > virt-qemu-run: 51943: starting guest /xml/secret-value > virt-qemu-run: cannot start VM: (domain_definition):1: Start tag expected, > '<' not found > 123456 > ^ > > 3.Start 2 guests using the same root dir: > # virt-qemu-run /xml/vm2.xml -d -v -r /xml/xml/xml/ > virt-qemu-run: 78: initializing libvirt > virt-qemu-run: 1464: initializing signal handlers > virt-qemu-run: 1477: preparing event loop thread > virt-qemu-run: 1610: opening qemu:///embed?root=/xml/xml/xml/ > 2020-09-27 02:02:30.575+0000: 106863: info : libvirt version: 6.6.0, > package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2020-09-17-01:41:02, ) > 2020-09-27 02:02:30.575+0000: 106863: info : hostname: dell*.com > 2020-09-27 02:02:30.575+0000: 106863: error : virStateInitialize:662 : > Initialisation of QEMU state driver failed: Failed to acquire pid file > '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable > virt-qemu-run: cannot open qemu:///embed?root=/xml/xml/xml/: Failed to > acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily > unavailable > virt-qemu-run: 39222: cleaned up, exiting > > 4.Start guest with luks disk but just provide secret xml: > # virt-qemu-run /xml/vm1.xml -d -v -s /xml/vm1.xml > virt-qemu-run: 75: initializing libvirt > virt-qemu-run: 1406: initializing signal handlers > virt-qemu-run: 1672: preparing event loop thread > virt-qemu-run: 1792: opening secret:///embed?root=/tmp/virt-qemu-run-RXD8Q0 > virt-qemu-run: expected a pair of filenames for --secret argument > virt-qemu-run: 3120: cleaned up, exiting > > > 5.Start guest with relative root dir: > # virt-qemu-run /xml/vm2.xml -d -v -r xml/ > virt-qemu-run: 76: initializing libvirt > virt-qemu-run: 1510: initializing signal handlers > virt-qemu-run: 1523: preparing event loop thread > virt-qemu-run: 1647: opening qemu:///embed?root=xml/ > 2020-09-27 02:14:45.121+0000: 107850: info : libvirt version: 6.6.0, > package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2020-09-17-01:41:02, ) > 2020-09-27 02:14:45.121+0000: 107850: info : hostname: dell*.com > 2020-09-27 02:14:45.121+0000: 107850: error : virStateInitialize:662 : > Initialisation of QEMU state driver failed: Failed to create state dir > xml//run/qemu: Invalid argument > virt-qemu-run: cannot open qemu:///embed?root=xml/: Failed to create state > dir xml//run/qemu: Invalid argument > virt-qemu-run: 38563: cleaned up, exiting > > > 6.Start guest using virtual network, but virtnetworkd.socket is inactive: > # virt-qemu-run /xml/vm2.xml -d -v -r /xml > virt-qemu-run: 66: initializing libvirt > virt-qemu-run: 2117: initializing signal handlers > virt-qemu-run: 2136: preparing event loop thread > virt-qemu-run: 2261: opening qemu:///embed?root=/xml > virt-qemu-run: 53686: starting guest /xml/vm2.xml > 2020-09-27 02:15:54.798+0000: 108161: info : libvirt version: 6.6.0, > package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2020-09-17-01:41:02, ) > 2020-09-27 02:15:54.798+0000: 108161: info : hostname: dell*.com > 2020-09-27 02:15:54.798+0000: 108161: warning : qemuProcessStop:7592 : > Unable to release network device '<null>' > virt-qemu-run: cannot start VM: Failed to connect socket to > '/var/run/libvirt/virtnetworkd-sock': Connection refused > virt-qemu-run: 116110: cleaned up, exiting Also negative test for option of 'virt-qemu-run': # virt-qemu-run -t virt-qemu-run: option parsing failed: Unknown option -t (In reply to Daniel Berrangé from comment #7) > This is a tricky policy question. Historically QEMU driver ran inside > libvirtd, and thus we have a policy rule designed around a "virtd_t" type. > > When we're embedding QEMU driver into application code, those applications > need to have equivalent permissions to virtd_t, but we don't want to run the > app as virtd_t in general. Each app that needs to use embedded QEMU will > need its own SELinux policy that inherits much, but not all, of virtd_t type. > > For this, we're really goiing to want to restructure libvirt selinux policy > to separate it out for the modular deamons (virtqemud, replacing libvirtd). > > I think we'll have to consider SELinux enforcement out of scope in the near > term, as I can't see us solving this for 8.3 Hi Daniel, Is there any existing bug which tracks the restructure of libvirt selinux policy? If no existing bug, do you mind me file a bug to track the issue? Thanks, Yan Fu (In reply to yafu from comment #19) > (In reply to Daniel Berrangé from comment #7) > > This is a tricky policy question. Historically QEMU driver ran inside > > libvirtd, and thus we have a policy rule designed around a "virtd_t" type. > > > > When we're embedding QEMU driver into application code, those applications > > need to have equivalent permissions to virtd_t, but we don't want to run the > > app as virtd_t in general. Each app that needs to use embedded QEMU will > > need its own SELinux policy that inherits much, but not all, of virtd_t type. > > > > For this, we're really goiing to want to restructure libvirt selinux policy > > to separate it out for the modular deamons (virtqemud, replacing libvirtd). > > > > I think we'll have to consider SELinux enforcement out of scope in the near > > term, as I can't see us solving this for 8.3 > > Hi Daniel, > > Is there any existing bug which tracks the restructure of libvirt selinux > policy? If no existing bug, do you mind me file a bug to track the issue? > > > Thanks, > Yan Fu Sorry, forgot to add needinfo for comment 19. Hi Daniel, I have another question. When using 'virt-qemu-run' to start guest in a new namespace, it fails. But using 'virsh start' can start guest successfully. Would you help to check the issue please? Thanks. #unshare --mount --fork --pid /bin/bash # virt-qemu-run /tmp/83.xml -d -v virt-qemu-run: 10: initializing libvirt virt-qemu-run: 1217: initializing signal handlers virt-qemu-run: 1374: preparing event loop thread virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0 virt-qemu-run: 38698: starting guest /tmp/83.xml virt-qemu-run: cannot start VM: Process with ID 215 does not exist. virt-qemu-run: 1131315: cleaned up, exiting # virsh start 83 Domain 83 started (In reply to yafu from comment #21) > Hi Daniel, > > I have another question. When using 'virt-qemu-run' to start guest in a new > namespace, it fails. But using 'virsh start' can start guest successfully. > Would you help to check the issue please? Thanks. > > #unshare --mount --fork --pid /bin/bash > > # virt-qemu-run /tmp/83.xml -d -v > virt-qemu-run: 10: initializing libvirt > virt-qemu-run: 1217: initializing signal handlers > virt-qemu-run: 1374: preparing event loop thread > virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0 > virt-qemu-run: 38698: starting guest /tmp/83.xml > virt-qemu-run: cannot start VM: Process with ID 215 does not exist. > virt-qemu-run: 1131315: cleaned up, exiting > > # virsh start 83 > Domain 83 started THis scenario doesn't really make sense - "virsh start" here is connecting to "qemu:///system", which is completely separate from the embedded driver. I presume you have a guest called "83" defined in your qemu:///system driver instance already (In reply to yafu from comment #19) > (In reply to Daniel Berrangé from comment #7) > > This is a tricky policy question. Historically QEMU driver ran inside > > libvirtd, and thus we have a policy rule designed around a "virtd_t" type. > > > > When we're embedding QEMU driver into application code, those applications > > need to have equivalent permissions to virtd_t, but we don't want to run the > > app as virtd_t in general. Each app that needs to use embedded QEMU will > > need its own SELinux policy that inherits much, but not all, of virtd_t type. > > > > For this, we're really goiing to want to restructure libvirt selinux policy > > to separate it out for the modular deamons (virtqemud, replacing libvirtd). > > > > I think we'll have to consider SELinux enforcement out of scope in the near > > term, as I can't see us solving this for 8.3 > > Hi Daniel, > > Is there any existing bug which tracks the restructure of libvirt selinux > policy? If no existing bug, do you mind me file a bug to track the issue? There is existing work planned to modularize the Libvirt SELinux policy, but that is not likely to be done in context of RHEL-8 due to its disruption It is tracked in this Jira plan: https://issues.redhat.com/browse/SELINUX-2283 (In reply to Daniel Berrangé from comment #22) > (In reply to yafu from comment #21) > > Hi Daniel, > > > > I have another question. When using 'virt-qemu-run' to start guest in a new > > namespace, it fails. But using 'virsh start' can start guest successfully. > > Would you help to check the issue please? Thanks. > > > > #unshare --mount --fork --pid /bin/bash > > > > # virt-qemu-run /tmp/83.xml -d -v > > virt-qemu-run: 10: initializing libvirt > > virt-qemu-run: 1217: initializing signal handlers > > virt-qemu-run: 1374: preparing event loop thread > > virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0 > > virt-qemu-run: 38698: starting guest /tmp/83.xml > > virt-qemu-run: cannot start VM: Process with ID 215 does not exist. > > virt-qemu-run: 1131315: cleaned up, exiting > > > > # virsh start 83 > > Domain 83 started > > THis scenario doesn't really make sense - "virsh start" here is connecting > to "qemu:///system", which is completely separate from the embedded driver. > I presume you have a guest called "83" defined in your qemu:///system driver > instance already Yes, i have a guest called '83' defined. But it also can start guest successfully by 'virsh create' if no guest defined. > > (In reply to yafu from comment #19) > > (In reply to Daniel Berrangé from comment #7) > > > This is a tricky policy question. Historically QEMU driver ran inside > > > libvirtd, and thus we have a policy rule designed around a "virtd_t" type. > > > > > > When we're embedding QEMU driver into application code, those applications > > > need to have equivalent permissions to virtd_t, but we don't want to run the > > > app as virtd_t in general. Each app that needs to use embedded QEMU will > > > need its own SELinux policy that inherits much, but not all, of virtd_t type. > > > > > > For this, we're really goiing to want to restructure libvirt selinux policy > > > to separate it out for the modular deamons (virtqemud, replacing libvirtd). > > > > > > I think we'll have to consider SELinux enforcement out of scope in the near > > > term, as I can't see us solving this for 8.3 > > > > Hi Daniel, > > > > Is there any existing bug which tracks the restructure of libvirt selinux > > policy? If no existing bug, do you mind me file a bug to track the issue? > > There is existing work planned to modularize the Libvirt SELinux policy, but > that is not likely to be done in context of RHEL-8 due to its disruption > > It is tracked in this Jira plan: > > https://issues.redhat.com/browse/SELINUX-2283 Test with session mode. 1.#su - test 2.$ virt-qemu-run -d -v /tmp/83.xml virt-qemu-run: 14: initializing libvirt virt-qemu-run: 1925: initializing signal handlers virt-qemu-run: 2034: preparing event loop thread virt-qemu-run: 2103: opening qemu:///embed?root=/tmp/virt-qemu-run-WQZKS0 virt-qemu-run: 19541: starting guest /tmp/83.xml virt-qemu-run: 1379396: guest running, Ctrl-C to stop now virt-qemu-run: XML: <domain type='kvm' id='1'> <name>83</name> <uuid>9350639d-1c8a-4f51-a4a6-4eaf8eabe83e</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://redhat.com/rhel/8.3"/> </libosinfo:libosinfo> </metadata> <memory unit='KiB'>1572864</memory> <currentMemory unit='KiB'>1572864</currentMemory> <vcpu placement='static'>2</vcpu> 3.Check the qemu process: $ ps aux | grep -i qemu test 410248 3.8 0.0 422264 30576 pts/1 Sl+ 05:32 0:00 virt-qemu-run -d -v /tmp/83.xml test 410280 95.0 0.1 2548432 56888 ? Sl 05:32 0:04 /usr/libexec/qemu-kvm -name guest=83,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/tmp/virt-qemu-run-WQZKS0/lib/qemu/domain-1-83/master-key.aes... Created attachment 1720192 [details]
code coverage report for patchset
From the code coverage report for patchset, we could see most code of the patches were tested except some negative test for error handler. Move the bug to VERIFIED according to comment 13 - comment 26. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137 Dan, while you were developing this, did you try to make any patches for libguestfs? And if so do you still have those? (In reply to Richard W.M. Jones from comment #31) > Dan, while you were developing this, did you try to make any > patches for libguestfs? And if so do you still have those? I did experiment with libguestfs patches, but I can't find them now. The hard bit I hit was the need to have a thread servicing an event loop. Because libguestfs is a library, it is a bit questionable as to whether it should spawn a thread for a event loop itself, and even if it does, how can it know whether the app has already spawned one for libvirt. I need to see if we can eliminate the need for the event loop entirely. I think it is possible now that we have a private thread per VM for handling the QEMU monitor. The only thing left needing the event loop was domain event dispatch and we might be able to solve that another way. |