Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1772943 - Support embedding the QEMU driver in client apps directly (TechPreview)
Summary: Support embedding the QEMU driver in client apps directly (TechPreview)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 8.2
Assignee: Daniel Berrangé
QA Contact: yafu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-15 16:38 UTC by Ademar Reis
Modified: 2020-11-17 17:46 UTC (History)
6 users (show)

Fixed In Version: libvirt-6.2.0-1.el8
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-17 17:45:34 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
code coverage report for patchset (27.92 KB, text/html)
2020-10-09 09:47 UTC, yafu 		no flags Details
View All

Description Ademar Reis 2019-11-15 16:38:09 UTC 
From https://www.redhat.com/archives/libvir-list/2019-May/msg00467.html

The above patch series proposes a new way to use the libvirt QEMU driver from applications by dynamically loading the QEMU driver module into the application process, instead of having it run inside a libvirt daemon.

This has some advantages and some disadvantages compared to the traditional way of using QEMU, but might open up some interesting new architectural options for applications using libvirt.

Specifically some benefits are

We can use this to embed the QEMU driver in unit tests allowing the full range of driver functionality to be exercised during unit testing without interfering with the host OS libvirtd.
Apps like libguestfs can embed the QEMU driver to allow them to spawn VMs that are immediate children of their app, thus inheriting process context, as well as hiding these "service VM" from the main libvirtd. This would avoid a common irritation where libguestfs VMs suddenly appear & disappear in virt-manager's VM list.
Some key downsides are

If either the application code or the QEMU drive code crashes, it takes out both. It is thus less reliable than a setup using a separate libvirt daemon
There is no ability to interact with the virtual machines run from the embedded driver using other libvirt tools. This has a negative impact on common troubleshooting procedures on production hosts. eg using "virsh" to inspect / interact with a running VM is no longer possible.
 

In some ways it overlaps with the previously proposed "libvirt shim" concept https://www.redhat.com/archives/libvir-list/2017-November/msg00526.html

The libvirt shim, however, was still intended to (optionally) have VMs registered with the central libvirtd daemon, and so accessible to tools like virsh. This eliminates one of the downsides of the embedded mode, but also looses some of the benefits. 

Thus the embedded mode should really be considered distinct/complementary from the shim concept.
Comment 1 Daniel Berrangé 2020-02-07 14:25:08 UTC 
Initial impl is merged upstream with

commit 217b564a77041dd5877527caf86bd5bb570d6f44
Author: Daniel P. Berrangé <berrange>
Date:   Wed Jan 29 11:17:54 2020 +0000

    docs: add warning about virt-qemu-run being experimental


commit d6006672788ec0f0290d35c76ceb9672476d1ea8
Author: Daniel P. Berrangé <berrange>
Date:   Fri May 17 13:01:59 2019 +0100

    qemu: introduce a new "virt-qemu-run" program
    
commit 3e9076e777aff2f4b08330ed17e559fcfb6b3529
Author: Daniel P. Berrangé <berrange>
Date:   Fri May 17 12:35:57 2019 +0100

    secrets: add support for running secret driver in embedded mode

commit 068efae5b1a9efeea4a9c3bc0ae80747da5024fb
Author: Daniel P. Berrangé <berrange>
Date:   Fri May 17 12:35:57 2019 +0100

    qemu: add support for running QEMU driver in embedded mode
    
commit 88446e07b2050c30ed07a2a7d26507674c8abe01
Author: Daniel P. Berrangé <berrange>
Date:   Fri May 17 12:42:04 2019 +0100

    libvirt: support an "embed" URI path selector for opening drivers
Comment 6 yafu 2020-06-09 09:58:35 UTC 
Hi Daniel,

Failed to connect virtlogd when trying to start guest with 'virt-qemu-run'. Could you help to check whether it's a bug of selinux-policy or not please? Thanks.

Test pkg:
libvirt-daemon-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64


1.# virt-qemu-run vm1.xml 
virt-qemu-run: cannot start VM: can't connect to virtlogd: Cannot recv data: Connection reset by peer

2.Check the audit log:
#  ausearch -m AVC -ts boot | grep -i virtlogd
type=PROCTITLE msg=audit(1591695650.402:285): proctitle="/usr/sbin/virtlogd"
type=SYSCALL msg=audit(1591695650.402:285): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=55d680a238e0 a2=0 a3=0 items=0 ppid=1 pid=37358 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="virtlogd" exe="/usr/sbin/virtlogd" subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1591695650.402:285): avc:  denied  { search } for  pid=37358 comm="virtlogd" name="37328" dev="proc" ino=52634 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dir permissive=0
type=PROCTITLE msg=audit(1591695975.266:291): proctitle="/usr/sbin/virtlogd"
Comment 7 Daniel Berrangé 2020-06-15 14:07:20 UTC 
This is a tricky policy question.  Historically QEMU driver ran inside libvirtd, and thus we have a policy rule designed around a "virtd_t" type.

When we're embedding QEMU driver into application code, those applications need to have equivalent permissions to virtd_t, but we don't want to run the app as virtd_t in general. Each app that needs to use embedded QEMU will need its own SELinux policy that inherits much, but not all, of virtd_t type.

For this, we're really goiing to want to restructure libvirt selinux policy to separate it out for the modular deamons (virtqemud, replacing libvirtd). 

I think we'll have to consider SELinux enforcement out of scope in the near term, as I can't see us solving this for 8.3
Comment 8 yafu 2020-07-21 09:35:28 UTC 
Hi Daniel,

I changed the setting of qemu.conf and it seems qemu.conf does not work for embedding QEMU driver. Is that expected behavior?
Comment 9 Daniel Berrangé 2020-07-21 09:37:44 UTC 
The embedded QEMU driver is intended to be *completely* isolated from the main libvirtd QEMU driver.  So yes, the /etc/libvirt/qemu.conf will have no effect on the embedded driver.  The embedded driver has its own private root directory with its own location, $EMBEDDED-ROOT/etc/qemu.conf file https://libvirt.org/drvqemu.html#embedTree
Comment 10 yafu 2020-07-21 12:42:53 UTC 
(In reply to Daniel Berrangé from comment #9)
> The embedded QEMU driver is intended to be *completely* isolated from the
> main libvirtd QEMU driver.  So yes, the /etc/libvirt/qemu.conf will have no
> effect on the embedded driver.  The embedded driver has its own private root
> directory with its own location, $EMBEDDED-ROOT/etc/qemu.conf file
> https://libvirt.org/drvqemu.html#embedTree

Thanks for your quickly reply. 

I have another question about starting guest with luks image. I setup the env as the following steps:
1. prepare a qcow2 luks image:
#qemu-img create --object secret,id=sec0,data=123456,for‐ -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G

2.Check the image:
# qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2
No errors were found on the image.
Image end offset: 2359296

3.Prepare secret xml and secret value file:
#cat /xml/secret.xml
  <secret ephemeral='no' private='yes'>
         <description>LUKS Sample Secret</description>
         <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid>
         <usage type='volume'>
            <volume>/var/lib/libvirt/images/luks.img</volume>
         </usage>
      </secret> 

#cat /xml/secret-value
123456

4.Prepare a guest xml with the luks image(the same uuid with the secret xml):
#cat /tmp/vm1.xml
...
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/>
      <source file='/var/lib/libvirt/images/luks.qcow2' index='1'>
              <encryption format='luks'>
        <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
</encryption>
     </source>
...
</disk>
...

5.Start guest:
# virt-qemu-run -v -d /tmp/vm1.xml -s /xml/secret.xml,/xml/secret-value 
virt-qemu-run: 77: initializing libvirt
virt-qemu-run: 1165: initializing signal handlers
virt-qemu-run: 1339: preparing event loop thread
virt-qemu-run: 1393: opening secret:///embed?root=/tmp/virt-qemu-run-CKP3N0
virt-qemu-run: 2259: loading secret /xml/secret.xml and /xml/secret-value
virt-qemu-run: 2848: opening qemu:///embed?root=/tmp/virt-qemu-run-CKP3N0
virt-qemu-run: 30921: starting guest /tmp/vm1.xml
virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-07-21T12:38:19.039565Z qemu-kvm: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","encrypt":{"format":"luks","key-secret":"libvirt-1-format-encryption-secret0"},"file":"libvirt-1-storage","backing":null}: Invalid password, cannot unlock any keyslot
virt-qemu-run: 3125617: cleaned up, exiting

It seems the raw value of secret is wrong. Did i skip any steps or was it anything wrong with the secret value file?
Comment 11 Daniel Berrangé 2020-07-21 15:23:59 UTC 
Do you have a trailing newline in your  secret-value file ?  eg  use "echo -n 123456 > secret-value", not a plain "echo 123456"
Comment 12 yafu 2020-07-22 03:00:04 UTC 
(In reply to Daniel Berrangé from comment #11)
> Do you have a trailing newline in your  secret-value file ?  eg  use "echo
> -n 123456 > secret-value", not a plain "echo 123456"

Use "echo -n 123456" > secret-value" works.
Thanks a lot.
Comment 13 yafu 2020-09-24 07:50:56 UTC 
Test with libvirt-daemon-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64.

S1.Start guest without specifying root dir:
1.#setenforce 0

2.#virt-qemu-run -d -v /tmp/vm1.xml 
virt-qemu-run: 17: initializing libvirt
virt-qemu-run: 2230: initializing signal handlers
virt-qemu-run: 2533: preparing event loop thread
virt-qemu-run: 2681: opening qemu:///embed?root=/tmp/virt-qemu-run-2K72Q0
virt-qemu-run: 36029: starting guest /tmp/vm1.xml
virt-qemu-run: 1963123: guest running, Ctrl-C to stop now
virt-qemu-run: XML: <domain type='kvm' id='1'>
  <name>avocado-vt-vm1</name>
  <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid>
  <metadata>
...

3.Check the root directory created by libvirt in another terminal:
#tree -f
.
├── ./cache
│   └── ./cache/qemu
│       └── ./cache/qemu/capabilities
│           └── ./cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml
├── ./lib
│   └── ./lib/qemu
│       ├── ./lib/qemu/channel
│       │   └── ./lib/qemu/channel/target
│       │       └── ./lib/qemu/channel/target/domain-1-avocado-vt-vm1
│       ├── ./lib/qemu/checkpoint
│       ├── ./lib/qemu/domain-1-avocado-vt-vm1
│       │   ├── ./lib/qemu/domain-1-avocado-vt-vm1/master-key.aes
│       │   └── ./lib/qemu/domain-1-avocado-vt-vm1/monitor.sock
│       ├── ./lib/qemu/dump
│       ├── ./lib/qemu/nvram
│       ├── ./lib/qemu/ram
│       ├── ./lib/qemu/save
│       └── ./lib/qemu/snapshot
├── ./log
│   └── ./log/qemu
│       └── ./log/qemu/avocado-vt-vm1.log
└── ./run
    └── ./run/qemu
        ├── ./run/qemu/autostarted
        ├── ./run/qemu/avocado-vt-vm1.pid
        ├── ./run/qemu/avocado-vt-vm1.xml
        ├── ./run/qemu/dbus
        ├── ./run/qemu/driver.pid
        └── ./run/qemu/slirp

21 directories, 8 files

4.Check the qemu proceess:
#ps aux | grep -i qemu
# ps aux | grep -i qemu
root       59773  0.0  0.0 437812 27684 pts/0    Sl+  03:25   0:00 virt-qemu-run -d -v /tmp/vm1.xml
qemu       59817  3.1  1.9 3076440 1242760 ?     Sl   03:25   0:40 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/tmp/virt-qemu-run-2K72Q0/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes ...

5.Enter 'Ctrl-C' to terminate 'virt-qemu-run' in step 2:
^Cvirt-qemu-run: 1425433879: cleaned up, exiting

6.Check the root dir:
# ls /tmp/virt-qemu-run-2K72Q0/
ls: cannot access '/tmp/virt-qemu-run-2K72Q0/': No such file or directory

7.Check the qemu process:
# pgrep qemu
no output
Comment 14 yafu 2020-09-24 09:08:56 UTC 
S2.Start guest with specifying root dir:
1.#setenfore 0

2.# virt-qemu-run -d -v /tmp/vm1.xml --root=/var/tmp/vm1
virt-qemu-run: 66: initializing libvirt
virt-qemu-run: 2278: initializing signal handlers
virt-qemu-run: 2298: preparing event loop thread
virt-qemu-run: 2450: opening qemu:///embed?root=/var/tmp/vm1
virt-qemu-run: 40390: starting guest /tmp/vm1.xml
virt-qemu-run: 2452882: guest running, Ctrl-C to stop now
virt-qemu-run: XML: <domain type='kvm' id='1'>
  <name>avocado-vt-vm1</name>
  <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid>
  <metadata>
...

3.Check the root directory:
# tree -f /var/tmp/vm1/
/var/tmp/vm1
├── /var/tmp/vm1/cache
│   └── /var/tmp/vm1/cache/qemu
│       └── /var/tmp/vm1/cache/qemu/capabilities
│           └── /var/tmp/vm1/cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml
├── /var/tmp/vm1/lib
│   └── /var/tmp/vm1/lib/qemu
│       ├── /var/tmp/vm1/lib/qemu/channel
│       │   └── /var/tmp/vm1/lib/qemu/channel/target
│       │       └── /var/tmp/vm1/lib/qemu/channel/target/domain-1-avocado-vt-vm1
│       ├── /var/tmp/vm1/lib/qemu/checkpoint
│       ├── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1
│       │   ├── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes
│       │   └── /var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/monitor.sock
│       ├── /var/tmp/vm1/lib/qemu/dump
│       ├── /var/tmp/vm1/lib/qemu/nvram
│       ├── /var/tmp/vm1/lib/qemu/ram
│       ├── /var/tmp/vm1/lib/qemu/save
│       └── /var/tmp/vm1/lib/qemu/snapshot
├── /var/tmp/vm1/log
│   └── /var/tmp/vm1/log/qemu
│       └── /var/tmp/vm1/log/qemu/avocado-vt-vm1.log
└── /var/tmp/vm1/run
    └── /var/tmp/vm1/run/qemu
        ├── /var/tmp/vm1/run/qemu/autostarted
        ├── /var/tmp/vm1/run/qemu/avocado-vt-vm1.pid
        ├── /var/tmp/vm1/run/qemu/avocado-vt-vm1.xml
        ├── /var/tmp/vm1/run/qemu/dbus
        ├── /var/tmp/vm1/run/qemu/driver.pid
        └── /var/tmp/vm1/run/qemu/slirp

21 directories, 8 files

4.Check qemu process:
# ps aux | grep -i qemu
root       61100  0.1  0.0 437684 27612 pts/0    Sl+  05:05   0:00 virt-qemu-run -d -v /tmp/vm1.xml --root=/var/tmp/vm1
qemu       61218 32.1  1.9 3136856 1242860 ?     Sl   05:05   0:30 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/tmp/vm1/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes...

5.Enter 'Ctrl-C' to terminate 'virt-qemu-run' in step 2:
^Cvirt-qemu-run: 130260216: cleaned up, exiting

6.Check the root dir, the root dir still exists:
# ls /var/tmp/vm1/
cache  lib  log  run

7.Check the qemu process:
# pgrep qemu
no output
Comment 15 yafu 2020-09-25 02:42:31 UTC 
S3: Start guest with luks image:
1. prepare a qcow2 luks image:
#qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0 /var/lib/libvirt/images/luks.qcow2 1G

2.Check the image:
# qemu-img check --object secret,id=sec0,data=123456 --image-opts driver=qcow2,encrypt.key-secret=sec0,file.filename=/var/lib/libvirt/images/luks.qcow2
No errors were found on the image.
Image end offset: 2359296

3.Prepare secret xml and secret value file:
#cat /xml/secret.xml
  <secret ephemeral='no' private='yes'>
         <description>LUKS Sample Secret</description>
         <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid>
         <usage type='volume'>
            <volume>/var/lib/libvirt/images/luks.img</volume>
         </usage>
      </secret> 

#echo -n 123456 > /xml/secret-value 

4.Prepare a guest xml with the luks image(the same uuid with the secret xml):
#cat /tmp/vm1.xml
...
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='threads' copy_on_read='off'/>
      <source file='/var/lib/libvirt/images/luks.qcow2' index='1'>
              <encryption format='luks'>
        <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
</encryption>
     </source>
...
</disk>
...

5.#setenfore 0

6.Start guest:
# virt-qemu-run -v -d /tmp/vm1.xml -s /xml/secret.xml,/xml/secret-value 
virt-qemu-run: 68: initializing libvirt
virt-qemu-run: 2048: initializing signal handlers
virt-qemu-run: 2362: preparing event loop thread
virt-qemu-run: 2491: opening secret:///embed?root=/tmp/virt-qemu-run-G224Q0
virt-qemu-run: 3852: loading secret /xml/secret.xml and /xml/secret-value
virt-qemu-run: 4852: opening qemu:///embed?root=/tmp/virt-qemu-run-G224Q0
virt-qemu-run: 34810: starting guest /tmp/vm1.xml
virt-qemu-run: 2230569: guest running, Ctrl-C to stop now
virt-qemu-run: XML: <domain type='kvm' id='1'>
  <name>avocado-vt-vm1</name>
  <uuid>6443e5ee-acf9-44cb-87f1-066cb8346721</uuid>
  <metadata>
...

7.Check qemu process:
#ps aux | grep -i secret
/usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0...-object secret,id=libvirt-1-format-encryption-secret0,data=3ZOmoPbAVVWKBW2pkY6Ivg==,keyid=masterKey0,iv=WVqjT/Q9WIun1FwkkrqUvw==,format=base64 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/luks.qcow2","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},...

8.Check the root dir, could see the secret info:
# tree -f /tmp/virt-qemu-run-KT56Q0
/tmp/virt-qemu-run-KT56Q0
├── /tmp/virt-qemu-run-KT56Q0/cache
│   └── /tmp/virt-qemu-run-KT56Q0/cache/qemu
│       └── /tmp/virt-qemu-run-KT56Q0/cache/qemu/capabilities
│           └── /tmp/virt-qemu-run-KT56Q0/cache/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml
├── /tmp/virt-qemu-run-KT56Q0/etc
│   └── /tmp/virt-qemu-run-KT56Q0/etc/secrets
│       ├── /tmp/virt-qemu-run-KT56Q0/etc/secrets/f981dd17-143f-45bc-88e6-ed1fe20ce9da.base64  ***
│       └── /tmp/virt-qemu-run-KT56Q0/etc/secrets/f981dd17-143f-45bc-88e6-ed1fe20ce9da.xml   ***
├── /tmp/virt-qemu-run-KT56Q0/lib
│   └── /tmp/virt-qemu-run-KT56Q0/lib/qemu
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel
│       │   └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel/target
│       │       └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/channel/target/domain-1-avocado-vt-vm1
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/checkpoint
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1
│       │   ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1/master-key.aes
│       │   └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/domain-1-avocado-vt-vm1/monitor.sock
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/dump
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/nvram
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/ram
│       ├── /tmp/virt-qemu-run-KT56Q0/lib/qemu/save
│       └── /tmp/virt-qemu-run-KT56Q0/lib/qemu/snapshot
├── /tmp/virt-qemu-run-KT56Q0/log
│   └── /tmp/virt-qemu-run-KT56Q0/log/qemu
│       └── /tmp/virt-qemu-run-KT56Q0/log/qemu/avocado-vt-vm1.log
└── /tmp/virt-qemu-run-KT56Q0/run
    ├── /tmp/virt-qemu-run-KT56Q0/run/qemu
    │   ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/autostarted
    │   ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/avocado-vt-vm1.pid
    │   ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/avocado-vt-vm1.xml
    │   ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/dbus
    │   ├── /tmp/virt-qemu-run-KT56Q0/run/qemu/driver.pid
    │   └── /tmp/virt-qemu-run-KT56Q0/run/qemu/slirp
    └── /tmp/virt-qemu-run-KT56Q0/run/secrets
        └── /tmp/virt-qemu-run-KT56Q0/run/secrets/driver.pid

24 directories, 11 files
Comment 16 yafu 2020-09-25 04:05:15 UTC 
S4：Modify config in $EMBEDDED-ROOT/etc/qemu.conf
1.Create a dir:
#mkdir -p /var/tmp/vm1-root-dir/etc

2.Enable vnc_tls in qemu.conf:
#vim /var/tmp/vm1-root-dir/etc/qemu.conf
vnc_tls_x509_cert_dir = "/etc/pki/libvirt-vnc"
vnc_tls = 1

3.#setenfore 0

4.Prepare guest xml with vnc graphica device:
#vim /xml/vm1.xml
...
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
...

5.Start guest, could see the conifg in qemu.conf works:
# virt-qemu-run -v -d /tmp/vm1.xml -r /var/tmp/vm1-root-dir/
virt-qemu-run: 67: initializing libvirt
virt-qemu-run: 1409: initializing signal handlers
virt-qemu-run: 1421: preparing event loop thread
virt-qemu-run: 1544: opening qemu:///embed?root=/var/tmp/vm1-root-dir/
virt-qemu-run: 51822: starting guest /tmp/vm1.xml
virt-qemu-run: cannot start VM: internal error: process exited while connecting to monitor: 2020-09-25T04:03:07.832404Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=no: Unable to access credentials /etc/pki/libvirt-vnc/ca-cert.pem: No such file or directory
virt-qemu-run: 1668310: cleaned up, exiting
Comment 17 yafu 2020-09-27 02:17:52 UTC 
S5.Negative test:
1.Start guest with non-existing guest xml:
# virt-qemu-run /xml/secret -d -v 
virt-qemu-run: 21: initializing libvirt
virt-qemu-run: 1406: initializing signal handlers
virt-qemu-run: 1563: preparing event loop thread
virt-qemu-run: 1694: opening qemu:///embed?root=/tmp/virt-qemu-run-T0KAR0
virt-qemu-run: 52945: starting guest /xml/secret
virt-qemu-run: cannot read /xml/secret: Failed to open file “/xml/secret”: No such file or directory
virt-qemu-run: 53656: cleaned up, exiting

2.Start guest with existing but not xml format file:
# virt-qemu-run /xml/secret-value -d -v 
virt-qemu-run: 18: initializing libvirt
virt-qemu-run: 1415: initializing signal handlers
virt-qemu-run: 1777: preparing event loop thread
virt-qemu-run: 1908: opening qemu:///embed?root=/tmp/virt-qemu-run-TX8ZQ0
virt-qemu-run: 51943: starting guest /xml/secret-value
virt-qemu-run: cannot start VM: (domain_definition):1: Start tag expected, '<' not found
123456
^

3.Start 2 guests using the same root dir:
# virt-qemu-run /xml/vm2.xml -d -v -r /xml/xml/xml/
virt-qemu-run: 78: initializing libvirt
virt-qemu-run: 1464: initializing signal handlers
virt-qemu-run: 1477: preparing event loop thread
virt-qemu-run: 1610: opening qemu:///embed?root=/xml/xml/xml/
2020-09-27 02:02:30.575+0000: 106863: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, )
2020-09-27 02:02:30.575+0000: 106863: info : hostname: dell*.com
2020-09-27 02:02:30.575+0000: 106863: error : virStateInitialize:662 : Initialisation of QEMU state driver failed: Failed to acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable
virt-qemu-run: cannot open qemu:///embed?root=/xml/xml/xml/: Failed to acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable
virt-qemu-run: 39222: cleaned up, exiting

4.Start guest with luks disk but just provide secret xml:
# virt-qemu-run /xml/vm1.xml -d -v -s /xml/vm1.xml
virt-qemu-run: 75: initializing libvirt
virt-qemu-run: 1406: initializing signal handlers
virt-qemu-run: 1672: preparing event loop thread
virt-qemu-run: 1792: opening secret:///embed?root=/tmp/virt-qemu-run-RXD8Q0
virt-qemu-run: expected a pair of filenames for --secret argument
virt-qemu-run: 3120: cleaned up, exiting


5.Start guest with relative root dir:
# virt-qemu-run /xml/vm2.xml -d -v -r xml/
virt-qemu-run: 76: initializing libvirt
virt-qemu-run: 1510: initializing signal handlers
virt-qemu-run: 1523: preparing event loop thread
virt-qemu-run: 1647: opening qemu:///embed?root=xml/
2020-09-27 02:14:45.121+0000: 107850: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, )
2020-09-27 02:14:45.121+0000: 107850: info : hostname: dell*.com
2020-09-27 02:14:45.121+0000: 107850: error : virStateInitialize:662 : Initialisation of QEMU state driver failed: Failed to create state dir xml//run/qemu: Invalid argument
virt-qemu-run: cannot open qemu:///embed?root=xml/: Failed to create state dir xml//run/qemu: Invalid argument
virt-qemu-run: 38563: cleaned up, exiting


6.Start guest using virtual network, but virtnetworkd.socket is inactive:
# virt-qemu-run /xml/vm2.xml -d -v -r /xml
virt-qemu-run: 66: initializing libvirt
virt-qemu-run: 2117: initializing signal handlers
virt-qemu-run: 2136: preparing event loop thread
virt-qemu-run: 2261: opening qemu:///embed?root=/xml
virt-qemu-run: 53686: starting guest /xml/vm2.xml
2020-09-27 02:15:54.798+0000: 108161: info : libvirt version: 6.6.0, package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-17-01:41:02, )
2020-09-27 02:15:54.798+0000: 108161: info : hostname: dell*.com
2020-09-27 02:15:54.798+0000: 108161: warning : qemuProcessStop:7592 : Unable to release network device '<null>'
virt-qemu-run: cannot start VM: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': Connection refused
virt-qemu-run: 116110: cleaned up, exiting
Comment 18 yafu 2020-09-27 02:38:14 UTC 
(In reply to yafu from comment #17)
> S5.Negative test:
> 1.Start guest with non-existing guest xml:
> # virt-qemu-run /xml/secret -d -v 
> virt-qemu-run: 21: initializing libvirt
> virt-qemu-run: 1406: initializing signal handlers
> virt-qemu-run: 1563: preparing event loop thread
> virt-qemu-run: 1694: opening qemu:///embed?root=/tmp/virt-qemu-run-T0KAR0
> virt-qemu-run: 52945: starting guest /xml/secret
> virt-qemu-run: cannot read /xml/secret: Failed to open file “/xml/secret”:
> No such file or directory
> virt-qemu-run: 53656: cleaned up, exiting
> 
> 2.Start guest with existing but not xml format file:
> # virt-qemu-run /xml/secret-value -d -v 
> virt-qemu-run: 18: initializing libvirt
> virt-qemu-run: 1415: initializing signal handlers
> virt-qemu-run: 1777: preparing event loop thread
> virt-qemu-run: 1908: opening qemu:///embed?root=/tmp/virt-qemu-run-TX8ZQ0
> virt-qemu-run: 51943: starting guest /xml/secret-value
> virt-qemu-run: cannot start VM: (domain_definition):1: Start tag expected,
> '<' not found
> 123456
> ^
> 
> 3.Start 2 guests using the same root dir:
> # virt-qemu-run /xml/vm2.xml -d -v -r /xml/xml/xml/
> virt-qemu-run: 78: initializing libvirt
> virt-qemu-run: 1464: initializing signal handlers
> virt-qemu-run: 1477: preparing event loop thread
> virt-qemu-run: 1610: opening qemu:///embed?root=/xml/xml/xml/
> 2020-09-27 02:02:30.575+0000: 106863: info : libvirt version: 6.6.0,
> package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> 2020-09-17-01:41:02, )
> 2020-09-27 02:02:30.575+0000: 106863: info : hostname: dell*.com
> 2020-09-27 02:02:30.575+0000: 106863: error : virStateInitialize:662 :
> Initialisation of QEMU state driver failed: Failed to acquire pid file
> '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily unavailable
> virt-qemu-run: cannot open qemu:///embed?root=/xml/xml/xml/: Failed to
> acquire pid file '/xml/xml/xml//run/qemu/driver.pid': Resource temporarily
> unavailable
> virt-qemu-run: 39222: cleaned up, exiting
> 
> 4.Start guest with luks disk but just provide secret xml:
> # virt-qemu-run /xml/vm1.xml -d -v -s /xml/vm1.xml
> virt-qemu-run: 75: initializing libvirt
> virt-qemu-run: 1406: initializing signal handlers
> virt-qemu-run: 1672: preparing event loop thread
> virt-qemu-run: 1792: opening secret:///embed?root=/tmp/virt-qemu-run-RXD8Q0
> virt-qemu-run: expected a pair of filenames for --secret argument
> virt-qemu-run: 3120: cleaned up, exiting
> 
> 
> 5.Start guest with relative root dir:
> # virt-qemu-run /xml/vm2.xml -d -v -r xml/
> virt-qemu-run: 76: initializing libvirt
> virt-qemu-run: 1510: initializing signal handlers
> virt-qemu-run: 1523: preparing event loop thread
> virt-qemu-run: 1647: opening qemu:///embed?root=xml/
> 2020-09-27 02:14:45.121+0000: 107850: info : libvirt version: 6.6.0,
> package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> 2020-09-17-01:41:02, )
> 2020-09-27 02:14:45.121+0000: 107850: info : hostname: dell*.com
> 2020-09-27 02:14:45.121+0000: 107850: error : virStateInitialize:662 :
> Initialisation of QEMU state driver failed: Failed to create state dir
> xml//run/qemu: Invalid argument
> virt-qemu-run: cannot open qemu:///embed?root=xml/: Failed to create state
> dir xml//run/qemu: Invalid argument
> virt-qemu-run: 38563: cleaned up, exiting
> 
> 
> 6.Start guest using virtual network, but virtnetworkd.socket is inactive:
> # virt-qemu-run /xml/vm2.xml -d -v -r /xml
> virt-qemu-run: 66: initializing libvirt
> virt-qemu-run: 2117: initializing signal handlers
> virt-qemu-run: 2136: preparing event loop thread
> virt-qemu-run: 2261: opening qemu:///embed?root=/xml
> virt-qemu-run: 53686: starting guest /xml/vm2.xml
> 2020-09-27 02:15:54.798+0000: 108161: info : libvirt version: 6.6.0,
> package: 6.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> 2020-09-17-01:41:02, )
> 2020-09-27 02:15:54.798+0000: 108161: info : hostname: dell*.com
> 2020-09-27 02:15:54.798+0000: 108161: warning : qemuProcessStop:7592 :
> Unable to release network device '<null>'
> virt-qemu-run: cannot start VM: Failed to connect socket to
> '/var/run/libvirt/virtnetworkd-sock': Connection refused
> virt-qemu-run: 116110: cleaned up, exiting

Also negative test for option of 'virt-qemu-run':
# virt-qemu-run -t
virt-qemu-run: option parsing failed: Unknown option -t
Comment 19 yafu 2020-09-27 03:01:15 UTC 
(In reply to Daniel Berrangé from comment #7)
> This is a tricky policy question.  Historically QEMU driver ran inside
> libvirtd, and thus we have a policy rule designed around a "virtd_t" type.
> 
> When we're embedding QEMU driver into application code, those applications
> need to have equivalent permissions to virtd_t, but we don't want to run the
> app as virtd_t in general. Each app that needs to use embedded QEMU will
> need its own SELinux policy that inherits much, but not all, of virtd_t type.
> 
> For this, we're really goiing to want to restructure libvirt selinux policy
> to separate it out for the modular deamons (virtqemud, replacing libvirtd). 
> 
> I think we'll have to consider SELinux enforcement out of scope in the near
> term, as I can't see us solving this for 8.3

Hi Daniel,

Is there any existing bug which tracks the restructure of libvirt selinux policy? If no existing bug, do you mind me file a bug to track the issue?


Thanks,
Yan Fu
Comment 20 yafu 2020-09-27 03:04:05 UTC 
(In reply to yafu from comment #19)
> (In reply to Daniel Berrangé from comment #7)
> > This is a tricky policy question.  Historically QEMU driver ran inside
> > libvirtd, and thus we have a policy rule designed around a "virtd_t" type.
> > 
> > When we're embedding QEMU driver into application code, those applications
> > need to have equivalent permissions to virtd_t, but we don't want to run the
> > app as virtd_t in general. Each app that needs to use embedded QEMU will
> > need its own SELinux policy that inherits much, but not all, of virtd_t type.
> > 
> > For this, we're really goiing to want to restructure libvirt selinux policy
> > to separate it out for the modular deamons (virtqemud, replacing libvirtd). 
> > 
> > I think we'll have to consider SELinux enforcement out of scope in the near
> > term, as I can't see us solving this for 8.3
> 
> Hi Daniel,
> 
> Is there any existing bug which tracks the restructure of libvirt selinux
> policy? If no existing bug, do you mind me file a bug to track the issue?
> 
> 
> Thanks,
> Yan Fu

Sorry, forgot to add needinfo for comment 19.
Comment 21 yafu 2020-09-27 11:57:06 UTC 
Hi Daniel,

I have another question. When using 'virt-qemu-run' to start guest in a new namespace, it fails. But using 'virsh start' can start guest successfully.
Would you help to check the issue please? Thanks.

#unshare --mount --fork --pid /bin/bash

# virt-qemu-run /tmp/83.xml -d -v
virt-qemu-run: 10: initializing libvirt
virt-qemu-run: 1217: initializing signal handlers
virt-qemu-run: 1374: preparing event loop thread
virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0
virt-qemu-run: 38698: starting guest /tmp/83.xml
virt-qemu-run: cannot start VM: Process with ID 215 does not exist.
virt-qemu-run: 1131315: cleaned up, exiting

# virsh start 83
Domain 83 started
Comment 22 Daniel Berrangé 2020-09-28 13:02:47 UTC 
(In reply to yafu from comment #21)
> Hi Daniel,
> 
> I have another question. When using 'virt-qemu-run' to start guest in a new
> namespace, it fails. But using 'virsh start' can start guest successfully.
> Would you help to check the issue please? Thanks.
> 
> #unshare --mount --fork --pid /bin/bash
> 
> # virt-qemu-run /tmp/83.xml -d -v
> virt-qemu-run: 10: initializing libvirt
> virt-qemu-run: 1217: initializing signal handlers
> virt-qemu-run: 1374: preparing event loop thread
> virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0
> virt-qemu-run: 38698: starting guest /tmp/83.xml
> virt-qemu-run: cannot start VM: Process with ID 215 does not exist.
> virt-qemu-run: 1131315: cleaned up, exiting
> 
> # virsh start 83
> Domain 83 started

THis scenario doesn't really make sense - "virsh start" here is connecting to "qemu:///system", which is completely separate from the embedded driver. I presume you have a guest called "83" defined in your qemu:///system driver instance already 

(In reply to yafu from comment #19)
> (In reply to Daniel Berrangé from comment #7)
> > This is a tricky policy question.  Historically QEMU driver ran inside
> > libvirtd, and thus we have a policy rule designed around a "virtd_t" type.
> > 
> > When we're embedding QEMU driver into application code, those applications
> > need to have equivalent permissions to virtd_t, but we don't want to run the
> > app as virtd_t in general. Each app that needs to use embedded QEMU will
> > need its own SELinux policy that inherits much, but not all, of virtd_t type.
> > 
> > For this, we're really goiing to want to restructure libvirt selinux policy
> > to separate it out for the modular deamons (virtqemud, replacing libvirtd). 
> > 
> > I think we'll have to consider SELinux enforcement out of scope in the near
> > term, as I can't see us solving this for 8.3
> 
> Hi Daniel,
> 
> Is there any existing bug which tracks the restructure of libvirt selinux
> policy? If no existing bug, do you mind me file a bug to track the issue?

There is existing work planned to modularize the Libvirt SELinux policy, but that is not likely to be done in context of RHEL-8 due to its disruption

It is tracked in this Jira plan:

https://issues.redhat.com/browse/SELINUX-2283
Comment 23 yafu 2020-09-29 08:37:12 UTC 
(In reply to Daniel Berrangé from comment #22)
> (In reply to yafu from comment #21)
> > Hi Daniel,
> > 
> > I have another question. When using 'virt-qemu-run' to start guest in a new
> > namespace, it fails. But using 'virsh start' can start guest successfully.
> > Would you help to check the issue please? Thanks.
> > 
> > #unshare --mount --fork --pid /bin/bash
> > 
> > # virt-qemu-run /tmp/83.xml -d -v
> > virt-qemu-run: 10: initializing libvirt
> > virt-qemu-run: 1217: initializing signal handlers
> > virt-qemu-run: 1374: preparing event loop thread
> > virt-qemu-run: 1423: opening qemu:///embed?root=/tmp/virt-qemu-run-4UW2R0
> > virt-qemu-run: 38698: starting guest /tmp/83.xml
> > virt-qemu-run: cannot start VM: Process with ID 215 does not exist.
> > virt-qemu-run: 1131315: cleaned up, exiting
> > 
> > # virsh start 83
> > Domain 83 started
> 
> THis scenario doesn't really make sense - "virsh start" here is connecting
> to "qemu:///system", which is completely separate from the embedded driver.
> I presume you have a guest called "83" defined in your qemu:///system driver
> instance already 

Yes, i have a guest called '83' defined. But it also can start guest successfully by 'virsh create' if no guest defined.


> 
> (In reply to yafu from comment #19)
> > (In reply to Daniel Berrangé from comment #7)
> > > This is a tricky policy question.  Historically QEMU driver ran inside
> > > libvirtd, and thus we have a policy rule designed around a "virtd_t" type.
> > > 
> > > When we're embedding QEMU driver into application code, those applications
> > > need to have equivalent permissions to virtd_t, but we don't want to run the
> > > app as virtd_t in general. Each app that needs to use embedded QEMU will
> > > need its own SELinux policy that inherits much, but not all, of virtd_t type.
> > > 
> > > For this, we're really goiing to want to restructure libvirt selinux policy
> > > to separate it out for the modular deamons (virtqemud, replacing libvirtd). 
> > > 
> > > I think we'll have to consider SELinux enforcement out of scope in the near
> > > term, as I can't see us solving this for 8.3
> > 
> > Hi Daniel,
> > 
> > Is there any existing bug which tracks the restructure of libvirt selinux
> > policy? If no existing bug, do you mind me file a bug to track the issue?
> 
> There is existing work planned to modularize the Libvirt SELinux policy, but
> that is not likely to be done in context of RHEL-8 due to its disruption
> 
> It is tracked in this Jira plan:
> 
> https://issues.redhat.com/browse/SELINUX-2283
Comment 24 yafu 2020-10-09 09:34:58 UTC 
Test with session mode.

1.#su - test

2.$ virt-qemu-run -d -v /tmp/83.xml
virt-qemu-run: 14: initializing libvirt
virt-qemu-run: 1925: initializing signal handlers
virt-qemu-run: 2034: preparing event loop thread
virt-qemu-run: 2103: opening qemu:///embed?root=/tmp/virt-qemu-run-WQZKS0
virt-qemu-run: 19541: starting guest /tmp/83.xml
virt-qemu-run: 1379396: guest running, Ctrl-C to stop now
virt-qemu-run: XML: <domain type='kvm' id='1'>
  <name>83</name>
  <uuid>9350639d-1c8a-4f51-a4a6-4eaf8eabe83e</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://redhat.com/rhel/8.3"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>
  <vcpu placement='static'>2</vcpu>

3.Check the qemu process：
$ ps aux | grep -i qemu
test      410248  3.8  0.0 422264 30576 pts/1    Sl+  05:32   0:00 virt-qemu-run -d -v /tmp/83.xml
test      410280 95.0  0.1 2548432 56888 ?       Sl   05:32   0:04 /usr/libexec/qemu-kvm -name guest=83,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/tmp/virt-qemu-run-WQZKS0/lib/qemu/domain-1-83/master-key.aes...
Comment 25 yafu 2020-10-09 09:47:26 UTC 
Created attachment 1720192 [details]
code coverage report for patchset
Comment 26 yafu 2020-10-09 09:49:48 UTC 
From the code coverage report for patchset, we could see most code of the patches were tested except some negative test for error handler.
Comment 28 yafu 2020-10-10 00:16:34 UTC 
Move the bug to VERIFIED according to comment 13 - comment 26.
Comment 30 errata-xmlrpc 2020-11-17 17:45:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137
Note You need to log in before you can comment on or make changes to this bug.