Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1354335 - overlay of disk images does not specify the format of the backing file
Summary: overlay of disk images does not specify the format of the backing file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2016-07-11 07:03 UTC by Xianghua Chen
Modified: 2016-11-03 18:02 UTC (History)
14 users (show)

Fixed In Version: libguestfs-1.32.6-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 18:02:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
log.virt-inspector-backing_file (8.63 KB, text/plain)
2016-07-11 07:03 UTC, Xianghua Chen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2576 0 normal SHIPPED_LIVE Moderate: libguestfs and virt-p2v security, bug fix, and enhancement update 2016-11-03 12:06:51 UTC

Description Xianghua Chen 2016-07-11 07:03:15 UTC
Created attachment 1178262 [details]
log.virt-inspector-backing_file

Description of problem:
virt-inspector failed to inspect new image created by qemu-img --backing-file if the selinux context of original file is not virt_content_t.

Please refer to full log in attachment.

Version-Release number of selected component (if applicable):
libguestfs-1.32.5-9.el7.x86_64
qemu-kvm-1.5.3-116.el7.x86_64
qemu-img-1.5.3-116.el7.x86_64
libvirt-2.0.0-1.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Copy a guest image from nfs server
# cp /tmp/images/RHEL-Server-6.8-32-hvm.raw images/RHEL-Server-6.8-32-hvm.raw
# ll -Z images/RHEL-Server-6.8-32-hvm.raw
-rw-r--r--. root root unconfined_u:object_r:home_root_t:s0 images/RHEL-Server-6.8-32-hvm.raw

2. # qemu-img create -f qcow2 -o backing_file=/home/git/libguestfs.autotest/client/tests/libguestfs/images/RHEL-Server-6.8-32-hvm.raw,backing_fmt=raw /var/run/guestfs-autotest/images/tmp.qcow2
# ll -Z images/tmp.qcow2
-rw-r--r--. root root unconfined_u:object_r:home_root_t:s0 images/tmp.qcow2

3. # virt-inspector -a images/tmp.qcow2
libguestfs: error: could not create appliance through libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while connecting to monitor: 2016-07-08T08:13:56.786262Z qemu-kvm: -drive file=/tmp/libguestfsDzDycQ/overlay1,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe: Could not open backing file: Could not open backing file: Could not open '/home/git/libguestfs.autotest/client/tests/libguestfs/images/RHEL-Server-6.8-32-hvm.raw': Permission denied [code=1 int1=-1]

4. If I have executed libguestfs command first on original guest image before step 3:
# virt-inspector -a images/RHEL-Server-6.8-32-hvm.raw
# ll -Z images/RHEL-Server-6.8-32-hvm.raw
-rw-r--r--. qemu qemu system_u:object_r:virt_content_t:s0 images/RHEL-Server-6.8-32-hvm.raw

Ps.:
Or just use chcon to instead executing libguestfs command:
# chcon -t virt_content_t RHEL-Server-6.8-32-hvm.raw

Then execute the command in step 3, it will success.


Actual results:
As above output.

Expected results:
Command in step 3 should be finished successfully.

Additional info:

Comment 1 Richard W.M. Jones 2016-07-11 08:43:48 UTC
Reproducer (all commands run as root):

# cd /var/tmp
# virt-builder centos-7.2
# ll -Z centos-7.2.img 
-rw-r--r--. root root system_u:object_r:svirt_image_t:s0:c146,c485 centos-7.2.img
# qemu-img create -f qcow2 -o backing_file=/var/tmp/centos-7.2.img,backing_fmt=raw overlay.qcow2
# ll -Z overlay.qcow2 
-rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 overlay.qcow2
# virt-inspector -a overlay.qcow2 
libguestfs: error: could not create appliance through libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while connecting to monitor: 2016-07-11T08:42:06.554484Z qemu-kvm: -drive file=/tmp/libguestfsx18fpO/overlay1,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=unsafe: Could not open backing file: Could not open backing file: Could not open '/var/tmp/centos-7.2.img': Permission denied
 [code=1 int1=-1]

# ll -Z overlay.qcow2 centos-7.2.img 
-rw-r--r--. root root system_u:object_r:svirt_image_t:s0:c146,c485 centos-7.2.img
-rw-r--r--. qemu qemu system_u:object_r:virt_content_t:s0 overlay.qcow2

Comment 2 Richard W.M. Jones 2016-07-11 08:46:28 UTC
Libvirt XML generated by virt-inspector is below.  Note that libguestfs
generates a temporary overlay on top of the overlay, so the full chain
of images is:
  /tmp/.../overlay1 -> /var/tmp/overlay.qcow2 -> /var/tmp/centos-7.2.img

<?xml version="1.0"?>
<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
  <name>guestfs-avt07jmbsov1y0sa</name>
  <memory unit="MiB">500</memory>
  <currentMemory unit="MiB">500</currentMemory>
  <cpu mode="host-passthrough">
    <model fallback="allow"/>
  </cpu>
  <vcpu>1</vcpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <os>
    <type>hvm</type>
    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>
    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>
    <cmdline>panic=1 console=ttyS0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>
    <bios useserial="yes"/>
  </os>
  <on_reboot>destroy</on_reboot>
  <devices>
    <controller type="scsi" index="0" model="virtio-scsi"/>
    <disk device="disk" type="file">
      <source file="/tmp/libguestfslgut5O/overlay1"/>
      <target dev="sda" bus="scsi"/>
      <driver name="qemu" type="qcow2" cache="unsafe"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="disk">
      <source file="/tmp/libguestfslgut5O/overlay2"/>
      <target dev="sdb" bus="scsi"/>
      <driver name="qemu" type="qcow2" cache="unsafe"/>
      <address type="drive" controller="0" bus="0" target="1" unit="0"/>
      <shareable/>
    </disk>
    <serial type="unix">
      <source mode="connect" path="/tmp/libguestfslgut5O/console.sock"/>
      <target port="0"/>
    </serial>
    <channel type="unix">
      <source mode="connect" path="/tmp/libguestfslgut5O/guestfsd.sock"/>
      <target type="virtio" name="org.libguestfs.channel.0"/>
    </channel>
  </devices>
  <qemu:commandline>
    <qemu:env name="TMPDIR" value="/var/tmp"/>
  </qemu:commandline>
</domain>

Comment 3 yafu 2016-07-21 11:29:49 UTC
I think i's not a svirt bug. The bug is caused by libguestfs generates a temporary overlay on top of the overlay without setting the backing_file format, which leads to libvirt even can not detect the base backing file. But the qemu can detect the whole backing-chain, so failed with permission denied when starting the guest.

steps to reproduce with libvirt:
1.qemu-img create -f qcow2 test.img 1G
Formatting 'test.img', fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

2.#qemu-img create -f qcow2 -o backing_file=test.img -F qcow2 test.s1  
Formatting 'test.s1', fmt=qcow2 size=1073741824 backing_file=test.img backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

3.#qemu-img create -f qcow2 -o backing_file=test.s1  test.s2      ------without setting the backing file format
  Formatting 'test.s2', fmt=qcow2 size=1073741824 backing_file=test.s1 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

4.The backing-chain is:
  test.s2->test.s1->test.img

5.Prepare a guest using test.s2:
  #virsh edit rhel7.3
   ...
   <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/test/test.s2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
  ...

6.Start the guest:
 #virsh start rhel7.3
  error: Failed to start domain mig1
error: internal error: qemu unexpectedly closed the monitor: 2016-07-21T10:12:59.392715Z qemu-kvm: -drive file=/var/lib/libvirt/images/test/test.s2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none: Could not open backing file: Could not open backing file: Could not open '/var/lib/libvirt/images/test/test.img': Permission denied

7.Check the libvirtd.log, find the detect the format of test.s1 is wrong:
  #cat /var/log/libvirt/libvirtd.log
   ...
   2016-07-21 10:04:46.209+0000: 31270: debug : virStorageFileGetMetadata:3318 : path=/var/lib/libvirt/images/test/test.s2 format=15 uid=107 gid=107 probe=0, report_broken=1
2016-07-21 10:04:46.209+0000: 31270: debug : virStorageFileGetMetadataRecurse:3201 : path=/var/lib/libvirt/images/test/test.s2 format=15 uid=107 gid=107 probe=0
2016-07-21 10:04:46.209+0000: 31270: debug : virStorageFileBackendFileInit:1497 : initializing FS storage file 0x7f86bc01ad10 (file:/var/lib/libvirt/images/test/test.s2)[107:107]
2016-07-21 10:04:46.211+0000: 31270: debug : virStorageFileGetMetadataInternal:814 : path=/var/lib/libvirt/images/test/test.s2, buf=0x7f86dc001e70, len=33280, *******meta->format=15*********   ----------test.s2's format is qcow2
...
2016-07-21 10:04:46.211+0000: 31270: debug : virStorageFileGetMetadataRecurse:3201 : path=/var/lib/libvirt/images/test/test.s1 format=1 uid=107 gid=107 probe=0
2016-07-21 10:04:46.211+0000: 31270: debug : virStorageFileBackendFileInit:1497 : initializing FS storage file 0x7f86dc016ca0 (file:/var/lib/libvirt/images/test/test.s1)[107:107]
2016-07-21 10:04:46.212+0000: 31270: debug : virStorageFileGetMetadataInternal:814 : path=/var/lib/libvirt/images/test/test.s1, buf=0x7f86dc01ca60, len=33280, ******meta->format=1*******   ----------test.s1's format is raw

The log shows libvirt get test.s1's format as raw, so it can not get the correct info in the test.s1's header file. 


Hi, jferlan,  Since libvirt does not detect images format for the reason of security in default, would you help to check whether it is the expected results with libvirt please?

Comment 4 Peter Krempa 2016-07-26 11:55:52 UTC
(In reply to yafu from comment #3)
> I think i's not a svirt bug. The bug is caused by libguestfs generates a
> temporary overlay on top of the overlay without setting the backing_file
> format, which leads to libvirt even can not detect the base backing file.
> But the qemu can detect the whole backing-chain, so failed with permission
> denied when starting the guest.

Indeed, this is even captured with the attached log file:

libguestfs: trace: disk_create "/tmp/libguestfsBW52iQ/overlay1" "qcow2" -1 "backingfile:/run/guestfs-autotest/images/tmp.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/run/guestfs-autotest/images/tmp.qcow2
libguestfs: command: run: \ /tmp/libguestfsBW52iQ/overlay1
Formatting '/tmp/libguestfsBW52iQ/overlay1', fmt=qcow2 size=8589934592 backing_file=/run/guestfs-autotest/images/tmp.qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

> steps to reproduce with libvirt:

[SNIP]

> 7.Check the libvirtd.log, find the detect the format of test.s1 is wrong:
>   #cat /var/log/libvirt/libvirtd.log
>    ...
>    2016-07-21 10:04:46.209+0000: 31270: debug :
> virStorageFileGetMetadata:3318 : path=/var/lib/libvirt/images/test/test.s2
> format=15 uid=107 gid=107 probe=0, report_broken=1
> 2016-07-21 10:04:46.209+0000: 31270: debug :
> virStorageFileGetMetadataRecurse:3201 :
> path=/var/lib/libvirt/images/test/test.s2 format=15 uid=107 gid=107 probe=0
> 2016-07-21 10:04:46.209+0000: 31270: debug :
> virStorageFileBackendFileInit:1497 : initializing FS storage file
> 0x7f86bc01ad10 (file:/var/lib/libvirt/images/test/test.s2)[107:107]
> 2016-07-21 10:04:46.211+0000: 31270: debug :
> virStorageFileGetMetadataInternal:814 :
> path=/var/lib/libvirt/images/test/test.s2, buf=0x7f86dc001e70, len=33280,
> *******meta->format=15*********   ----------test.s2's format is qcow2
> ...
> 2016-07-21 10:04:46.211+0000: 31270: debug :
> virStorageFileGetMetadataRecurse:3201 :
> path=/var/lib/libvirt/images/test/test.s1 format=1 uid=107 gid=107 probe=0
> 2016-07-21 10:04:46.211+0000: 31270: debug :
> virStorageFileBackendFileInit:1497 : initializing FS storage file
> 0x7f86dc016ca0 (file:/var/lib/libvirt/images/test/test.s1)[107:107]
> 2016-07-21 10:04:46.212+0000: 31270: debug :
> virStorageFileGetMetadataInternal:814 :
> path=/var/lib/libvirt/images/test/test.s1, buf=0x7f86dc01ca60, len=33280,
> ******meta->format=1*******   ----------test.s1's format is raw
> 
> The log shows libvirt get test.s1's format as raw, so it can not get the
> correct info in the test.s1's header file. 
> 
> 
> Hi, jferlan,  Since libvirt does not detect images format for the reason of
> security in default, would you help to check whether it is the expected
> results with libvirt please?

Yes. It's deliberate that libvirt does not perform (backing) image type detection. This would create a security hole by allowing guests to overwrite a raw image with a qcow2 header and thus tricking the host to making an arbitrary host file accessible. In the default configuration of libvirt any backing file that does not have the format specified explicitly is treated as raw.

Assigning back to libguestfs since the behavior in libvirt is deliberate.

Comment 5 Richard W.M. Jones 2016-07-26 12:47:15 UTC
(In reply to Peter Krempa from comment #4)
> (In reply to yafu from comment #3)
> > I think i's not a svirt bug. The bug is caused by libguestfs generates a
> > temporary overlay on top of the overlay without setting the backing_file
> > format, which leads to libvirt even can not detect the base backing file.
> > But the qemu can detect the whole backing-chain, so failed with permission
> > denied when starting the guest.
> 
> Indeed, this is even captured with the attached log file:
> 
> libguestfs: trace: disk_create "/tmp/libguestfsBW52iQ/overlay1" "qcow2" -1
> "backingfile:/run/guestfs-autotest/images/tmp.qcow2"
> libguestfs: command: run: qemu-img
> libguestfs: command: run: \ create
> libguestfs: command: run: \ -f qcow2
> libguestfs: command: run: \ -o
> backing_file=/run/guestfs-autotest/images/tmp.qcow2
> libguestfs: command: run: \ /tmp/libguestfsBW52iQ/overlay1
> Formatting '/tmp/libguestfsBW52iQ/overlay1', fmt=qcow2 size=8589934592
> backing_file=/run/guestfs-autotest/images/tmp.qcow2 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16

This is deliberate.  Unless the user specifies the source
file format, we don't set the backing_fmt for the overlay.
IOW we (or rather, the user) desires autodetection in this case.

...

I don't understand your explanation for why libvirt doesn't do autodetection,
because by the same argument qemu would be wrong to do autodetection, yet
qemu is doing autodetection.  Anyway I will just say that ...

> Yes. It's deliberate that libvirt does not perform (backing) image type
> detection. This would create a security hole by allowing guests to overwrite
> a raw image with a qcow2 header and thus tricking the host to making an
> arbitrary host file accessible.

... this part is wrong.  Since Nov 2014, qemu has detected this
situation and prevents writes to the first sector of the disk if
the format is raw and the disk format was probed.
(See qemu commit 38f3ef574b48afc507c6f636ae4393fd36bda072).

Comment 6 Daniel Berrangé 2016-07-26 12:59:22 UTC
(In reply to Richard W.M. Jones from comment #5)
> I don't understand your explanation for why libvirt doesn't do autodetection,
> because by the same argument qemu would be wrong to do autodetection, yet
> qemu is doing autodetection.  Anyway I will just say that ...

QEMU only does auto-detection by default because it needs to avoid breaking historical usage of QEMU. If QEMU didn't care about historic compat, it would not auto-detect at all.


> > Yes. It's deliberate that libvirt does not perform (backing) image type
> > detection. This would create a security hole by allowing guests to overwrite
> > a raw image with a qcow2 header and thus tricking the host to making an
> > arbitrary host file accessible.
> 
> ... this part is wrong.  Since Nov 2014, qemu has detected this
> situation and prevents writes to the first sector of the disk if
> the format is raw and the disk format was probed.
> (See qemu commit 38f3ef574b48afc507c6f636ae4393fd36bda072).

That commit was added as a safety net for applications/users who accidentally still invoke QEMU in a way which triggers auto-detection. That doesn't imply that auto-detection is a good/valid thing to permit / enable when launching. Any usage of QEMU should none the less always strive to *never* use auto-detection, which is what libvirt does. 

IOW just because QEMU added a safety net doesn't mean libvirt should now allow auto-detection.

Comment 7 Jaroslav Suchanek 2016-07-29 07:37:45 UTC
Rich, based on comment 4 and comment 6 this can't be fixed in libvirt and would be a resolution for this. Is it possible to fix it in libguestfs?

Comment 8 Richard W.M. Jones 2016-08-02 13:50:35 UTC
I posted a workaround for this:
https://www.redhat.com/archives/libguestfs/2016-August/msg00004.html

Comment 9 Richard W.M. Jones 2016-08-02 19:37:58 UTC
Upstream commit c8c181e8d9729ea09e56431b568dceba3873af7b.

Comment 11 Xianghua Chen 2016-08-05 07:49:09 UTC
Verified with the packages:
libguestfs-1.32.6-4.el7.x86_64
virt-dib-1.32.6-4.el7.x86_64

Verify steps:
1. # virt-builder centos-7.2
2. # qemu-img create -f qcow2 -o backing_file=centos-7.2.img,backing_fmt=raw overlay.qcow2
Check the selinux context:
 # ll -Z overlay.qcow2 centos-7.2.img
-rw-r--r--. root root unconfined_u:object_r:user_home_t:s0 centos-7.2.img
-rw-r--r--. root root unconfined_u:object_r:home_root_t:s0 overlay.qcow2
4. # virt-inspector -a overlay.qcow2
5. Check the selinux context:
# ll -Z overlay.qcow2 centos-7.2.img
-rw-r--r--. root root system_u:object_r:virt_content_t:s0 centos-7.2.img
-rw-r--r--. root root system_u:object_r:virt_content_t:s0 overlay.qcow2


Command in step 4 finished successfully.

So verified.

Comment 13 errata-xmlrpc 2016-11-03 18:02:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2576.html


Note You need to log in before you can comment on or make changes to this bug.