Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1134878
Summary: | libvirt reports json "backing file" is missing | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Richard W.M. Jones <rjones> |
Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.1 | CC: | amureini, bmcclain, dyuan, eblake, hhan, jsuchane, libvirt-maint, mzhan, pkrempa, rbalakri, tnisan, tzheng, xuzhang, yanyang |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-2.0.0-4.el7 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | 1134592 | Environment: | |
Last Closed: | 2016-11-03 18:09:54 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1134592, 1367260, 1457088 | ||
Bug Blocks: | 910269, 1254282 |
Description
Richard W.M. Jones
2014-08-28 11:51:03 UTC
How to test: $ cd /tmp $ truncate -s 1G test1.img $ qemu-img create -f qcow2 \ -b 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' \ test2.img $ guestfish -a test2.img run The original libvirt will give an error like this: invalid argument: Backing file 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' of image '/tmp/test2.img' is missing. [code=8 domain=10] (In reply to Richard W.M. Jones from comment #0) > +++ This bug was initially created as a clone of Bug #1134592 +++ > > Description of problem: > > $ qemu-img create -f qcow2 -b 'json: { "file.driver":"ssh", > "file.user":"root", "file.host":"avon.home.annexia.org", > "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk", > "file.host_key_check":"no" }' /tmp/foo > > qemu-img file is successful, but: > > $ guestfish -a /tmp/foo -i > [...] > libvirt: invalid argument: Backing file 'json: { "file.driver":"ssh", > "file.user":"root", "file.host":"avon.home.annexia.org", > "file.path":"/vmfs/volumes/datastore1/Fedora 20/Fedora 20-flat.vmdk", > "file.host_key_check":"no" }' of image '/tmp/foo' is missing. [code=8 > domain=10] > > This is presumably because libvirt has trouble parsing the > new qemu 2.1.0 "json:" specification. TBH I think it shouldn't > even try, it should just ignore them. Certainly it shouldn't > be giving an error here. Upstream's stance on this is that if libvirt doesn't know how to parse a backing file specification we should error out rather than silently accept it. > > Version-Release number of selected component (if applicable): > > libvirt-1.1.3.5-2.fc20.x86_64 > > How reproducible: > > 100% - see above. > > --- Additional comment from Eric Blake on 2014-08-27 17:37:18 EDT --- > > More generically, there are two problems here: > 1. Libvirt needs to catch up on the additional formats supported by upstream > qemu. (This is one case where a qemu library that parses qcow2 files would > be nice - if only it weren't for the fact that no such library exists in > part because current qemu aborts on OOM which is inappropriate for library > code) Eric: Is the JSON backend file format stable enough to actually try to implement it? If we implement the parser we will also need to implement support for "ssh" network protocol so that we'll be able to parse the protocol and create a source structure describing it. This should bypass all the other checks as the backing file will be known. Without implementation of ssh storage driver we won't be able to make sure that the remote files have correct permissions though, but we have prior art in this. > > 2. Libvirt needs to have sane behavior when a backing format looks like a > protocol but for which libvirt does not (yet) know the protocol. It may > still be an error message, but the message should be more like 'json: > protocol not yet handled by libvirt', not 'stat of file "json: ..." failed'. > In other words, libvirt should NEVER attempt to stat() any protocol, known > or unknown. The error message seems sufficient IMHO: invalid argument: Backing file 'json: { "file.driver":"file", "file.filename":"/tmp/test1.img" }' of image '/tmp/test2.img' is missing. (In reply to Peter Krempa from comment #3) > > This is presumably because libvirt has trouble parsing the > > new qemu 2.1.0 "json:" specification. TBH I think it shouldn't > > even try, it should just ignore them. Certainly it shouldn't > > be giving an error here. > > Upstream's stance on this is that if libvirt doesn't know how to parse a > backing file specification we should error out rather than silently accept > it. In the past, upstream has historically treated unknown strings as raw, and handed it to qemu with hopes for the best; but this is not very wise to do (if the string is not actually raw, qemu will be acting on different files than what libvirt expects), so I agree with the stance that libvirt SHOULD be complaining loudly on unknown content so that we then know what needs patching for support. > > More generically, there are two problems here: > > 1. Libvirt needs to catch up on the additional formats supported by upstream > > qemu. (This is one case where a qemu library that parses qcow2 files would > > be nice - if only it weren't for the fact that no such library exists in > > part because current qemu aborts on OOM which is inappropriate for library > > code) > > Eric: Is the JSON backend file format stable enough to actually try to > implement it? Yes, the json: format is a stable interface that wraps the QMP command blockdev-add. It doesn't yet describe all possible block devices that are possible on the command line or hotplug (hence we still have to use HMP fallbacks for hotplug of some block device backends), but for the formats that are supported, the API is stable and strongly typed, with a promise of backwards compatibility. And I've been reviewing the qemu patches that implement various block drivers to make sure it stays stable. At any rate, figuring out how to support json: format will be a net win for eventually switching over to QMP hotplug - since we need JSON descriptions of block devices for both reasons (backing files and hotplug), it is shared code that will eventually be reused. > > If we implement the parser we will also need to implement support for "ssh" > network protocol so that we'll be able to parse the protocol and create a > source structure describing it. This should bypass all the other checks as > the backing file will be known. > > Without implementation of ssh storage driver we won't be able to make sure > that the remote files have correct permissions though, but we have prior art > in this. Correct - to date, we have worried about expressing protocols, but not worried about security on the remote side of such protocols. Is there a plan to fix this? Currently virt-v2v prints out an error naming bug 1134592 and telling people to not use libvirt as a workaround. $ virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest [ 0.0] Opening the source -i libvirt -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest virt-v2v: error: because of libvirt bug https://bugzilla.redhat.com/show_bug.cgi?id=1134592 you must set this environment variable: export LIBGUESTFS_BACKEND=direct and then rerun the virt-v2v command. *** Bug 1134592 has been marked as a duplicate of this bug. *** Hi Peter, I noticed that rbd protocol was not added in the pacthes and libvirt-2.0.0-5.el7.x86_64 is not support json backimg from rbd: # qemu-img create -f qcow2 -b 'json: {"file.driver":"rbd", "file.filename": "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"}' /var/lib/libvirt/images/rbd_json.img Formatting '/var/lib/libvirt/images/rbd_json.img', fmt=qcow2 size=5368709120 backing_file=json: {"file.driver":"rbd",, "file.filename": "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk vm1 /var/lib/libvirt/images/rbd_json.img vdc --subdriver qcow2 Failed to attach disk internal error: missing parser implementation for JSON backing volume driver 'rbd' Will we add rbd json backing support? I think it is necessary. (In reply to Han Han from comment #15) > Hi Peter, > I noticed that rbd protocol was not added in the pacthes and > libvirt-2.0.0-5.el7.x86_64 is not support json backimg from rbd: > # qemu-img create -f qcow2 -b 'json: {"file.driver":"rbd", "file.filename": > "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"}' > /var/lib/libvirt/images/rbd_json.img > Formatting '/var/lib/libvirt/images/rbd_json.img', fmt=qcow2 size=5368709120 > backing_file=json: {"file.driver":"rbd",, "file.filename": > "rbd:libvirt-pool/vol1:mon_host=10.73.75.52"} encryption=off > cluster_size=65536 lazy_refcounts=off refcount_bits=16 > > # virsh attach-disk vm1 /var/lib/libvirt/images/rbd_json.img vdc --subdriver > qcow2 > Failed to attach disk > internal error: missing parser implementation for JSON backing volume driver > 'rbd' > > Will we add rbd json backing support? I think it is necessary. New a bug BZ1371758 to track the rbd json backing support. hi Peter, the bug BZ1372251 may be a test blocker for this bug. Do we need to do attach/detach the json images when verifying? attach/detach should work properly with other types of images. BZ1372251 tracks a very specific issue with gluster images, whereas attach/detach should work properly with different types of images. See the list of bugs this one "depends on". The QEMU corner cases are: Bug 1365708, Bug 1373710 and Bug 1373786. (In reply to Ademar Reis from comment #22) > See the list of bugs this one "depends on". The QEMU corner cases are: Bug > 1365708, Bug 1373710 and Bug 1373786. I'm removing them from the "depends on" list and also removing their TestBlocker keywords. As Peter suggested, they should be handled as standlone issues and should not block the verification of this BZ. The only BZ currently marked as TestBlocker is the one about glusterfs via json. Which doesn't appear to be a v2v use-case. (In reply to Ademar Reis from comment #23) > (In reply to Ademar Reis from comment #22) > > See the list of bugs this one "depends on". The QEMU corner cases are: Bug > > 1365708, Bug 1373710 and Bug 1373786. Yes I seem to remember we saw bug 1365708, but passing the right type in the json fixed that. Don't know why I didn't file a BZ, perhaps I forgot to or assumed it was expected behavious. The other two wouldn't affect virt-v2v. > I'm removing them from the "depends on" list and also removing their > TestBlocker keywords. As Peter suggested, they should be handled as > standlone issues and should not block the verification of this BZ. > > The only BZ currently marked as TestBlocker is the one about glusterfs via > json. Which doesn't appear to be a v2v use-case. Correct, glusterfs isn't be used by virt-v2v. Hi Peter, From the discussions above, we know that the BZ1365708, BZ1373710, BZ1373786 will not block the bug. So I will verify the bug with following scenarios: 1. For file, host_device, https, gluster(tcp), iscsi, nbd, rbd protocols, they are fully supported by libvirt and qemu-kvm-rhev. I will test them with pool-refresh, hot-plug/hot-unplug 2. For host_cdrom, http, ftp, ftps, sheepdog, they are not whitelisted by qemu-kvm program and ssh is no supported by libvirt when attaching. So I will test them with pool-refesh only. And I will create a libvirt TestOnly bug to track these protocols. Do you have any suggestion about the bug verify? Sorry forget to add needinfo for the comment25 Yes, that looks reasonable to me. A image that has a SSH backing disk can be hotplugged though since libvirt should be able to parse it. Due to BZ1140166, ssh json backing disk cannot be hotplugged. So I will only use pool-refresh to check if libvirt can parse it. Verify it on libvirt-2.0.0-8.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 1. Prepare a running VM # virsh list Id Name State ---------------------------------------------------- 36 V running 2. Test on fully supported protocols(by attach/detach, read/write, pool-refresh) File protocol: # qemu-img create -f qcow2 -b 'json:{"file.driver":"file", "file.filename":"/tmp/a"}' /var/lib/libvirt/images/file.img Formatting '/var/lib/libvirt/images/file.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"file",, "file.filename":"/tmp/a"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/file.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Host_device protocol: # qemu-img create -f qcow2 -b 'json:{"file.driver":"host_device", "file.filename":"/dev/sdb"}' /var/lib/libvirt/images/host_device.img Formatting '/var/lib/libvirt/images/host_device.img', fmt=qcow2 size=10737418240 backing_file=json:{"file.driver":"host_device",, "file.filename":"/dev/sdb"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/host_device.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Https protocol: Since we use self signed certificate in https, so we use file.sslverify:off to skip the cert-insecure error. # qemu-img create -f qcow2 -b 'json:{"file.driver":"https", "file.url":"https://lab.work.me/https", "file.sslverify":"off"}' /var/lib/libvirt/images/https.img Formatting '/var/lib/libvirt/images/https.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"https",, "file.url":"https://lab.work.me/https",, "file.sslverify":"off"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/https.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Glusterfs protocol: Use glusterfs url format: # qemu-img create -f qcow2 -b 'json:{"file.driver":"gluster", "file.filename":"gluster://10.66.4.233/gluster-vol1/V"}' /var/lib/libvirt/images/gluster_file.img Formatting '/var/lib/libvirt/images/gluster_file.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"gluster",, "file.filename":"gluster://10.66.4.233/gluster-vol1/V"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/gluster_file.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Iscsi protocol: # qemu-img create -f qcow2 -b 'json:{"file.driver":"iscsi", "file.filename":"iscsi://10.66.7.179:3260/iqn.2015-05.com.redhat:libvirt.rhevm-test-group3-2/1"}' /var/lib/libvirt/images/iscsi_file.img Formatting '/var/lib/libvirt/images/iscsi_file.img', fmt=qcow2 size=21474836480 backing_file=json:{"file.driver":"iscsi",, "file.filename":"iscsi://10.66.7.179:3260/iqn.2015-05.com.redhat:libvirt.rhevm-test-group3-2/1"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/iscsi_file.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Nbd protocol: # qemu-img create -f qcow2 -b 'json:{"file.driver":"nbd", "file.host":"10.66.6.236", "file.port":"8888"}' /var/lib/libvirt/images/nbd.img Formatting '/var/lib/libvirt/images/nbd.img', fmt=qcow2 size=524288000 backing_file=json:{"file.driver":"nbd",, "file.host":"10.66.6.236",, "file.port":"8888"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # virsh attach-disk V /var/lib/libvirt/images/nbd.img vdb --subdriver qcow2 Disk attached successfully Do some read/write operation on vdb, it works well. # virsh detach-disk V vdb Disk detached successfully Pool refresh: # virsh pool-refresh default Pool default refreshed 3. Test on partly supported protocols(via pool-refresh) Create host_cdrom backing image: # qemu-img create -f qcow2 -b 'json:{"file.driver":"host_cdrom", "file.filename":"/dev/sr0"}' /var/lib/libvirt/images/host_cdrom.img Formatting '/var/lib/libvirt/images/host_cdrom.img', fmt=qcow2 size=3743416320 backing_file=json:{"file.driver":"host_cdrom",, "file.filename":"/dev/sr0"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 Create http backing image: # qemu-img create -f qcow2 -b 'json:{"file.driver":"http", "file.url":"http://10.66.6.236:81/tune.qcow2"}' /var/lib/libvirt/images/http.img Formatting '/var/lib/libvirt/images/http.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"http",, "file.url":"http://10.66.6.236:81/tune.qcow2"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 Create ssh backing image: # qemu-img create -f qcow2 -b 'json:{"file.driver":"ssh", "file.host":"10.66.6.236", "file.path":"/tmp/ssh", "file.user":"root", "file.port":22, "file.host_key_check":"no"}' /var/lib/libvirt/images/ssh.img Formatting '/var/lib/libvirt/images/ssh.img', fmt=qcow2 size=104857600 backing_file=json:{"file.driver":"ssh",, "file.host":"10.66.6.236",, "file.path":"/tmp/ssh",, "file.user":"root",, "file.port":22,, "file.host_key_check":"no"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 Create sheepdog backing image: # qemu-img create -f qcow2 -b 'json:{"file.driver":"sheepdog", "file.filename":"sheepdog://10.66.4.170:7000/Alice"}' /var/lib/libvirt/images/sheepdog.img Formatting '/var/lib/libvirt/images/sheepdog.img', fmt=qcow2 size=5368709120 backing_file=json:{"file.driver":"sheepdog",, "file.filename":"sheepdog://10.66.4.170:7000/Alice"} encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 Do pool-refresh: # virsh pool-refresh default Pool default refreshed # virsh vol-list default|grep img file.img /var/lib/libvirt/images/file.img gluster.img /var/lib/libvirt/images/gluster.img gluster_file.img /var/lib/libvirt/images/gluster_file.img gluster_multi.img /var/lib/libvirt/images/gluster_multi.img gluster_socket.img /var/lib/libvirt/images/gluster_socket.img host_cdrom.img /var/lib/libvirt/images/host_cdrom.img host_device.img /var/lib/libvirt/images/host_device.img http.img /var/lib/libvirt/images/http.img https.img /var/lib/libvirt/images/https.img iscsi_file.img /var/lib/libvirt/images/iscsi_file.img nbd.img /var/lib/libvirt/images/nbd.img sheepdog.img /var/lib/libvirt/images/sheepdog.img ssh.img /var/lib/libvirt/images/ssh.img All json backing image can be parsed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html |