The fundamental problem seems to be that libvirt needs the
home directory to exist in order to create directories
and files under $HOME/.cache/libvirt.

I will note there are two things you can do on the libguestfs side:

(1) Set LIBGUESTFS_ATTACH_METHOD=libvirt:[URI...] which changes
the URI used to connect to libvirt.

(2) Set LIBGUESTFS_ATTACH_METHOD=appliance to go back to running
qemu directly, without involving libvirt.

Note that calling g.set_attach_method is equivalent to setting
these environment variables.  You have to do it before calling
g.launch.

Environment variables are described here:
http://libguestfs.org/guestfs.3.html#environment-variables

Attach methods are described here:
http://libguestfs.org/guestfs.3.html#attach-method

g.set_attach_method is described here:
http://libguestfs.org/guestfs.3.html#guestfs_set_attach_method

Okay. That makes sense. I just confirmed the following patch to Nova would fix the issue:

--- a/nova/virt/disk/vfs/guestfs.py
+++ b/nova/virt/disk/vfs/guestfs.py
@@ -94,6 +94,7 @@ class VFSGuestFS(vfs.VFS):
         self.handle = guestfs.GuestFS()
 
         try:
+            self.handle.set_attach_method('appliance')
             self.handle.add_drive_opts(self.imgfile, format=self.imgfmt)
             self.handle.launch()

----

This also makes sense because the default guestfs attach_method was 'appliance' on Fedora 17. The default is now 'libvirt' on Fedora 18 (which clearly breaks things a bit..)

-----

I tried a slightly different fix as well which was to call this instead:

  self.handle.set_attach_method('libvirt:qemu:///system')

This seems to cause a slightly different issue related to file permissions on the disk image:

2013-02-21 10:31:27.448 ERROR nova.compute.manager [req-35a2cd70-bd93-43bf-ba82-f1168d82346e 6dcaa7505f9340268bcc7b0f1f11f651 d0c6d5dec6474bd68c880061060199bc] [instance: c377092d-7b82-4a1f-ac53-e4cbd127bdfe] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 771, in _run_instance\n    injected_files, admin_password)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1040, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1480, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1857, in _create_image\n    mandatory=(\'files\',))\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 304, in inject_data\n    fs.setup()\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/disk/vfs/guestfs.py", line 111, in setup\n    {\'imgfile\': self.imgfile, \'e\': e})\n', 'NovaException: Error mounting /var/lib/nova/instances/c377092d-7b82-4a1f-ac53-e4cbd127bdfe/disk with libguestfs (could not create appliance through libvirt: internal error process exited while connecting to monitor: connect(unix:/tmp/libguestfstPXW8g/console.sock): Permission denied\nchardev: opening backend "socket" failed\n [code=1 domain=10])\n']

When this happens the 'disk' being injected was actually owned as root:root:

[root@nova1 instances]# ll c377092d-7b82-4a1f-ac53-e4cbd127bdfe/
total 10252
-rw-rw----. 1 nova nova       0 Feb 21 10:31 console.log
-rw-r--r--. 1 root root  197120 Feb 21 10:31 disk
-rw-r--r--. 1 nova nova 4404752 Feb 21 10:31 kernel
-rw-r--r--. 1 nova nova    1732 Feb 21 10:31 libvirt.xml
-rw-r--r--. 1 nova nova 5882349 Feb 21 10:31 ramdisk

** thus libvirt (qemu) was prevented access.

So I suppose the question here is should we make Nova default the guestfs attach_method to 'appliance'... or perhaps add a config option in Nova to explicitly set this (I would like to avoid new config options if possible).

Using the default setting for attach_method if at all possible would be nice though. Something like:


--- a/nova/virt/disk/vfs/guestfs.py
+++ b/nova/virt/disk/vfs/guestfs.py
@@ -94,6 +94,8 @@ class VFSGuestFS(vfs.VFS):
         self.handle = guestfs.GuestFS()
 
         try:
+            if self.handle.get_attach_method() == 'libvirt':
+                self.handle.set_attach_method('libvirt:' + CONF.libvirt_uri)
             self.handle.add_drive_opts(self.imgfile, format=self.imgfmt)
             self.handle.launch()

(In reply to comment #2)

> -----
> 
> I tried a slightly different fix as well which was to call this instead:
> 
>   self.handle.set_attach_method('libvirt:qemu:///system')
> 
> This seems to cause a slightly different issue related to file permissions
> on the disk image:
> 
> 2013-02-21 10:31:27.448 ERROR nova.compute.manager
> [req-35a2cd70-bd93-43bf-ba82-f1168d82346e 6dcaa7505f9340268bcc7b0f1f11f651
> d0c6d5dec6474bd68c880061060199bc] [instance:
> c377092d-7b82-4a1f-ac53-e4cbd127bdfe] Error: ['Traceback (most recent call
> last):\n', '  File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 771, in
> _run_instance\n    injected_files, admin_password)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1040, in
> _spawn\n    block_device_info)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1480,
> in spawn\n    admin_pass=admin_password)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1857,
> in _create_image\n    mandatory=(\'files\',))\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 304, in
> inject_data\n    fs.setup()\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/disk/vfs/guestfs.py", line 111,
> in setup\n    {\'imgfile\': self.imgfile, \'e\': e})\n', 'NovaException:
> Error mounting
> /var/lib/nova/instances/c377092d-7b82-4a1f-ac53-e4cbd127bdfe/disk with
> libguestfs (could not create appliance through libvirt: internal error
> process exited while connecting to monitor:
> connect(unix:/tmp/libguestfstPXW8g/console.sock): Permission
> denied\nchardev: opening backend "socket" failed\n [code=1 domain=10])\n']

This is a bug Rich has seen with libguestfs + libvirt but so far we've not diagnosed it. We need to investigate this and figure out what is failing, because this is the desired configuration for libguestfs going forward.

> 
> When this happens the 'disk' being injected was actually owned as root:root:

Actually libvirt will change permissions dynamically on startup / shutdown of QEMU, so since QEMU is no longer running, what you're seeing here is not accurate.

> 
> [root@nova1 instances]# ll c377092d-7b82-4a1f-ac53-e4cbd127bdfe/
> total 10252
> -rw-rw----. 1 nova nova       0 Feb 21 10:31 console.log
> -rw-r--r--. 1 root root  197120 Feb 21 10:31 disk
> -rw-r--r--. 1 nova nova 4404752 Feb 21 10:31 kernel
> -rw-r--r--. 1 nova nova    1732 Feb 21 10:31 libvirt.xml
> -rw-r--r--. 1 nova nova 5882349 Feb 21 10:31 ramdisk
> 
> ** thus libvirt (qemu) was prevented access.
> 
> So I suppose the question here is should we make Nova default the guestfs
> attach_method to 'appliance'... or perhaps add a config option in Nova to
> explicitly set this (I would like to avoid new config options if possible).
> 
> Using the default setting for attach_method if at all possible would be nice
> though. Something like:
> 
> 
> --- a/nova/virt/disk/vfs/guestfs.py
> +++ b/nova/virt/disk/vfs/guestfs.py
> @@ -94,6 +94,8 @@ class VFSGuestFS(vfs.VFS):
>          self.handle = guestfs.GuestFS()
>  
>          try:
> +            if self.handle.get_attach_method() == 'libvirt':
> +                self.handle.set_attach_method('libvirt:' + CONF.libvirt_uri)

Yep, this is what we ought to aim for

(In reply to comment #2)
> Okay. That makes sense. I just confirmed the following patch to Nova would
> fix the issue:
> 
> --- a/nova/virt/disk/vfs/guestfs.py
> +++ b/nova/virt/disk/vfs/guestfs.py
> @@ -94,6 +94,7 @@ class VFSGuestFS(vfs.VFS):
>          self.handle = guestfs.GuestFS()
>  
>          try:
> +            self.handle.set_attach_method('appliance')
>              self.handle.add_drive_opts(self.imgfile, format=self.imgfmt)
>              self.handle.launch()
> 
> ----
> 
> This also makes sense because the default guestfs attach_method was
> 'appliance' on Fedora 17. The default is now 'libvirt' on Fedora 18 (which
> clearly breaks things a bit..)

This analysis and preliminary fix is correct, as long as you
understand the limitations of using direct launch ('appliance')
instead of libvirt.  It's possible that none of these apply to
OpenStack, but by using direct launch, you *lose* the following:

 - sVirt confinement of the appliance
   (see the diagram here: http://libguestfs.org/guestfs.3.html#security)

 - ability to hotplug disks
   (http://libguestfs.org/guestfs.3.html#hotplugging)

 - ability to list and manage libguestfs appliances using libvirt
   (eg. virsh list or equivalent libvirt APIs)

> -----
> 
> I tried a slightly different fix as well which was to call this instead:
> 
>   self.handle.set_attach_method('libvirt:qemu:///system')
> 
> This seems to cause a slightly different issue related to file permissions
> on the disk image:
> 
> 2013-02-21 10:31:27.448 ERROR nova.compute.manager
> [req-35a2cd70-bd93-43bf-ba82-f1168d82346e 6dcaa7505f9340268bcc7b0f1f11f651
> d0c6d5dec6474bd68c880061060199bc] [instance:
> c377092d-7b82-4a1f-ac53-e4cbd127bdfe] Error: ['Traceback (most recent call
> last):\n', '  File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 771, in
> _run_instance\n    injected_files, admin_password)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1040, in
> _spawn\n    block_device_info)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1480,
> in spawn\n    admin_pass=admin_password)\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1857,
> in _create_image\n    mandatory=(\'files\',))\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 304, in
> inject_data\n    fs.setup()\n', '  File
> "/usr/lib/python2.7/site-packages/nova/virt/disk/vfs/guestfs.py", line 111,
> in setup\n    {\'imgfile\': self.imgfile, \'e\': e})\n', 'NovaException:
> Error mounting
> /var/lib/nova/instances/c377092d-7b82-4a1f-ac53-e4cbd127bdfe/disk with
> libguestfs (could not create appliance through libvirt: internal error
> process exited while connecting to monitor:
> connect(unix:/tmp/libguestfstPXW8g/console.sock): Permission
> denied\nchardev: opening backend "socket" failed\n [code=1 domain=10])\n']
[...]

As Dan [B] says, please file a separate bug about this.  This isn't
a configuration we've spent any time on, but it needs to be supported
and working.

Okay. A fix for this has gone into upstream here (for Grizzly):

https://github.com/openstack/nova/commit/014499acf5d6d6a557c9415aa49c536817a02a0a

This will make it so that guestfs uses the same libvirt URI as Nova when the attach mode is 'libvirt'.

Closing, since this was not a libvirt bug in the end - instead it was an upstream OpenStack bug for code not in Fedora.

I suspect this fix has cause a regression, see:
https://bugzilla.redhat.com/show_bug.cgi?id=984409#c2