1057535 – libvirt hangs running 'virsh capabilities'

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1057535 - libvirt hangs running 'virsh capabilities'

Summary: libvirt hangs running 'virsh capabilities'

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Virtualization Tools
Classification:	Community
Component:	libvirt
Sub Component:
Version:	unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Libvirt Maintainers
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	TRACKER-bugs-affecting-libguestfs
TreeView+	depends on / blocked

Reported:	2014-01-24 10:00 UTC by Richard W.M. Jones
Modified:	2016-04-10 17:36 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-04-10 17:36:12 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
LIBVIRT_DEBUG=1 virsh capabilities (38.96 KB, text/plain) 2014-01-24 10:00 UTC, Richard W.M. Jones	no flags	Details
View All

Description Richard W.M. Jones 2014-01-24 10:00:44 UTC

Created attachment 854872 [details]
LIBVIRT_DEBUG=1 virsh capabilities

Description of problem:

$ virsh capabilities

hangs, with no further output.

LIBVIRT_DEBUG=1 output is attached.

Version-Release number of selected component (if applicable):

libvirt-client-1.1.3.2-1.fc20.x86_64
libvirt-daemon-1.1.3.2-1.fc20.x86_64

How reproducible:

Unknown

Steps to Reproduce:
1. Run 'virsh capabilities'

Additional info:

This is *non root*.

This also stopped libguestfs from working since the first thing
it does is to query libvirt capabilities.

Comment 1 Richard W.M. Jones 2014-01-24 10:01:03 UTC

21825 ?        Sl     5:32 /usr/sbin/libvirtd --timeout=30

Comment 2 Richard W.M. Jones 2014-01-24 10:05:12 UTC

PID 21825 (libvirtd) is unkillable.

It is blocked on a futex:

$ strace -p 21825
Process 21825 attached
futex(0x7f91806bb664, FUTEX_WAIT_PRIVATE, 1, NULL [hangs here]

$ ps -efL | grep libvirtd
rjones   21825     1 21825  0    7 Jan16 ?        00:00:58 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 21826  0    7 Jan16 ?        00:00:30 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 21830  0    7 Jan16 ?        00:00:28 /usr/sbin/libvirtd --timeout=30
rjones   21825     1  4519  0    7 Jan16 ?        00:00:30 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 30897  0    7 Jan20 ?        00:00:14 /usr/sbin/libvirtd --timeout=30
rjones   21825     1   464  0    7 Jan20 ?        00:00:13 /usr/sbin/libvirtd --timeout=30
rjones   21825     1   467  0    7 Jan20 ?        00:00:13 /usr/sbin/libvirtd --timeout=30

Comment 3 Richard W.M. Jones 2014-01-24 10:09:42 UTC

Stack trace of libvirtd (there are no clients connected):

(gdb) t a a bt

Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3  0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4  0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5  0x00007f917f75db1f in main ()

Comment 4 Richard W.M. Jones 2014-01-24 10:16:58 UTC

Stack trace with one client hanging in virsh capabilities:

(gdb) t a a bt

Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3  0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4  0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5  0x00007f917f75db1f in main ()

Comment 5 Daniel Berrangé 2014-01-24 10:28:23 UTC

Hmm, that stack trace actually shows 5 separate calls to virsh capabilities, all being blocked by a single call to virDomainCreateXML, which is stuck waiting for a QEMU binary to respond to an attempt to query capabilities:


Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Is there a hung QEMU process on the machine that you can see that would correspond to this command invocation ?

Comment 6 Richard W.M. Jones 2014-01-24 10:32:53 UTC

I've just killed (with -9) the libvirtd instance.

However I suspect that I know the answer.  I was testing out
strace of qemu this morning (see:
https://www.redhat.com/archives/libguestfs/2014-January/msg00224.html )
which might have caused qemu to hang.

Comment 7 Daniel Berrangé 2014-01-24 10:36:56 UTC

Yeah, strace is certainly a likely candidate for messing things up when it re-parents and generally screws around with the process.

FWIW, there's a trick to let you attach GDB to a QEMU process without using a wrapper script. 

https://www.berrange.com/posts/2011/10/12/debugging-early-startup-of-kvm-with-gdb-when-launched-by-libvirtd/

Comment 8 Cole Robinson 2016-04-10 17:36:12 UTC

Sounds like this is NOTABUG

Note You need to log in before you can comment on or make changes to this bug.