Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1057535
Summary: | libvirt hangs running 'virsh capabilities' | ||||||
---|---|---|---|---|---|---|---|
Product: | [Community] Virtualization Tools | Reporter: | Richard W.M. Jones <rjones> | ||||
Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||
Status: | CLOSED NOTABUG | QA Contact: | |||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | unspecified | CC: | berrange, crobinso, rbalakri | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2016-04-10 17:36:12 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 910269 | ||||||
Attachments: |
|
21825 ? Sl 5:32 /usr/sbin/libvirtd --timeout=30 PID 21825 (libvirtd) is unkillable. It is blocked on a futex: $ strace -p 21825 Process 21825 attached futex(0x7f91806bb664, FUTEX_WAIT_PRIVATE, 1, NULL [hangs here] $ ps -efL | grep libvirtd rjones 21825 1 21825 0 7 Jan16 ? 00:00:58 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 21826 0 7 Jan16 ? 00:00:30 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 21830 0 7 Jan16 ? 00:00:28 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 4519 0 7 Jan16 ? 00:00:30 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 30897 0 7 Jan20 ? 00:00:14 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 464 0 7 Jan20 ? 00:00:13 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 467 0 7 Jan20 ? 00:00:13 /usr/sbin/libvirtd --timeout=30 Stack trace of libvirtd (there are no clients connected): (gdb) t a a bt Thread 7 (Thread 0x7f9176053700 (LWP 21826)): #0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0 #2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0 #3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a871b in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c62167e in qemuDomainCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0 #7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper () #8 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700) at pthread_create.c:309 #13 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7f9174850700 (LWP 21830)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7f916b177700 (LWP 4519)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7f916a175700 (LWP 30897)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7f9169974700 (LWP 464)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7f912bfff700 (LWP 467)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0 #3 0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0 #4 0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0 #5 0x00007f917f75db1f in main () Stack trace with one client hanging in virsh capabilities: (gdb) t a a bt Thread 7 (Thread 0x7f9176053700 (LWP 21826)): #0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0 #2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0 #3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a871b in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c62167e in qemuDomainCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0 #7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper () #8 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700) at pthread_create.c:309 #13 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7f9174850700 (LWP 21830)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7f916b177700 (LWP 4519)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7f916a175700 (LWP 30897)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7f9169974700 (LWP 464)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7f912bfff700 (LWP 467)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0 #2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f916c5a8613 in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a8bfa in virQEMUCapsInit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007f916c621b20 in qemuConnectGetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0 #9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper () #10 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700) at pthread_create.c:309 #15 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0 #3 0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0 #4 0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0 #5 0x00007f917f75db1f in main () Hmm, that stack trace actually shows 5 separate calls to virsh capabilities, all being blocked by a single call to virDomainCreateXML, which is stuck waiting for a QEMU binary to respond to an attempt to query capabilities: Thread 7 (Thread 0x7f9176053700 (LWP 21826)): #0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0 #2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0 #3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f916c5a871b in virQEMUCapsCacheLookup () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f916c62167e in qemuDomainCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0 #7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper () #8 0x00007f917ed8c23a in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0 #10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0 #11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0 #12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700) at pthread_create.c:309 #13 0x00007f917b824ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Is there a hung QEMU process on the machine that you can see that would correspond to this command invocation ? I've just killed (with -9) the libvirtd instance. However I suspect that I know the answer. I was testing out strace of qemu this morning (see: https://www.redhat.com/archives/libguestfs/2014-January/msg00224.html ) which might have caused qemu to hang. Yeah, strace is certainly a likely candidate for messing things up when it re-parents and generally screws around with the process. FWIW, there's a trick to let you attach GDB to a QEMU process without using a wrapper script. https://www.berrange.com/posts/2011/10/12/debugging-early-startup-of-kvm-with-gdb-when-launched-by-libvirtd/ Sounds like this is NOTABUG |
Created attachment 854872 [details] LIBVIRT_DEBUG=1 virsh capabilities Description of problem: $ virsh capabilities hangs, with no further output. LIBVIRT_DEBUG=1 output is attached. Version-Release number of selected component (if applicable): libvirt-client-1.1.3.2-1.fc20.x86_64 libvirt-daemon-1.1.3.2-1.fc20.x86_64 How reproducible: Unknown Steps to Reproduce: 1. Run 'virsh capabilities' Additional info: This is *non root*. This also stopped libguestfs from working since the first thing it does is to query libvirt capabilities.