Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 981729
Summary: | Improve handling of "max_clients" setting | |||
---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Daniel Berrangé <berrange> | |
Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | |
Severity: | unspecified | Docs Contact: | ||
Priority: | unspecified | |||
Version: | 7.0 | CC: | acathrow, ajia, dallan, dyuan, gsun, jdenemar, lsu, mprivozn, rjones, weizhan, zpeng | |
Target Milestone: | rc | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | libvirt-1.1.1-3.el7 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 992980 1058606 1070221 (view as bug list) | Environment: | ||
Last Closed: | 2014-06-13 10:00:59 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 910269, 992980, 1058606, 1086175 |
Description
Daniel Berrangé
2013-07-05 15:32:09 UTC
# tail -2 /etc/libvirt/libvirtd.conf max_clients = 20 max_workers = 20 # for i in {1..30}; do virt-sandbox-service create -C -u httpd.service -N dhcp myapache$i;done # for i in {1..30}; do virt-sandbox-service create start myapache$i & done XXX # Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe And check libvirtd log: 2013-07-08 10:13:58.933+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 2013-07-08 10:13:58.941+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 2013-07-08 10:13:58.943+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 I've just proposed patches upstream: https://www.redhat.com/archives/libvir-list/2013-July/msg01646.html Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-August/msg00079.html NB the upstream patches only implement half of this bug. There's still no separation of the limits for anonymous vs authenticated clients. Ah, then I shouldn't have moved this to POST. Sorry. After IRC discussion with Dan, we agreed to split this bug into two. The first part which is done (introducing "max_client" setting) is to be done in this bug. For the issue Dan's mentioning in comment #5 I've cloned this bug into bug 992980. Hence moving to POST again. (In reply to Alex Jia from comment #2) > # tail -2 /etc/libvirt/libvirtd.conf > max_clients = 20 > max_workers = 20 # tail -3 /etc/libvirt/libvirtd.conf max_clients = 20 max_workers = 20 max_queued_clients = 20 > # for i in {1..30}; do virt-sandbox-service create -C -u httpd.service -N > dhcp myapache$i;done > > # for i in {1..30}; do virt-sandbox-service start myapache$i & done > # rpm -q libvirt-sandbox libvirt kernel libvirt-sandbox-0.5.0-6.el7.x86_64 libvirt-1.1.1-13.el7.x86_64 kernel-3.10.0-0.rc7.64.el7.x86_64 Using new 'virsh' method parallel to start containers: # for i in {1..30}; do virsh -c lxc:/// start myapache$i & done And can't hit the following issues, Michal, is it an expected result? or I must run many more containers to reproduce this? > > Unable to open connection: Unable to open lxc:///: Cannot write data: Broken > pipe > > And check libvirtd log: > > 2013-07-08 10:13:58.933+0000: 8034: error : virNetServerAddClient:262 : Too > many active clients (20), dropping connection from 127.0.0.1;0 <slice> 2013-12-02 08:52:16.173+0000: 12384: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 51 2013-12-02 08:52:16.178+0000: 12383: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 56 2013-12-02 08:52:16.183+0000: 12021: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 54 2013-12-02 08:52:16.191+0000: 12380: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 69 2013-12-02 08:52:16.198+0000: 15201: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 75 2013-12-02 08:52:16.202+0000: 15202: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 62 2013-12-02 08:52:16.203+0000: 12387: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 83 2013-12-02 08:52:16.212+0000: 15359: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 81 2013-12-02 08:52:16.227+0000: 12378: debug : lxcContainerWaitForContinue:392 : Wait continue on fd 67 2013-12-02 08:52:16.291+0000: 12384: debug : lxcContainerWaitForContinue:394 : Got continue on fd 51 1 2013-12-02 08:52:16.292+0000: 12018: debug : virLXCMonitorHandleEventInit:107 : Event init 19730 2013-12-02 08:52:16.292+0000: 12384: debug : virDomainFree:2428 : dom=0x7f9258014df0, (VM: name=myapache24, uuid=b295c4f7-7921-46e7-8142-ed795724671e) 2013-12-02 08:52:16.293+0000: 12024: debug : virDomainLookupByUUID:2186 : conn=0x7f929c004560, uuid=b295c4f7-7921-46e7-8142-ed795724671e 2013-12-02 08:52:16.293+0000: 12024: debug : virDomainFree:2428 : dom=0x7f9284008db0, (VM: name=myapache24, uuid=b295c4f7-7921-46e7-8142-ed795724671e) 2013-12-02 08:52:16.294+0000: 12018: debug : virConnectClose:1523 : conn=0x7f929c004560 2013-12-02 08:52:16.308+0000: 12383: debug : lxcContainerWaitForContinue:394 : Got continue on fd 56 1 2013-12-02 08:52:16.308+0000: 12018: debug : virLXCMonitorHandleEventInit:107 : Event init 19776 2013-12-02 08:52:16.309+0000: 12383: debug : virDomainFree:2428 : dom=0x7f926400d3b0, (VM: name=myapache27, uuid=0d1096c3-792c-4be8-a701-3e0067d12e0a) 2013-12-02 08:52:16.310+0000: 12385: debug : virDomainLookupByUUID:2186 : conn=0x7f9298011110, uuid=0d1096c3-792c-4be8-a701-3e0067d12e0a 2013-12-02 08:52:16.310+0000: 12385: debug : virDomainFree:2428 : dom=0x7f925c010e60, (VM: name=myapache27, uuid=0d1096c3-792c-4be8-a701-3e0067d12e0a) 2013-12-02 08:52:16.312+0000: 12018: debug : virConnectClose:1523 : conn=0x7f9298011110 </slice> Daniel, I can successfully start 41 containers not 40 now, is it an expected result? # tail -3 /etc/libvirt/libvirtd.conf max_clients = 20 max_workers = 20 max_queued_clients = 20 # for i in {1..50}; do virt-sandbox-service create -C -u httpd.service -N dhcp myapache$i;done # for i in {1..50}; do virsh -c lxc:/// start myapache$i & done # virsh -c lxc:/// -q list |wc -l 41 # rpm -q libvirt-daemon libvirt-sandbox kernel libvirt-daemon-1.1.1-23.el7.x86_64 libvirt-sandbox-0.5.0-9.el7.x86_64 kernel-3.10.0-86.el7.x86_64 Additional info: error: Failed to start domain myapache36 error: internal error: Failed to allocate free veth pair after 10 attempts error: Failed to start domain myapache29 error: internal error: Failed to allocate free veth pair after 10 attempts NOTE: Maybe, 10 attempts are too few for some users then they possibly want to change this, so I think it will be better if we have a configuration item for it, otherwise, we should document 10 attempts in libvirtd.conf or relevant guide. (In reply to Alex Jia from comment #10) > Daniel, I can successfully start 41 containers not 40 now, is it an expected > result? > > # tail -3 /etc/libvirt/libvirtd.conf > max_clients = 20 > max_workers = 20 > max_queued_clients = 20 > > # for i in {1..50}; do virt-sandbox-service create -C -u httpd.service -N > dhcp myapache$i;done > > # for i in {1..50}; do virsh -c lxc:/// start myapache$i & done > > # virsh -c lxc:/// -q list |wc -l > 41 Yes and no. Kernel does some caching on sockets and some partial opening even if the server is not currently responsive too. So you may end up with more than 40 guests running. Hence I think anything above or equal to 40 is okay. > > # rpm -q libvirt-daemon libvirt-sandbox kernel > libvirt-daemon-1.1.1-23.el7.x86_64 > libvirt-sandbox-0.5.0-9.el7.x86_64 > kernel-3.10.0-86.el7.x86_64 > > Additional info: > > error: Failed to start domain myapache36 > error: internal error: Failed to allocate free veth pair after 10 attempts > > error: Failed to start domain myapache29 > error: internal error: Failed to allocate free veth pair after 10 attempts > This is an internal (buggy) implementation. Let me see if I can fix this. Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2014-February/msg01548.html Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-February/msg00829.html So, after discussion on my backport, the bug raised in comment 10 is a separate issue and deserves own bug. I'm moving this back to MODIFIED, as the request is complete and creating a new bug for the veth issue: bug 1070221. Move to VERIFIED since the separate bug is filed and already verified. This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. |