yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1304481] Re: Nova operates network failed by "Maximum attempts reached" error when Cloud under heavy workload

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Zhi Yan Liu <1304481@xxxxxxxxxxxxxxxxxx>
Date: Fri, 30 May 2014 02:27:30 -0000
Reply-to: Bug 1304481 <1304481@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Invalid => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1304481

Title:
  Nova operates network failed by "Maximum attempts reached" error when
  Cloud under heavy workload

Status in OpenStack Neutron (virtual network service):
  Won't Fix
Status in OpenStack Compute (Nova):
  New

Bug description:
  In my deployment when I provision or terminate numerous VM, e.g. 100+,
  I got some "Maximum attempts reached" issues in Nova internal which
  happened on Nova operates network/neutron by neutronclient. You might
  check follow log snippet for details.

  I propose adding a "neutron_http_retries" option to Nova, and to make
  each neutron client object leverage this option (in
  nova/network/neutronv2/api.py). IMO we could make the fix base on this
  change: https://review.openstack.org/#/c/71464/

  IFY, in my deployment I verified that when I set http retries of
  neutronclient to 5 times (most time 3 is just fine) the problem is
  gone.

  216492 2014-04-08 02:16:09.844 3777 ERROR oslo.messaging._executors.base [-] Exception during message handling
  216493 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base Traceback (most recent call last):
  216494 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch
  216495 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     incoming.reply(self.callback(incoming.ctxt, incoming.message))
  216496 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in __call__
  216497 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     return self._dispatch(endpoint, method, ctxt, args)
  216498 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 92, in _dispatch
  216499 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     result = getattr(endpoint, method)(ctxt, **new_args)
  216500 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
  216501 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     payload)
  216502 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  216503 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     six.reraise(self.type_, self.value, self.tb)
  216504 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
  216505 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     return f(self, context, *args, **kw)
  216506 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 249, in decorated_function
  216507 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     pass
  216508 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  216509 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     six.reraise(self.type_, self.value, self.tb)
  216510 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 235, in decorated_function
  216511 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     return function(self, context, *args, **kwargs)
  216512 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 300, in decorated_function
  216513 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     function(self, context, *args, **kwargs)
  216514 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 277, in decorated_function
  216515 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     e, sys.exc_info())
  216516 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  216517 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     six.reraise(self.type_, self.value, self.tb)
  216518 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 264, in decorated_function
  216519 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     return function(self, context, *args, **kwargs)
  216520 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1894, in run_instance
  216521 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     do_run_instance()
  216522 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  216523 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     return f(*args, **kwargs)
  216524 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1893, in do_run_instance
  216525 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     legacy_bdm_in_spec)
  216526 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1008, in _run_instance
  216527 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     notify("error", msg=unicode(e))  # notify that build failed
  216528 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  216529 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     six.reraise(self.type_, self.value, self.tb)
  216530 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 992, in _run_instance
  216531 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     instance, image_meta, legacy_bdm_in_spec)
  216532 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1110, in _build_instance
  216533 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     filter_properties, bdms, legacy_bdm_in_spec)
  216534 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1154, in _reschedule_or_error
  216535 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     self._log_original_error(exc_info, instance_uuid)
  216536 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  216537 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     six.reraise(self.type_, self.value, self.tb)
  216538 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1149, in _reschedule_or_error
  216539 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     bdms, requested_networks)
  216540 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1920, in _shutdown_instance
  216541 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     network_info = self._get_instance_nw_info(context, instance)
  216542 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 920, in _get_instance_nw_info
  216543 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     instance)
  216544 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 48, in wrapper
  216545 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     res = f(self, context, *args, **kwargs)
  216546 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 451, in get_instance_nw_info
  216547 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     result = self._get_instance_nw_info(context, instance, networks)
  216548 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 458, in _get_instance_nw_info
  216549 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     nw_info = self._build_network_info_model(context, instance, networks)
  216550 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 1022, in _build_network_info_model
  216551 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     data = client.list_ports(**search_opts)
  216552 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
  216553 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     ret = self.function(instance, *args, **kwargs)
  216554 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 306, in list_ports
  216555 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     **_params)
  216556 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1250, in list
  216557 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     for r in self._pagination(collection, path, **params):
  216558 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
  216559 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     res = self.get(path, params=params)
  216560 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1236, in get
  216561 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     headers=headers, params=params)
  216562 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base   File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1228, in retry_request
  216563 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base     raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
  216564 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base ConnectionFailed: Connection to neutron failed: Maximum attempts reached
  216565 2014-04-08 02:16:09.844 3777 TRACE oslo.messaging._executors.base
  216566 2014-04-08 02:16:09.847 3777 ERROR oslo.messaging._drivers.common [-] Returning exception Connection to neutron failed: Maximum attempts reached to caller

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1304481/+subscriptions
References

[Bug 1304481] [NEW] Nova operates network failed by "Maximum attempts reached" error when Cloud under heavy workload
From: Zhi Yan Liu, 2014-04-08