yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1321626] [NEW] Error during the termination of large group of VMs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Timur Nurlygayanov <tnurlygayanov@xxxxxxxxxxxx>
Date: Wed, 21 May 2014 07:42:05 -0000
Reply-to: Bug 1321626 <1321626@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Environment:
OpenStack Havanna, CentOS 6.4, Neutron with GRE, 1 controller node and 4 compute nodes.

Steps To Reproduce:
1. Create 200 VMs with Cirros Image.
2. Delete 100+ VMs with one request (in Horizon dashboard)

Observed Result:
part of VMs will be successfully removed, but when we select the large group of servers, we can see the error message in Horizon and the traceback in nova.log:

-288a-4c9f-96cb-4e01bebceb4d HTTP/1.1" status: 204 len: 179 time: 0.1963580
<182>May 21 07:36:20 node-4 nova-nova.osapi_compute.wsgi.server INFO: (17832) accepted ('198.11.197.103', 47982)
<0>May 21 07:36:50 node-4 �<179>nova-nova.api.openstack ERROR: Caught error: Request Failed: internal server error while processing your request.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 119, in __call__
    return req.get_response(self.application)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 545, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 938, in __call__
    content_type, body, accept)
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1023, in _process_stack
    request, action_args)
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 911, in post_process_extensions
    **action_args)
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 578, in detail
    self._extend_servers(req, list(resp_obj.obj['servers']))
  File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 527, in _extend_servers
    servers))
  File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 348, in get_instances_security_groups_bindings
    security_groups = self._get_secgroups_from_port_list(ports, neutron)
  File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 332, in _get_secgroups_from_port_list
    search_results = neutron.list_security_groups(**sg_search_opts)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 474, in list_security_groups
    retrieve_all, **_params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1250, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1236, in get
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1164, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
    message=error_dict)
NeutronClientException: Request Failed: internal server error while processing your request.
<0>May 21 07:36:50 node-4 �<182>nova-nova.api.openstack INFO: http://198.11.197.103:8774/v2/cda5c4502688428d93e6570696716717/servers/detail?project_id=cda5c4502688428d93e6570696716717&limit=201 returned with HTTP 500
<182>May 21 07:36:50 node-4 nova-nova.osapi_compute.wsgi.server INFO: 198.11.197.103 "GET /v2/cda5c4502688428d93e6570696716717/servers/detail?project_id=cda5c4502688428d93e6570696716717&limit=201 HTTP/1.1" status: 500 len: 335 time: 29.3159540

** Affects: neutron
     Importance: Undecided
         Status: New

** Summary changed:

- Error during the group of VMs termination
+ Error during the termination of large group of VMs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1321626

Title:
  Error during the termination of large group of VMs

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Environment:
  OpenStack Havanna, CentOS 6.4, Neutron with GRE, 1 controller node and 4 compute nodes.

  Steps To Reproduce:
  1. Create 200 VMs with Cirros Image.
  2. Delete 100+ VMs with one request (in Horizon dashboard)

  Observed Result:
  part of VMs will be successfully removed, but when we select the large group of servers, we can see the error message in Horizon and the traceback in nova.log:

  -288a-4c9f-96cb-4e01bebceb4d HTTP/1.1" status: 204 len: 179 time: 0.1963580
  <182>May 21 07:36:20 node-4 nova-nova.osapi_compute.wsgi.server INFO: (17832) accepted ('198.11.197.103', 47982)
  <0>May 21 07:36:50 node-4 �<179>nova-nova.api.openstack ERROR: Caught error: Request Failed: internal server error while processing your request.
  Traceback (most recent call last):
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 119, in __call__
      return req.get_response(self.application)
    File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 545, in __call__
      return self.app(env, start_response)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 938, in __call__
      content_type, body, accept)
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1023, in _process_stack
      request, action_args)
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 911, in post_process_extensions
      **action_args)
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 578, in detail
      self._extend_servers(req, list(resp_obj.obj['servers']))
    File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 527, in _extend_servers
      servers))
    File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 348, in get_instances_security_groups_bindings
      security_groups = self._get_secgroups_from_port_list(ports, neutron)
    File "/usr/lib/python2.6/site-packages/nova/network/security_group/neutron_driver.py", line 332, in _get_secgroups_from_port_list
      search_results = neutron.list_security_groups(**sg_search_opts)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
      ret = self.function(instance, *args, **kwargs)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 474, in list_security_groups
      retrieve_all, **_params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1250, in list
      for r in self._pagination(collection, path, **params):
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
      res = self.get(path, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1236, in get
      headers=headers, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
      headers=headers, params=params)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1164, in do_request
      self._handle_fault_response(status_code, replybody)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
      exception_handler_v20(status_code, des_error_body)
    File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
      message=error_dict)
  NeutronClientException: Request Failed: internal server error while processing your request.
  <0>May 21 07:36:50 node-4 �<182>nova-nova.api.openstack INFO: http://198.11.197.103:8774/v2/cda5c4502688428d93e6570696716717/servers/detail?project_id=cda5c4502688428d93e6570696716717&limit=201 returned with HTTP 500
  <182>May 21 07:36:50 node-4 nova-nova.osapi_compute.wsgi.server INFO: 198.11.197.103 "GET /v2/cda5c4502688428d93e6570696716717/servers/detail?project_id=cda5c4502688428d93e6570696716717&limit=201 HTTP/1.1" status: 500 len: 335 time: 29.3159540

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1321626/+subscriptions
Follow ups

[Bug 1321626] [NEW] Error during the termination of large group of VMs
From: Timur Nurlygayanov, 2014-05-21
References

[Bug 1321626] [NEW] Error during the termination of large group of VMs
From: Timur Nurlygayanov, 2014-05-21