yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1443483] [NEW] 'nova list' does not work when #ports*#instances is big number

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miroslav Suchý <1443483@xxxxxxxxxxxxxxxxxx>
Date: Mon, 13 Apr 2015 14:46:46 -0000
Reply-to: Bug 1443483 <1443483@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have OpenStack Icehouse and I got this error:
[root@fed-cloud09 ~(keystone_admin)]# nova list
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3b8fc3e4-310b-482c-ba20-0666502a1bcb)

And in /var/log/nova/nova-api.log there is:
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack Traceback (most recent call last):
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return req.get_response(self.application)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 679, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return self.app(env, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     content_type, body, accept)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 1009, in _process_stack
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     request, action_args)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     **action_args)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     servers))
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     **_params)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1250, in list
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     res = self.get(path, params=params)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1236, in get
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     headers=headers, params=params)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     headers=headers, params=params)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1164, in do_request
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     message=message)
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack NeutronClientException: <html><body><h1>400 Bad request</h1>
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack Your browser sent an invalid request.
2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack </body></html>


After some investigation I find that the problem seems to be in 
  self._get_ports_from_server_list() which request (when incepted) by GET method.
when I debug do_request() of Client class. I found that the it tries to do:
 /v2.0/ports.json?device_id=0980a371-4991-466b-a471-e847c7d605aa&device_id=f9f8c2d8-7665-4ee8-b6e1-5446b61a1e55&device_id=3055cab9-c0a1-454b-88c3-f34ff92fbc1d&device_id=6d6e64b4-b043-4ba4-93b6-92c06aa7f4e9........

Which is likely too long.

Steps to reproduce:
1. Have 11 security groups
2. spin up 200 instances (both numbers are for one tenant)
3. try to do "nova list" on command line

This way the GET request have  aprox. 9400 bytes, which exceed usual implementation of 8K.
I assume that correct way would be to send this query via POST.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1443483

Title:
  'nova list' does not work when #ports*#instances is big number

Status in OpenStack Compute (Nova):
  New

Bug description:
  I have OpenStack Icehouse and I got this error:
  [root@fed-cloud09 ~(keystone_admin)]# nova list
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3b8fc3e4-310b-482c-ba20-0666502a1bcb)

  And in /var/log/nova/nova-api.log there is:
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack Traceback (most recent call last):
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 679, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return self.app(env, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     content_type, body, accept)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 1009, in _process_stack
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     request, action_args)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     **action_args)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     servers))
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 306, in list_ports
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     **_params)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1250, in list
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     res = self.get(path, params=params)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1236, in get
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     headers=headers, params=params)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     headers=headers, params=params)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1164, in do_request
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1134, in _handle_fault_response
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack     message=message)
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack NeutronClientException: <html><body><h1>400 Bad request</h1>
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack Your browser sent an invalid request.
  2015-04-13 13:47:22.065 18123 TRACE nova.api.openstack </body></html>

  
  After some investigation I find that the problem seems to be in 
    self._get_ports_from_server_list() which request (when incepted) by GET method.
  when I debug do_request() of Client class. I found that the it tries to do:
   /v2.0/ports.json?device_id=0980a371-4991-466b-a471-e847c7d605aa&device_id=f9f8c2d8-7665-4ee8-b6e1-5446b61a1e55&device_id=3055cab9-c0a1-454b-88c3-f34ff92fbc1d&device_id=6d6e64b4-b043-4ba4-93b6-92c06aa7f4e9........

  Which is likely too long.

  Steps to reproduce:
  1. Have 11 security groups
  2. spin up 200 instances (both numbers are for one tenant)
  3. try to do "nova list" on command line

  This way the GET request have  aprox. 9400 bytes, which exceed usual implementation of 8K.
  I assume that correct way would be to send this query via POST.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1443483/+subscriptions
Follow ups

[Bug 1443483] Re: 'nova list' does not work when #ports*#instances is big number
From: Liping Mao, 2015-08-28
[Bug 1443483] Re: 'nova list' does not work when #ports*#instances is big number
From: Launchpad Bug Tracker, 2015-06-28
[Bug 1443483] [NEW] 'nova list' does not work when #ports*#instances is big number
From: Launchpad Bug Tracker, 2015-04-13
[Bug 1443483] [NEW] 'nova list' does not work when #ports*#instances is big number
From: Miroslav Suchý, 2015-04-13
References

[Bug 1443483] [NEW] 'nova list' does not work when #ports*#instances is big number
From: Miroslav Suchý, 2015-04-13