← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #69397

[Bug 1734012] [NEW] Handle exception in get_instance_sorted when scattering gather results from all cells.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Currently, when we get servers list in multi cell, we scatter gather results from cells, but if we get back exception or timeout from cell, we will get 500 error finally.

We should handle raise or timeout after getting back all results.


Maybe we could just skip the error result like quota stuff:
https://github.com/openstack/nova/blob/e9ce5c4c95edc869ab2cf82ca0733a2821c384ad/nova/quota.py#L1865


Steps to reproduce
==================
1. raise some error exception when gather results.

2. get server list
curl -g -i -X GET http://XXX/compute/v2.1/servers -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: $TOKEN"
HTTP/1.1 500 Internal Server Error
Date: Wed, 22 Nov 2017 07:11:39 GMT
Server: Apache/2.4.18 (Ubuntu)
OpenStack-API-Version: compute 2.53
X-OpenStack-Nova-API-Version: 2.53
Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version
Content-Type: application/json; charset=UTF-8
Content-Length: 193
x-openstack-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
x-compute-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
Connection: close

{"computeFault": {"message": "Unexpected API Error. Please report this
at http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible.\n<type 'exceptions.TypeError'>", "code": 500}}

2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 152, in index
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=False)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 308, in _get_servers
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2394, in get_all
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     context, filters, limit, marker, fields, sort_keys, sort_dirs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/instance_list.py", line 251, in get_instance_objects_sorted
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     expected_attrs)
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 1196, in _make_instance_list
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for db_inst in db_inst_list:
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/instance_list.py", line 228, in get_instances_sorted
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for i in heapq.merge(*results.values()):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/heapq.py", line 373, in merge
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for itnum, it in enumerate(map(iter, iterables)):
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions TypeError: 'object' object is not iterable
2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions
2017-11-22 02:11:39.935 9792 INFO nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2017-11-22 02:11:39.937 9792 DEBUG nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029
2017-11-22 02:11:39.938 9792 INFO nova.api.openstack.requestlog [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] 10.76.6.31 "GET /compute/v2.1/servers" status: 500 len: 193 microversion: 2.53 time: 0.746751

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734012

Title:
  Handle exception in get_instance_sorted when scattering gather results
  from all cells.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Currently, when we get servers list in multi cell, we scatter gather results from cells, but if we get back exception or timeout from cell, we will get 500 error finally.

  We should handle raise or timeout after getting back all results.

  
  Maybe we could just skip the error result like quota stuff:
  https://github.com/openstack/nova/blob/e9ce5c4c95edc869ab2cf82ca0733a2821c384ad/nova/quota.py#L1865

  
  Steps to reproduce
  ==================
  1. raise some error exception when gather results.

  2. get server list
  curl -g -i -X GET http://XXX/compute/v2.1/servers -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: $TOKEN"
  HTTP/1.1 500 Internal Server Error
  Date: Wed, 22 Nov 2017 07:11:39 GMT
  Server: Apache/2.4.18 (Ubuntu)
  OpenStack-API-Version: compute 2.53
  X-OpenStack-Nova-API-Version: 2.53
  Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version
  Content-Type: application/json; charset=UTF-8
  Content-Length: 193
  x-openstack-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
  x-compute-request-id: req-429c02e7-84db-4c2d-98fd-dd1af0186383
  Connection: close

  {"computeFault": {"message": "Unexpected API Error. Please report this
  at http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible.\n<type 'exceptions.TypeError'>", "code": 500}}

  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 152, in index
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=False)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 308, in _get_servers
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2394, in get_all
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     context, filters, limit, marker, fields, sort_keys, sort_dirs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/instance_list.py", line 251, in get_instance_objects_sorted
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     expected_attrs)
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 1196, in _make_instance_list
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for db_inst in db_inst_list:
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/instance_list.py", line 228, in get_instances_sorted
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for i in heapq.merge(*results.values()):
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/heapq.py", line 373, in merge
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions     for itnum, it in enumerate(map(iter, iterables)):
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions TypeError: 'object' object is not iterable
  2017-11-22 02:11:39.925 9792 ERROR nova.api.openstack.extensions
  2017-11-22 02:11:39.935 9792 INFO nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>
  2017-11-22 02:11:39.937 9792 DEBUG nova.api.openstack.wsgi [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029
  2017-11-22 02:11:39.938 9792 INFO nova.api.openstack.requestlog [req-429c02e7-84db-4c2d-98fd-dd1af0186383 - admin] 10.76.6.31 "GET /compute/v2.1/servers" status: 500 len: 193 microversion: 2.53 time: 0.746751

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1734012/+subscriptions
  Thread Previous • Date Previous • • Thread Next