yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1643444] Re: TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 02 Oct 2018 15:20:43 -0000
Reply-to: Bug 1643444 <1643444@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** No longer affects: nova/ocata

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1643444

Title:
  TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Committed

Bug description:
  I have a newton setup with 3 api(controller) node.

  TenantUsagesTestJSON.test_list_usage_all_tenants failed once , the
  failure looks similar to one described in the already fixed in
  https://bugs.launchpad.net/nova/+bug/1487570 , but it is different api
  call so it can have similar issue.

  Likely you have an old list of ids, and trying to fetch more info
  about an already deleted instance.

  
  The tempest exception:

  
  2016-11-20 00:07:18,606 27600 INFO     [tempest.lib.common.rest_client] Request (TenantUsagesTestJSON:test_list_usage_all_tenants): 500 GET http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 0.134s
  2016-11-20 00:07:18,607 27600 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'status': '500', 'content-length': '205', 'content-location': 'http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313', 'x-compute-request-id': 'req-3ff84c48-b03e-4f23-8f33-227719a0ced4', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Sun, 20 Nov 2016 05:07:18 GMT', 'content-type': 'application/json; charset=UTF-8'}
          Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.InstanceNotFound'>", "code": 500}}
    File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 73, in test_list_usage_all_tenants
      start=self.start, end=self.end, detailed="1")['tenant_usages'][0]
    File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 63, in call_until_valid
      self.assertEqual(test_utils.call_until_true(is_valid, duration, 1),
    File "/home/stack/tempest-dir/tempest/lib/common/utils/test_utils.py", line 103, in call_until_true
      if func():
    File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 59, in is_valid
      self.resp = func(*args, **kwargs)
    File "/home/stack/tempest-dir/tempest/lib/services/compute/tenant_usages_client.py", line 37, in list_tenant_usages
      resp, body = self.get(url)
    File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 291, in get
      return self.request('GET', url, extra_headers, headers)
    File "/home/stack/tempest-dir/tempest/lib/services/compute/base_compute_client.py", line 48, in request
      method, url, extra_headers, headers, body, chunked)
    File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 664, in request
      self._error_checker(resp, resp_body)
    File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 827, in _error_checker
      message=message)
  tempest.lib.exceptions.ServerFault: Got server fault
  Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceNotFound'>


  The related nova api log (node-2):
  2016-11-20 05:07:18.476 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Calling method '<bound method SimpleTenantUsageController.index of <nova.api.openstack.compute.simple_tenant_usage.SimpleTenantUsageController object at 0x7fc62d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2016-11-20 05:07:18.507 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013
  2016-11-20 05:07:18.556 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Unexpected exception in API method
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     detailed=detailed)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     info['memory_mb'] = instance.flavor.memory_mb
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     self._load_flavor()
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions InstanceNotFound: Instance 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 could not be found.
  2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions 
  2016-11-20 05:07:18.601 111884 INFO nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceNotFound'>
  2016-11-20 05:07:18.601 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceNotFound'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044
  2016-11-20 05:07:18.602 111884 INFO nova.osapi_compute.wsgi.server [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] fd00:fd00:fd00:2000::11 "GET /v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 HTTP/1.1" status: 500 len: 567 time: 0.1304212
  2016-11-20 05:07:18.929 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?tenant_id=72ac28353e7f4c03a79ba6abd32c189a&shared=False used request id req-9f2b5b0f-5b12-4cb4-981a-000e3bb3fe8c _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127
  2016-11-20 05:07:18.989 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?shared=True used request id req-4bdabb6e-cbd8-449d-a4b8-61355625bd88 _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1643444/+subscriptions
References

[Bug 1643444] [NEW] TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer
From: Attila Fazekas, 2016-11-21