yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1487570] [NEW] test_list_servers_by_admin_with_all_tenants fails with InstanceNotFound trying to lazy-load flavor

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Fri, 21 Aug 2015 18:08:25 -0000
Reply-to: Bug 1487570 <1487570@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/70/215170/1/check/gate-tempest-dsvm-
nova-v21-full/3fdc0d6/console.html#_2015-08-21_16_04_53_513

2015-08-21 16:04:53.514 | Captured traceback:
2015-08-21 16:04:53.514 | ~~~~~~~~~~~~~~~~~~~
2015-08-21 16:04:53.514 |     Traceback (most recent call last):
2015-08-21 16:04:53.514 |       File "tempest/api/compute/admin/test_servers.py", line 81, in test_list_servers_by_admin_with_all_tenants
2015-08-21 16:04:53.514 |         body = self.client.list_servers(detail=True, **params)
2015-08-21 16:04:53.514 |       File "tempest/services/compute/json/servers_client.py", line 159, in list_servers
2015-08-21 16:04:53.514 |         resp, body = self.get(url)
2015-08-21 16:04:53.514 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 271, in get
2015-08-21 16:04:53.514 |         return self.request('GET', url, extra_headers, headers)
2015-08-21 16:04:53.514 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 643, in request
2015-08-21 16:04:53.515 |         resp, resp_body)
2015-08-21 16:04:53.515 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 754, in _error_checker
2015-08-21 16:04:53.515 |         raise exceptions.ServerFault(resp_body, message=message)
2015-08-21 16:04:53.515 |     tempest_lib.exceptions.ServerFault: Got server fault
2015-08-21 16:04:53.515 |     Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2015-08-21 16:04:53.515 |     <class 'nova.exception.InstanceNotFound'>

There is a trace in the n-api logs when trying to lazy-load a flavor on
an instance:

http://logs.openstack.org/70/215170/1/check/gate-tempest-dsvm-
nova-v21-full/3fdc0d6/logs/screen-n-api.txt.gz?level=TRACE#_2015-08-21_15_39_06_148

2015-08-21 15:39:06.148 ERROR nova.api.openstack.extensions [req-5eca1fa9-7948-4a3d-bc80-7e84441bb74e tempest-ServersAdminTestJSON-973647583 tempest-ServersAdminTestJSON-692147874] Unexpected exception in API method
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 264, in detail
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 389, in _get_servers
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     response = self._view_builder.detail(req, instance_list)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 126, in detail
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return self._list_view(self.show, request, instances, coll_name)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 138, in _list_view
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     server_list = [func(request, server)["server"] for server in servers]
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 266, in show
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     "flavor": self._get_flavor(request, instance),
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 198, in _get_flavor
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     instance_type = instance.get_flavor()
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 890, in get_flavor
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return getattr(self, attr)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 65, in getter
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 880, in obj_load_attr
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     self._load_flavor()
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 811, in _load_flavor
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 169, in wrapper
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 435, in get_by_uuid
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     use_slave=use_slave)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 644, in instance_get_by_uuid
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 230, in wrapper
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1722, in instance_get_by_uuid
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1734, in _instance_get_by_uuid
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions InstanceNotFound: Instance b4025d7a-c2ef-4234-a6ee-28edf966bfe7 could not be found.
2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions 

So between the time that we've retrieved the flavor from the database
and the time we try to lazy-load the flavor, the instance is deleted and
this fails.

We could pre-load the flavor in this case, or read_deleted='yes' when
we're lazy-loading a flavor.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: api unified-objects

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1487570

Title:
  test_list_servers_by_admin_with_all_tenants fails with
  InstanceNotFound trying to lazy-load flavor

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/70/215170/1/check/gate-tempest-dsvm-
  nova-v21-full/3fdc0d6/console.html#_2015-08-21_16_04_53_513

  2015-08-21 16:04:53.514 | Captured traceback:
  2015-08-21 16:04:53.514 | ~~~~~~~~~~~~~~~~~~~
  2015-08-21 16:04:53.514 |     Traceback (most recent call last):
  2015-08-21 16:04:53.514 |       File "tempest/api/compute/admin/test_servers.py", line 81, in test_list_servers_by_admin_with_all_tenants
  2015-08-21 16:04:53.514 |         body = self.client.list_servers(detail=True, **params)
  2015-08-21 16:04:53.514 |       File "tempest/services/compute/json/servers_client.py", line 159, in list_servers
  2015-08-21 16:04:53.514 |         resp, body = self.get(url)
  2015-08-21 16:04:53.514 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 271, in get
  2015-08-21 16:04:53.514 |         return self.request('GET', url, extra_headers, headers)
  2015-08-21 16:04:53.514 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 643, in request
  2015-08-21 16:04:53.515 |         resp, resp_body)
  2015-08-21 16:04:53.515 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 754, in _error_checker
  2015-08-21 16:04:53.515 |         raise exceptions.ServerFault(resp_body, message=message)
  2015-08-21 16:04:53.515 |     tempest_lib.exceptions.ServerFault: Got server fault
  2015-08-21 16:04:53.515 |     Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  2015-08-21 16:04:53.515 |     <class 'nova.exception.InstanceNotFound'>

  There is a trace in the n-api logs when trying to lazy-load a flavor
  on an instance:

  http://logs.openstack.org/70/215170/1/check/gate-tempest-dsvm-
  nova-v21-full/3fdc0d6/logs/screen-n-api.txt.gz?level=TRACE#_2015-08-21_15_39_06_148

  2015-08-21 15:39:06.148 ERROR nova.api.openstack.extensions [req-5eca1fa9-7948-4a3d-bc80-7e84441bb74e tempest-ServersAdminTestJSON-973647583 tempest-ServersAdminTestJSON-692147874] Unexpected exception in API method
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 478, in wrapped
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 264, in detail
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 389, in _get_servers
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     response = self._view_builder.detail(req, instance_list)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 126, in detail
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return self._list_view(self.show, request, instances, coll_name)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 138, in _list_view
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     server_list = [func(request, server)["server"] for server in servers]
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 266, in show
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     "flavor": self._get_flavor(request, instance),
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 198, in _get_flavor
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     instance_type = instance.get_flavor()
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 890, in get_flavor
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return getattr(self, attr)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 65, in getter
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 880, in obj_load_attr
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     self._load_flavor()
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 811, in _load_flavor
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 169, in wrapper
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 435, in get_by_uuid
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 644, in instance_get_by_uuid
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 230, in wrapper
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1722, in instance_get_by_uuid
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1734, in _instance_get_by_uuid
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions InstanceNotFound: Instance b4025d7a-c2ef-4234-a6ee-28edf966bfe7 could not be found.
  2015-08-21 15:39:06.148 22838 ERROR nova.api.openstack.extensions 

  So between the time that we've retrieved the flavor from the database
  and the time we try to lazy-load the flavor, the instance is deleted
  and this fails.

  We could pre-load the flavor in this case, or read_deleted='yes' when
  we're lazy-loading a flavor.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1487570/+subscriptions
Follow ups

[Bug 1487570] Re: test_list_servers_by_admin_with_all_tenants fails with InstanceNotFound trying to lazy-load flavor
From: Thierry Carrez, 2015-09-03