yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1352659] [NEW] race in server show api

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Christopher Yeoh <cyeoh@xxxxxxxxxxx>
Date: Tue, 05 Aug 2014 03:46:55 -0000
Reply-to: Bug 1352659 <1352659@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Because of the instance object lazy loading its possible to get into
situations where the API code is half way through assembling data to
return to the client when the instance disappears underneath it. We
really need to ensure everything we will need is retreived up front so
we have a consistent snapshot view of the instance

[req-5ca39eb3-c1d2-433b-8dac-1bf5f338ce1f ServersAdminNegativeV3Test-1453501114 ServersAdminNegativeV3Test-364813115] Unexpected exception in API method
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 473, in wrapped
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/plugins/v3/servers.py", line 410, in show
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     return self._view_builder.show(req, instance)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 268, in show
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     _inst_fault = self._get_fault(request, instance)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 214, in _get_fault
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     fault = instance.fault
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/base.py", line 67, in getter
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     self.obj_load_attr(name)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 520, in obj_load_attr
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     expected_attrs=[attrname])
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/base.py", line 153, in wrapper
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 310, in get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 676, in instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1715, in instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1727, in _instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions InstanceNotFound: Instance fcff276a-d410-4760-9b98-4014024b1353 could not be found.
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions 

        http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-
nova-v3-full-master/a278802/logs/screen-n-api.txt

** Affects: nova
     Importance: Medium
     Assignee: Christopher Yeoh (cyeoh-0)
         Status: New


** Tags: api

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1352659

Title:
  race in server show api

Status in OpenStack Compute (Nova):
  New

Bug description:
  Because of the instance object lazy loading its possible to get into
  situations where the API code is half way through assembling data to
  return to the client when the instance disappears underneath it. We
  really need to ensure everything we will need is retreived up front so
  we have a consistent snapshot view of the instance

  [req-5ca39eb3-c1d2-433b-8dac-1bf5f338ce1f ServersAdminNegativeV3Test-1453501114 ServersAdminNegativeV3Test-364813115] Unexpected exception in API method
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions Traceback (most recent call last):
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 473, in wrapped
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/plugins/v3/servers.py", line 410, in show
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     return self._view_builder.show(req, instance)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 268, in show
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     _inst_fault = self._get_fault(request, instance)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 214, in _get_fault
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     fault = instance.fault
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/base.py", line 67, in getter
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     self.obj_load_attr(name)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 520, in obj_load_attr
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     expected_attrs=[attrname])
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/base.py", line 153, in wrapper
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 310, in get_by_uuid
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     use_slave=use_slave)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 676, in instance_get_by_uuid
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1715, in instance_get_by_uuid
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1727, in _instance_get_by_uuid
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions InstanceNotFound: Instance fcff276a-d410-4760-9b98-4014024b1353 could not be found.
  2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions 

          http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-
  nova-v3-full-master/a278802/logs/screen-n-api.txt

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1352659/+subscriptions
Follow ups

[Bug 1352659] Re: race in server show api
From: Thierry Carrez, 2014-09-05
[Bug 1352659] [NEW] race in server show api
From: Christopher Yeoh, 2014-08-05
References

[Bug 1352659] [NEW] race in server show api
From: Christopher Yeoh, 2014-08-05