yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1204598] Re: Race condition in aggregates list when an aggregate is deleted

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 05 Sep 2013 15:39:31 -0000
Reply-to: Bug 1204598 <1204598@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1204598

Title:
  Race condition in aggregates list when an aggregate is deleted

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  The aggregate list api call will return a 404 not found if an
  aggregate is removed while

  The tempest testr run hit this by running the JSON and XML tests in
  different processes so that an aggregate created for a JSON test was
  deleted from underneath the list command in an XML test while it was
  looping over the list of aggregates. Which led to this stack trace:

  TRACE nova.api.openstack Traceback (most recent call last):
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 110, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 461, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 903, in __call__
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     content_type, body, accept)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 962, in _process_stack
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 1043, in dispatch
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return method(req=request, **action_args)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/contrib/aggregates.py", line 53, in index
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     aggregates = self.api.get_aggregate_list(context)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 2924, in get_aggregate_list
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return [self._get_aggregate_info(context, a) for a in aggregates]
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 3022, in _get_aggregate_info
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     metadata = self.db.aggregate_metadata_get(context, aggregate['id'])
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/db/api.py", line 1663, in aggregate_metadata_get
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return IMPL.aggregate_metadata_get(context, aggregate_id)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 103, in wrapper
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return f(*args, **kwargs)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 148, in wrapper
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     aggregate_get(context, aggregate_id)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 103, in wrapper
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     return f(*args, **kwargs)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4651, in aggregate_get
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack     raise exception.AggregateNotFound(aggregate_id=aggregate_id)
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack AggregateNotFound: Aggregate 21 could not be found.
  2013-07-24 14:18:01.389 20700 TRACE nova.api.openstack 

  Logs from this run can be found here:
  http://logs.openstack.org/79/38379/2/check/gate-tempest-devstack-vm-testr-full/1408/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1204598/+subscriptions