← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #01604

[Bug 1154292] Re: run_instance should not traceback if instance is deleted

  Thread Previous • Date Previous • • Thread Next 
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1154292

Title:
  run_instance should not traceback if instance is deleted

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  Right now, if an instance is deleted before we grab the lock to go run
  it, we harmlessly but loudly traceback on the failure:

  2013-03-12 18:57:11.487 DEBUG nova.compute.manager [req-c65d021f-ef0e-42f0-957d-f98f0d2df5ed ListServersNegativeTest549863-user ListServersNegativeTest549863-tenant] [instance: 6799974f-2cad-4cf1-8b67-20e421180b0f] Instance has been destroyed from under us while trying to set it to ERROR _set_instance_error_state /opt/stack/new/nova/nova/compute/manager.py:388
  2013-03-12 18:57:11.723 ERROR nova.openstack.common.rpc.amqp [req-c65d021f-ef0e-42f0-957d-f98f0d2df5ed ListServersNegativeTest549863-user ListServersNegativeTest549863-tenant] Exception during message handling
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/exception.py", line 117, in wrapped
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/exception.py", line 94, in wrapped
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     pass
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 196, in decorated_function
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 261, in decorated_function
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 225, in decorated_function
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 1168, in run_instance
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     do_run_instance()
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 242, in inner
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 1167, in do_run_instance
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     admin_password, is_first_time, node, instance)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 826, in _run_instance
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     self._set_instance_error_state(context, instance['uuid'])
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 762, in _run_instance
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     self._start_building(context, instance)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 994, in _start_building
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     None))
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/compute/manager.py", line 372, in _instance_update
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/conductor/api.py", line 380, in instance_update
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     updates, 'conductor')
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 107, in instance_update
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     version='1.38')
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return rpc.call(context, self._get_topic(topic), msg, timeout)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 798, in call
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 613, in call
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     rv = list(rv)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 562, in __iter__
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     raise result
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp InstanceNotFound_Remote: Instance 6799974f-2cad-4cf1-8b67-20e421180b0f could not be found.
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/openstack/common/rpc/common.py", line 407, in catch_client_exception
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return func(*args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/conductor/manager.py", line 96, in instance_update
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     context, instance_uuid, updates)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/api.py", line 686, in instance_update_and_get_original
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     rv = IMPL.instance_update_and_get_original(context, instance_uuid, values)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 112, in wrapper
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1767, in instance_update_and_get_original
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     copy_old_instance=True)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1802, in _instance_update
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     session=session)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 112, in wrapper
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1492, in _instance_get_by_uuid
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp     raise exception.InstanceNotFound(instance_id=uuid)
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp InstanceNotFound: Instance 6799974f-2cad-4cf1-8b67-20e421180b0f could not be found.
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp 
  2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1154292/+subscriptions
  Thread Previous • Date Previous • • Thread Next