yahoo-eng-team team mailing list archive

[Rui Chen <chenrui.momo@xxxxxxxxx>]

Public bug reported:

The admin context is used when nova-compute init host, the instance with
task_states.POWERING_OFF would been stopped, but the method decorators
@wrap_instance_event would try to query the InstanceAction record by
request_id in admin context and instance_uuid, because the request_id
was generated when init host, it was different with the stopping action
request_id, the InstanceAction record couldn't been found, exception was
raised, the instance can't been stopped.

unpausing, rebooting, starting instance in init-host have same issue.


2015-03-23 16:17:49.394 ERROR nova.compute.manager [req-7f5d1a87-475a-4403-88eb-01843f744125 None None] [instance: b9aadd79-e8f4-486f-8980-114b54a87d4d] Failed to stop instance
 Traceback (most recent call last):
   File "/opt/stack/nova/nova/compute/manager.py", line 1050, in _init_instance
     self.stop_instance(context, instance)
   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
     payload)
   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
     return f(self, context, *args, **kw)
   File "/opt/stack/nova/nova/compute/manager.py", line 324, in decorated_function
     LOG.warning(msg, e, instance_uuid=instance_uuid)
   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/opt/stack/nova/nova/compute/manager.py", line 295, in decorated_function
     return function(self, context, *args, **kwargs)
   File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
     with compute_utils.EventReporter(context, event_name, instance_uuid):
   File "/opt/stack/nova/nova/compute/utils.py", line 477, in __enter__
     self.context, uuid, self.event_name, want_result=False)
   File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
     args, kwargs)
   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 334, in object_class_action
     objver=objver, args=args, kwargs=kwargs)
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
     retry=self.retry)
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
     timeout=timeout, retry=retry)
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 417, in send
     retry=retry)
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 408, in _send
     raise result
 InstanceActionNotFound_Remote: Action for request_id req-5e38756f-d29f-4865-af22-68a44ccd83cb on instance b9aadd79-e8f4-486f-8980-114b54a87d4d not found
 Traceback (most recent call last):

   File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
     return getattr(target, method)(*args, **kwargs)

   File "/opt/stack/nova/nova/objects/base.py", line 163, in wrapper
     result = fn(cls, context, *args, **kwargs)

   File "/opt/stack/nova/nova/objects/instance_action.py", line 170, in event_start
     db_event = db.action_event_start(context, values)

   File "/opt/stack/nova/nova/db/api.py", line 1850, in action_event_start
     return IMPL.action_event_start(context, values)

   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 5697, in action_event_start
     instance_uuid=values['instance_uuid'])

 InstanceActionNotFound: Action for request_id req-5e38756f-
d29f-4865-af22-68a44ccd83cb on instance
b9aadd79-e8f4-486f-8980-114b54a87d4d not found

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1435198

Title:
  Failed to stop intance when nova-compute init host

Status in OpenStack Compute (Nova):
  New

Bug description:
  The admin context is used when nova-compute init host, the instance
  with task_states.POWERING_OFF would been stopped, but the method
  decorators @wrap_instance_event would try to query the InstanceAction
  record by request_id in admin context and instance_uuid, because the
  request_id was generated when init host, it was different with the
  stopping action request_id, the InstanceAction record couldn't been
  found, exception was raised, the instance can't been stopped.

  unpausing, rebooting, starting instance in init-host have same issue.

  
  2015-03-23 16:17:49.394 ERROR nova.compute.manager [req-7f5d1a87-475a-4403-88eb-01843f744125 None None] [instance: b9aadd79-e8f4-486f-8980-114b54a87d4d] Failed to stop instance
   Traceback (most recent call last):
     File "/opt/stack/nova/nova/compute/manager.py", line 1050, in _init_instance
       self.stop_instance(context, instance)
     File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
       payload)
     File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
       return f(self, context, *args, **kw)
     File "/opt/stack/nova/nova/compute/manager.py", line 324, in decorated_function
       LOG.warning(msg, e, instance_uuid=instance_uuid)
     File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
       six.reraise(self.type_, self.value, self.tb)
     File "/opt/stack/nova/nova/compute/manager.py", line 295, in decorated_function
       return function(self, context, *args, **kwargs)
     File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
       with compute_utils.EventReporter(context, event_name, instance_uuid):
     File "/opt/stack/nova/nova/compute/utils.py", line 477, in __enter__
       self.context, uuid, self.event_name, want_result=False)
     File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
       args, kwargs)
     File "/opt/stack/nova/nova/conductor/rpcapi.py", line 334, in object_class_action
       objver=objver, args=args, kwargs=kwargs)
     File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
       retry=self.retry)
     File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
       timeout=timeout, retry=retry)
     File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 417, in send
       retry=retry)
     File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 408, in _send
       raise result
   InstanceActionNotFound_Remote: Action for request_id req-5e38756f-d29f-4865-af22-68a44ccd83cb on instance b9aadd79-e8f4-486f-8980-114b54a87d4d not found
   Traceback (most recent call last):

     File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
       return getattr(target, method)(*args, **kwargs)

     File "/opt/stack/nova/nova/objects/base.py", line 163, in wrapper
       result = fn(cls, context, *args, **kwargs)

     File "/opt/stack/nova/nova/objects/instance_action.py", line 170, in event_start
       db_event = db.action_event_start(context, values)

     File "/opt/stack/nova/nova/db/api.py", line 1850, in action_event_start
       return IMPL.action_event_start(context, values)

     File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 5697, in action_event_start
       instance_uuid=values['instance_uuid'])

   InstanceActionNotFound: Action for request_id req-5e38756f-
  d29f-4865-af22-68a44ccd83cb on instance
  b9aadd79-e8f4-486f-8980-114b54a87d4d not found

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1435198/+subscriptions