yahoo-eng-team team mailing list archive

[hifieli <hifieli2@xxxxxxxxx>]

Public bug reported:

Greetings,


We can't seem to be able to evacuate instances from a failed compute node using shared storage. We are using Ceph Ephemeral RBD as the storage medium.


Steps to reproduce:

nova evacuate --on-shared-storage 6e2081ec-2723-43c7-a730-488bb863674c node-24
or
POST  to http://ip-address:port/v2/tenant_id/servers/server_id/action with 
{"evacuate":{"host":"node-24","onSharedStorage":1}}


Here is what shows up in the logs:


180>Jul 10 20:36:48 node-24 nova-nova.compute.manager AUDIT: Rebuilding instance
<179>Jul 10 20:36:48 node-24 nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5554, in _error_out_instance_on_exception
    yield
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
    _("Invalid state of instance files on shared"
InvalidSharedStorage: Invalid state of instance files on shared storage
<179>Jul 10 20:36:49 node-24 nova-oslo.messaging.rpc.dispatcher ERROR: Exception during message handling: Invalid state of instance files on shared storage
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
    payload)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
    pass
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
    function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
    e, sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
    _("Invalid state of instance files on shared"
InvalidSharedStorage: Invalid state of instance files on shared storage

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: ceph evacuate

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1340411

Title:
  Evacuate Fails 'Invalid state of instance files' using Ceph Ephemeral
  RBD

Status in OpenStack Compute (Nova):
  New

Bug description:
  Greetings,

  
  We can't seem to be able to evacuate instances from a failed compute node using shared storage. We are using Ceph Ephemeral RBD as the storage medium.

  
  Steps to reproduce:

  nova evacuate --on-shared-storage 6e2081ec-2723-43c7-a730-488bb863674c node-24
  or
  POST  to http://ip-address:port/v2/tenant_id/servers/server_id/action with 
  {"evacuate":{"host":"node-24","onSharedStorage":1}}

  
  Here is what shows up in the logs:

  
  180>Jul 10 20:36:48 node-24 nova-nova.compute.manager AUDIT: Rebuilding instance
  <179>Jul 10 20:36:48 node-24 nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5554, in _error_out_instance_on_exception
      yield
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
      _("Invalid state of instance files on shared"
  InvalidSharedStorage: Invalid state of instance files on shared storage
  <179>Jul 10 20:36:49 node-24 nova-oslo.messaging.rpc.dispatcher ERROR: Exception during message handling: Invalid state of instance files on shared storage
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
      incoming.message))
    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
      return func(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
      payload)
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
      return f(self, context, *args, **kw)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
      pass
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
      function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
      e, sys.exc_info())
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2434, in rebuild_instance
      _("Invalid state of instance files on shared"
  InvalidSharedStorage: Invalid state of instance files on shared storage

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1340411/+subscriptions