yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1624797] [NEW] Failed to evacuate instances booted from volume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: javeme <javaloveme@xxxxxxxxx>
Date: Sun, 18 Sep 2016 06:08:50 -0000
Reply-to: Bug 1624797 <1624797@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description:
Failed to evacuate instances booted from volume

Steps to reproduce:
1.Create an instance from volume
nova boot --boot-volume <volume_id> --flavor <flavor> --nic <net-id=net-uuid> test

2.Force nova-compute service to down
nova service-force-down <hostname> nova-compute

3.Evacuate server from failed host
nova evacuate test

Result:
instance "test" fails to evacuate:
InvalidSharedStorage: Invalid state of instance files on shared storage

Environment:
version: Liberty
nova image backend: Ceph RBD

Error Log:
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 353, in decorated_function
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 403, in decorated_function
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 381, in decorated_function
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2751, in _do_rebuild_instance_with_claim
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2776, in _do_rebuild_instance
2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

- Description
- ===========
+ Description:
  Failed to evacuate instances booted from volume
  
- 
- Steps to reproduce
- ==================
+ Steps to reproduce:
  1.Create an instance from volume
  nova boot --boot-volume <volume_id> --flavor <flavor> --nic <net-id=net-uuid> test
  
  2.Force nova-compute service to down
  nova service-force-down <hostname> nova-compute
  
  3.Evacuate server from failed host
  nova evacuate test
  
- 
- Result
- ======
+ Result:
  instance "test" fails to evacuate
  
  
- Environment
- ===========
+ Environment:
  version: Liberty
  nova image backend: Ceph RBD
  
  
  Error Log:
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 353, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 403, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 381, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2751, in _do_rebuild_instance_with_claim
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2776, in _do_rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage

** Description changed:

  Description:
  Failed to evacuate instances booted from volume
  
  Steps to reproduce:
  1.Create an instance from volume
  nova boot --boot-volume <volume_id> --flavor <flavor> --nic <net-id=net-uuid> test
  
  2.Force nova-compute service to down
  nova service-force-down <hostname> nova-compute
  
  3.Evacuate server from failed host
  nova evacuate test
  
  Result:
- instance "test" fails to evacuate
- 
+ instance "test" fails to evacuate:
+ InvalidSharedStorage: Invalid state of instance files on shared storage
  
  Environment:
  version: Liberty
  nova image backend: Ceph RBD
- 
  
  Error Log:
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 353, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 403, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 381, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2751, in _do_rebuild_instance_with_claim
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2776, in _do_rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1624797

Title:
  Failed to evacuate instances booted from volume

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:
  Failed to evacuate instances booted from volume

  Steps to reproduce:
  1.Create an instance from volume
  nova boot --boot-volume <volume_id> --flavor <flavor> --nic <net-id=net-uuid> test

  2.Force nova-compute service to down
  nova service-force-down <hostname> nova-compute

  3.Evacuate server from failed host
  nova evacuate test

  Result:
  instance "test" fails to evacuate:
  InvalidSharedStorage: Invalid state of instance files on shared storage

  Environment:
  version: Liberty
  nova image backend: Ceph RBD

  Error Log:
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 353, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 403, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 381, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2751, in _do_rebuild_instance_with_claim
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2776, in _do_rebuild_instance
  2016-08-14 17:37:40.257 930041 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1624797/+subscriptions