yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1687511] [NEW] live migration job blowing up with "InvalidSharedStorage: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk."

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 02 May 2017 01:00:43 -0000
Reply-to: Bug 1687511 <1687511@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-multinode-
live-migration-ubuntu-
xenial/a3f4426/logs/screen-n-cpu.txt.gz?level=TRACE#_May_01_22_15_43_039605

May 01 22:15:43.039605 ubuntu-xenial-2-node-osic-cloud1-s3500-8668392 nova-compute[22173]: ERROR oslo_messaging.rpc.server [req-233741a8-bb50-4bf6-a9ec-6c67e6360d39 tempest-LiveBlockMigrationTestJSON-978755190 tempest-LiveBlockMigrationTestJSON-978755190] Exception during message handling
                                                                                           ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                                           ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                                           ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                                           ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                                           ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                           ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                           ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                                           ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                           ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                           ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5178, in check_can_live_migrate_destination
                                                                                           ERROR oslo_messaging.rpc.server     disk_over_commit)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5194, in _do_check_can_live_migrate_destination
                                                                                           ERROR oslo_messaging.rpc.server     dest_check_data)
                                                                                           ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 478, in check_can_live_migrate_source
                                                                                           ERROR oslo_messaging.rpc.server     dest_check_data=dest_check_data)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
                                                                                           ERROR oslo_messaging.rpc.server     retry=self.retry)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 98, in _send
                                                                                           ERROR oslo_messaging.rpc.server     timeout=timeout, retry=retry)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
                                                                                           ERROR oslo_messaging.rpc.server     retry=retry)
                                                                                           ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
                                                                                           ERROR oslo_messaging.rpc.server     raise result
                                                                                           ERROR oslo_messaging.rpc.server InvalidSharedStorage_Remote: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.


Looks like this just started today (5/1):

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Live%20migration%20can%20not%20be%20used%20without%20shared%20storage%20except%20a%20booted%20from%20volume%20VM%20which%20does%20not%20have%20a%20local%20disk%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

** Affects: nova
     Importance: Undecided
         Status: Confirmed


** Tags: live-migration

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1687511

Title:
  live migration job blowing up with "InvalidSharedStorage: ubuntu-
  xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared
  storage: Live migration can not be used without shared storage except
  a booted from volume VM which does not have a local disk."

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-
  multinode-live-migration-ubuntu-
  xenial/a3f4426/logs/screen-n-cpu.txt.gz?level=TRACE#_May_01_22_15_43_039605

  May 01 22:15:43.039605 ubuntu-xenial-2-node-osic-cloud1-s3500-8668392 nova-compute[22173]: ERROR oslo_messaging.rpc.server [req-233741a8-bb50-4bf6-a9ec-6c67e6360d39 tempest-LiveBlockMigrationTestJSON-978755190 tempest-LiveBlockMigrationTestJSON-978755190] Exception during message handling
                                                                                             ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                                             ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                                             ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                                             ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                                             ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                             ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                             ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                                             ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                             ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                             ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5178, in check_can_live_migrate_destination
                                                                                             ERROR oslo_messaging.rpc.server     disk_over_commit)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5194, in _do_check_can_live_migrate_destination
                                                                                             ERROR oslo_messaging.rpc.server     dest_check_data)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 478, in check_can_live_migrate_source
                                                                                             ERROR oslo_messaging.rpc.server     dest_check_data=dest_check_data)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
                                                                                             ERROR oslo_messaging.rpc.server     retry=self.retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 98, in _send
                                                                                             ERROR oslo_messaging.rpc.server     timeout=timeout, retry=retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
                                                                                             ERROR oslo_messaging.rpc.server     retry=retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
                                                                                             ERROR oslo_messaging.rpc.server     raise result
                                                                                             ERROR oslo_messaging.rpc.server InvalidSharedStorage_Remote: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.

  
  Looks like this just started today (5/1):

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Live%20migration%20can%20not%20be%20used%20without%20shared%20storage%20except%20a%20booted%20from%20volume%20VM%20which%20does%20not%20have%20a%20local%20disk%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1687511/+subscriptions
Follow ups

[Bug 1687511] Re: live migration job blowing up with "InvalidSharedStorage: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk."
From: OpenStack Infra, 2017-05-02