yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1687511] Re: live migration job blowing up with "InvalidSharedStorage: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk."

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1687511@xxxxxxxxxxxxxxxxxx>
Date: Tue, 02 May 2017 16:07:03 -0000
Reply-to: Bug 1687511 <1687511@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/461803
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e59eee6db5876fa86954f1cab10c667c80f35763
Submitter: Jenkins
Branch:    master

commit e59eee6db5876fa86954f1cab10c667c80f35763
Author: Sean Dague <sean@xxxxxxxxx>
Date:   Tue May 2 10:16:51 2017 -0400

    Use systemctl to restart services
    
    The restart code for the live migration configure on the fly tests was
    really really fragile clippings of old bits of devstack code that
    barely worked (and assumed a very specific run environment).
    
    Given that we're going to get rid of everything other than systemd in
    devstack, just jump straight to using systemd primitives here. This
    dramatically simplifies the code, and will ensure things actually
    restart correctly.
    
    Closes-Bug: #1687511
    
    Change-Id: I914430d68f64d29932c9409d055b15e4cb384ec4


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1687511

Title:
  live migration job blowing up with "InvalidSharedStorage: ubuntu-
  xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared
  storage: Live migration can not be used without shared storage except
  a booted from volume VM which does not have a local disk."

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seen here:

  http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-
  multinode-live-migration-ubuntu-
  xenial/a3f4426/logs/screen-n-cpu.txt.gz?level=TRACE#_May_01_22_15_43_039605

  May 01 22:15:43.039605 ubuntu-xenial-2-node-osic-cloud1-s3500-8668392 nova-compute[22173]: ERROR oslo_messaging.rpc.server [req-233741a8-bb50-4bf6-a9ec-6c67e6360d39 tempest-LiveBlockMigrationTestJSON-978755190 tempest-LiveBlockMigrationTestJSON-978755190] Exception during message handling
                                                                                             ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                                             ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                                             ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                                             ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                                             ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                             ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                             ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                                             ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                             ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                             ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                                             ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5178, in check_can_live_migrate_destination
                                                                                             ERROR oslo_messaging.rpc.server     disk_over_commit)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5194, in _do_check_can_live_migrate_destination
                                                                                             ERROR oslo_messaging.rpc.server     dest_check_data)
                                                                                             ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 478, in check_can_live_migrate_source
                                                                                             ERROR oslo_messaging.rpc.server     dest_check_data=dest_check_data)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
                                                                                             ERROR oslo_messaging.rpc.server     retry=self.retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 98, in _send
                                                                                             ERROR oslo_messaging.rpc.server     timeout=timeout, retry=retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
                                                                                             ERROR oslo_messaging.rpc.server     retry=retry)
                                                                                             ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
                                                                                             ERROR oslo_messaging.rpc.server     raise result
                                                                                             ERROR oslo_messaging.rpc.server InvalidSharedStorage_Remote: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.

  
  Looks like this just started today (5/1):

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Live%20migration%20can%20not%20be%20used%20without%20shared%20storage%20except%20a%20booted%20from%20volume%20VM%20which%20does%20not%20have%20a%20local%20disk%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1687511/+subscriptions
References

[Bug 1687511] [NEW] live migration job blowing up with "InvalidSharedStorage: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk."
From: Matt Riedemann, 2017-05-02