← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76726

[Bug 1813216] [NEW] legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since Jan 21

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Seen here:

http://logs.openstack.org/04/632904/2/check/legacy-grenade-dsvm-neutron-
multinode-live-
migration/8f886af/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Jan_24_20_30_16_310783

Jan 24 20:30:16.310783 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server [None req-86fed6ff-bca5-4365-a11e-9fa2bf88635c tempest-LiveMigrationTest-1327523144 tempest-LiveMigrationTest-1327523144] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
Jan 24 20:30:16.311129 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 24 20:30:16.311370 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
Jan 24 20:30:16.311626 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jan 24 20:30:16.311888 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Jan 24 20:30:16.312172 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jan 24 20:30:16.312414 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Jan 24 20:30:16.312731 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jan 24 20:30:16.313186 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 79, in wrapped
Jan 24 20:30:16.313500 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
Jan 24 20:30:16.313752 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 24 20:30:16.314065 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jan 24 20:30:16.314300 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 24 20:30:16.314573 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jan 24 20:30:16.314791 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 69, in wrapped
Jan 24 20:30:16.315031 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Jan 24 20:30:16.315243 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 1141, in decorated_function
Jan 24 20:30:16.315481 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Jan 24 20:30:16.315684 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 216, in decorated_function
Jan 24 20:30:16.315898 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
Jan 24 20:30:16.316131 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 24 20:30:16.316433 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jan 24 20:30:16.316706 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 24 20:30:16.316913 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jan 24 20:30:16.317109 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 204, in decorated_function
Jan 24 20:30:16.317323 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Jan 24 20:30:16.317515 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 6115, in check_can_live_migrate_source
Jan 24 20:30:16.317749 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     block_device_info)
Jan 24 20:30:16.318009 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6753, in check_can_live_migrate_source
Jan 24 20:30:16.318235 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)
Jan 24 20:30:16.318459 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
Jan 24 20:30:16.318797 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server 

Looks like this started around Jan 21:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Shared%20storage
%20live-migration%20requires%20either%20shared%20storage%20or%20boot-
from-
volume%20with%20no%20local%20disks.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

I don't see anything in the nova commit history around that time.

I wonder if https://review.openstack.org/#/c/631811/ could be related,
but I'm not sure how it would be, and we should see weird DB errors in
the logs if that were the case (and all of the tests should fail).

I am seeing MessagingTimeouts in the nova-compute logs trying to hit
conductor which is weird.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1813216

Title:
  legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is
  not on shared storage: Shared storage live-migration requires either
  shared storage or boot-from-volume with no local disks." since Jan 21

Status in OpenStack Compute (nova):
  New

Bug description:
  Seen here:

  http://logs.openstack.org/04/632904/2/check/legacy-grenade-dsvm-
  neutron-multinode-live-
  migration/8f886af/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Jan_24_20_30_16_310783

  Jan 24 20:30:16.310783 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server [None req-86fed6ff-bca5-4365-a11e-9fa2bf88635c tempest-LiveMigrationTest-1327523144 tempest-LiveMigrationTest-1327523144] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Jan 24 20:30:16.311129 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jan 24 20:30:16.311370 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jan 24 20:30:16.311626 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jan 24 20:30:16.311888 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  Jan 24 20:30:16.312172 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jan 24 20:30:16.312414 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  Jan 24 20:30:16.312731 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jan 24 20:30:16.313186 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 79, in wrapped
  Jan 24 20:30:16.313500 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  Jan 24 20:30:16.313752 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 24 20:30:16.314065 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 24 20:30:16.314300 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 24 20:30:16.314573 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 24 20:30:16.314791 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 69, in wrapped
  Jan 24 20:30:16.315031 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jan 24 20:30:16.315243 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 1141, in decorated_function
  Jan 24 20:30:16.315481 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 24 20:30:16.315684 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 216, in decorated_function
  Jan 24 20:30:16.315898 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jan 24 20:30:16.316131 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 24 20:30:16.316433 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 24 20:30:16.316706 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 24 20:30:16.316913 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 24 20:30:16.317109 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 204, in decorated_function
  Jan 24 20:30:16.317323 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 24 20:30:16.317515 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 6115, in check_can_live_migrate_source
  Jan 24 20:30:16.317749 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     block_device_info)
  Jan 24 20:30:16.318009 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6753, in check_can_live_migrate_source
  Jan 24 20:30:16.318235 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)
  Jan 24 20:30:16.318459 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Jan 24 20:30:16.318797 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server 

  Looks like this started around Jan 21:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Shared%20storage
  %20live-migration%20requires%20either%20shared%20storage%20or%20boot-
  from-
  volume%20with%20no%20local%20disks.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

  I don't see anything in the nova commit history around that time.

  I wonder if https://review.openstack.org/#/c/631811/ could be related,
  but I'm not sure how it would be, and we should see weird DB errors in
  the logs if that were the case (and all of the tests should fail).

  I am seeing MessagingTimeouts in the nova-compute logs trying to hit
  conductor which is weird.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1813216/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next