← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76860

[Bug 1813216] Re: legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since Jan 21

  Thread Previous • Date Previous • • Thread Next 
Here is a stable/queens job run from Jan 2 and nova.conf is being used
in the nova-compute service as well:

http://logs.openstack.org/10/627010/2/check/legacy-grenade-dsvm-neutron-
multinode-live-migration/281c4a4/logs/

And in that job the only thing in nova-cpu.conf is this:

[key_manager]
fixed_key = c7e7e2d754d569f54c685e24c5bb73d46eef4aa8806e7457bbf9ebba53279a23a5c73942

So it definitely seems like https://review.openstack.org/#/c/631811/ or
https://review.openstack.org/#/c/609755/ messed something up here.

Also, when I diff the nova-cpu.conf and nova.conf from a failing job,
this is in nova.conf:

[libvirt]
live_migration_uri = qemu+ssh://stack@%s/system
cpu_mode = none
virt_type = qemu
rbd_user = cinder
rbd_secret_uuid = ff43d118-c258-490f-9142-a4b9a7663842
inject_key = false
inject_partition = -2
disk_cachemodes = network=writeback
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf

But not nova-cpu.conf, and that's the main issue - we're running with
ceph but nova-compute is using a config file that's not configured for
ceph.

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Changed in: nova/queens
       Status: New => Triaged

** Changed in: nova/rocky
       Status: New => Triaged

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/rocky
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1813216

Title:
  legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is
  not on shared storage: Shared storage live-migration requires either
  shared storage or boot-from-volume with no local disks." since Jan 21

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) queens series:
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Triaged

Bug description:
  Seen here:

  http://logs.openstack.org/04/632904/2/check/legacy-grenade-dsvm-
  neutron-multinode-live-
  migration/8f886af/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Jan_24_20_30_16_310783

  Jan 24 20:30:16.310783 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server [None req-86fed6ff-bca5-4365-a11e-9fa2bf88635c tempest-LiveMigrationTest-1327523144 tempest-LiveMigrationTest-1327523144] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Jan 24 20:30:16.311129 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jan 24 20:30:16.311370 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  Jan 24 20:30:16.311626 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jan 24 20:30:16.311888 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  Jan 24 20:30:16.312172 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jan 24 20:30:16.312414 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  Jan 24 20:30:16.312731 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jan 24 20:30:16.313186 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 79, in wrapped
  Jan 24 20:30:16.313500 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  Jan 24 20:30:16.313752 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 24 20:30:16.314065 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 24 20:30:16.314300 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 24 20:30:16.314573 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 24 20:30:16.314791 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 69, in wrapped
  Jan 24 20:30:16.315031 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jan 24 20:30:16.315243 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 1141, in decorated_function
  Jan 24 20:30:16.315481 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 24 20:30:16.315684 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 216, in decorated_function
  Jan 24 20:30:16.315898 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jan 24 20:30:16.316131 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 24 20:30:16.316433 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jan 24 20:30:16.316706 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 24 20:30:16.316913 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jan 24 20:30:16.317109 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 204, in decorated_function
  Jan 24 20:30:16.317323 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jan 24 20:30:16.317515 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 6115, in check_can_live_migrate_source
  Jan 24 20:30:16.317749 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     block_device_info)
  Jan 24 20:30:16.318009 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6753, in check_can_live_migrate_source
  Jan 24 20:30:16.318235 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)
  Jan 24 20:30:16.318459 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server InvalidSharedStorage: ubuntu-xenial-ovh-bhs1-0002107027 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Jan 24 20:30:16.318797 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[2857]: ERROR oslo_messaging.rpc.server 

  Looks like this started around Jan 21:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Shared%20storage
  %20live-migration%20requires%20either%20shared%20storage%20or%20boot-
  from-
  volume%20with%20no%20local%20disks.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

  I don't see anything in the nova commit history around that time.

  I wonder if https://review.openstack.org/#/c/631811/ could be related,
  but I'm not sure how it would be, and we should see weird DB errors in
  the logs if that were the case (and all of the tests should fail).

  I am seeing MessagingTimeouts in the nova-compute logs trying to hit
  conductor which is weird.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1813216/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next