yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1720191] [NEW] test_live_block_migration fails in gate-grenade-dsvm-neutron-multinode-live-migration-nv with "Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since ~8/18/2017

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 28 Sep 2017 17:06:53 -0000
Reply-to: Bug 1720191 <1720191@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

The gate-grenade-dsvm-neutron-multinode-live-migration-nv job has been
failing at about 100% since August 18:

http://graphite.openstack.org/render/?from=-2160hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%202160%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job
.gate-grenade-dsvm-neutron-multinode-live-migration-
nv.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job.gate-
grenade-dsvm-neutron-multinode-live-migration-
nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-grenade-dsvm-neutron-
multinode-live-migration-nv%20%28check%29%27),%27ff0000%27),1)

With this failure:

http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-
multinode-live-migration-
nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958

Sep 26 14:28:11.637958 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server [None req-b8c949a1-606a-48dc-88df-f848ac421d75 tempest-LiveMigrationRemoteConsolesV26Test-1985366765 tempest-LiveMigrationRemoteConsolesV26Test-1985366765] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-2-node-rax-ord-11140716-924370 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
Sep 26 14:28:11.638227 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 26 14:28:11.638476 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
Sep 26 14:28:11.638699 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 26 14:28:11.638980 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
Sep 26 14:28:11.639207 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 26 14:28:11.639413 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
Sep 26 14:28:11.639617 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 26 14:28:11.639823 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 76, in wrapped
Sep 26 14:28:11.639981 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
Sep 26 14:28:11.640148 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Sep 26 14:28:11.640311 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Sep 26 14:28:11.640449 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Sep 26 14:28:11.640609 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Sep 26 14:28:11.640863 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 67, in wrapped
Sep 26 14:28:11.641005 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Sep 26 14:28:11.641157 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 874, in decorated_function
Sep 26 14:28:11.641301 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Sep 26 14:28:11.641460 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 217, in decorated_function
Sep 26 14:28:11.641601 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
Sep 26 14:28:11.641731 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Sep 26 14:28:11.641848 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Sep 26 14:28:11.641982 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Sep 26 14:28:11.642132 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Sep 26 14:28:11.642291 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 205, in decorated_function
Sep 26 14:28:11.642474 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
Sep 26 14:28:11.642659 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 5380, in check_can_live_migrate_source
Sep 26 14:28:11.642827 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     block_device_info)
Sep 26 14:28:11.643055 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6001, in check_can_live_migrate_source
Sep 26 14:28:11.643637 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)


Looking at the tests that fail and pass:

http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-
multinode-live-migration-
nv/ae8875f/console.html#_2017-09-26_14_28_03_894125

Anything using the block_migration=auto flag with the 2.25 microversion
is successful, so all microversions before that are failing when we have
a mixed compute (one pike, one queens) environment.

In this failure case, it's the n-1 Pike compute:

http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-
multinode-live-migration-nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz

Sep 26 14:09:42.382331 ubuntu-xenial-2-node-rax-ord-11140716-924370
nova-compute[23418]: INFO nova.service [-] Starting compute node
(version 16.0.1)

So we might be sending something from the queens compute that the pike
compute doesn't understand.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: grenade live-migration

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1720191

Title:
  test_live_block_migration fails in gate-grenade-dsvm-neutron-
  multinode-live-migration-nv with "Shared storage live-migration
  requires either shared storage or boot-from-volume with no local
  disks." since ~8/18/2017

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  The gate-grenade-dsvm-neutron-multinode-live-migration-nv job has been
  failing at about 100% since August 18:

  http://graphite.openstack.org/render/?from=-2160hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%202160%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job
  .gate-grenade-dsvm-neutron-multinode-live-migration-
  nv.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job
  .gate-grenade-dsvm-neutron-multinode-live-migration-
  nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-grenade-dsvm-
  neutron-multinode-live-migration-nv%20%28check%29%27),%27ff0000%27),1)

  With this failure:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958

  Sep 26 14:28:11.637958 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server [None req-b8c949a1-606a-48dc-88df-f848ac421d75 tempest-LiveMigrationRemoteConsolesV26Test-1985366765 tempest-LiveMigrationRemoteConsolesV26Test-1985366765] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-2-node-rax-ord-11140716-924370 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Sep 26 14:28:11.638227 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Sep 26 14:28:11.638476 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
  Sep 26 14:28:11.638699 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Sep 26 14:28:11.638980 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  Sep 26 14:28:11.639207 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Sep 26 14:28:11.639413 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  Sep 26 14:28:11.639617 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Sep 26 14:28:11.639823 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 76, in wrapped
  Sep 26 14:28:11.639981 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Sep 26 14:28:11.640148 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Sep 26 14:28:11.640311 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Sep 26 14:28:11.640449 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Sep 26 14:28:11.640609 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Sep 26 14:28:11.640863 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 67, in wrapped
  Sep 26 14:28:11.641005 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Sep 26 14:28:11.641157 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 874, in decorated_function
  Sep 26 14:28:11.641301 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Sep 26 14:28:11.641460 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 217, in decorated_function
  Sep 26 14:28:11.641601 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Sep 26 14:28:11.641731 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Sep 26 14:28:11.641848 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Sep 26 14:28:11.641982 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Sep 26 14:28:11.642132 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Sep 26 14:28:11.642291 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 205, in decorated_function
  Sep 26 14:28:11.642474 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Sep 26 14:28:11.642659 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 5380, in check_can_live_migrate_source
  Sep 26 14:28:11.642827 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     block_device_info)
  Sep 26 14:28:11.643055 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6001, in check_can_live_migrate_source
  Sep 26 14:28:11.643637 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)

  
  Looking at the tests that fail and pass:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/console.html#_2017-09-26_14_28_03_894125

  Anything using the block_migration=auto flag with the 2.25
  microversion is successful, so all microversions before that are
  failing when we have a mixed compute (one pike, one queens)
  environment.

  In this failure case, it's the n-1 Pike compute:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz

  Sep 26 14:09:42.382331 ubuntu-xenial-2-node-rax-ord-11140716-924370
  nova-compute[23418]: INFO nova.service [-] Starting compute node
  (version 16.0.1)

  So we might be sending something from the queens compute that the pike
  compute doesn't understand.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1720191/+subscriptions
Follow ups

[Bug 1720191] Re: test_live_block_migration fails in gate-grenade-dsvm-neutron-multinode-live-migration-nv with "Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since ~8/18/2017
From: OpenStack Infra, 2017-10-30