yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1720191] Re: test_live_block_migration fails in gate-grenade-dsvm-neutron-multinode-live-migration-nv with "Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since ~8/18/2017

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1720191@xxxxxxxxxxxxxxxxxx>
Date: Mon, 30 Oct 2017 21:26:19 -0000
Reply-to: Bug 1720191 <1720191@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/508271
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e3309672f399be1bc3558e260104a2db2595970
Submitter: Zuul
Branch:    master

commit 3e3309672f399be1bc3558e260104a2db2595970
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Thu Sep 28 14:52:56 2017 -0400

    Fix live migration grenade ceph setup
    
    Grenade runs in singleconductor mode for queens
    as of change:
    
      If4c82ca12fe7b8b1ca7cfd8181d24dbd8dad3baa
    
    However, the nova configuration during ceph setup
    was using NOVA_CPU_CONF which is /etc/nova/nova-cpu.conf,
    which is not what we want when configuring nova.conf
    for the compute service in singleconductor mode.
    
    Devstack has similar logic for stuff like this, so
    we just have to handle it here since we're in a special
    snowflake.
    
    The stable/queens systemd stuff is all removed too since
    we run with systemd on the old pike side and can restart
    services properly with systemd on the new queens side
    during grenade live migration CI job runs.
    
    Change-Id: Iccb8eb55a5cc2a3d08e7fd6e31c89b3b5f8d0c70
    Closes-Bug: #1720191


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1720191

Title:
  test_live_block_migration fails in gate-grenade-dsvm-neutron-
  multinode-live-migration-nv with "Shared storage live-migration
  requires either shared storage or boot-from-volume with no local
  disks." since ~8/18/2017

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The gate-grenade-dsvm-neutron-multinode-live-migration-nv job has been
  failing at about 100% since August 18:

  http://graphite.openstack.org/render/?from=-2160hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%202160%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job
  .gate-grenade-dsvm-neutron-multinode-live-migration-
  nv.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job
  .gate-grenade-dsvm-neutron-multinode-live-migration-
  nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-grenade-dsvm-
  neutron-multinode-live-migration-nv%20%28check%29%27),%27ff0000%27),1)

  With this failure:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958

  Sep 26 14:28:11.637958 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server [None req-b8c949a1-606a-48dc-88df-f848ac421d75 tempest-LiveMigrationRemoteConsolesV26Test-1985366765 tempest-LiveMigrationRemoteConsolesV26Test-1985366765] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-2-node-rax-ord-11140716-924370 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.
  Sep 26 14:28:11.638227 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Sep 26 14:28:11.638476 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
  Sep 26 14:28:11.638699 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Sep 26 14:28:11.638980 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  Sep 26 14:28:11.639207 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Sep 26 14:28:11.639413 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  Sep 26 14:28:11.639617 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Sep 26 14:28:11.639823 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 76, in wrapped
  Sep 26 14:28:11.639981 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Sep 26 14:28:11.640148 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Sep 26 14:28:11.640311 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Sep 26 14:28:11.640449 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Sep 26 14:28:11.640609 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Sep 26 14:28:11.640863 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/exception_wrapper.py", line 67, in wrapped
  Sep 26 14:28:11.641005 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Sep 26 14:28:11.641157 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/utils.py", line 874, in decorated_function
  Sep 26 14:28:11.641301 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Sep 26 14:28:11.641460 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 217, in decorated_function
  Sep 26 14:28:11.641601 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Sep 26 14:28:11.641731 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Sep 26 14:28:11.641848 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Sep 26 14:28:11.641982 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Sep 26 14:28:11.642132 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Sep 26 14:28:11.642291 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 205, in decorated_function
  Sep 26 14:28:11.642474 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Sep 26 14:28:11.642659 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/compute/manager.py", line 5380, in check_can_live_migrate_source
  Sep 26 14:28:11.642827 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     block_device_info)
  Sep 26 14:28:11.643055 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6001, in check_can_live_migrate_source
  Sep 26 14:28:11.643637 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server     raise exception.InvalidSharedStorage(reason=reason, path=source)

  
  Looking at the tests that fail and pass:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/console.html#_2017-09-26_14_28_03_894125

  Anything using the block_migration=auto flag with the 2.25
  microversion is successful, so all microversions before that are
  failing when we have a mixed compute (one pike, one queens)
  environment.

  In this failure case, it's the n-1 Pike compute:

  http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-
  neutron-multinode-live-migration-
  nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz

  Sep 26 14:09:42.382331 ubuntu-xenial-2-node-rax-ord-11140716-924370
  nova-compute[23418]: INFO nova.service [-] Starting compute node
  (version 16.0.1)

  So we might be sending something from the queens compute that the pike
  compute doesn't understand.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1720191/+subscriptions
References

[Bug 1720191] [NEW] test_live_block_migration fails in gate-grenade-dsvm-neutron-multinode-live-migration-nv with "Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since ~8/18/2017
From: Matt Riedemann, 2017-09-28