yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1493512] Re: n-cpu misreports shared storage when attempting to block migrate

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1493512@xxxxxxxxxxxxxxxxxx>
Date: Tue, 02 Feb 2016 04:17:21 -0000
Reply-to: Bug 1493512 <1493512@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1493512

Title:
  n-cpu misreports shared storage when attempting to block migrate

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Nova is not allowing any block-migrate transactions to occur due to
  false positives when testing for shared storage. This is occurring
  with even a simple two node setup.

  
  ------Original Entry------

  If using an environment where there are both nodes that share the nova
  instances directory and nodes that do NOT share the directory, any
  attempt to block-migrate to non-shared nodes from the controller node
  will fail.

  Example:
  1. Create an environment with one controller node and two cpu nodes.
  2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
  3. Add this new share to the fstab of just one of the CPU nodes.
  4. Mount the new share after stacking the appropriate CPU node.
  5. Stack the unshared CPU node.

  The next step applied to my scenario, but may not be necessary.
  6. Create a bootable volume in Cinder.
  7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
  8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)

  In the past it was possible to block-migrate to and from the unshared
  node, then migrate without block between the shared nodes. Now in
  Liberty (master) the following errors appears:

  2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
      executor_callback))

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
      executor_callback)

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
      result = func(ctxt, **new_args)

    File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
      payload)

    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
      return f(self, context, *args, **kw)

    File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
      kwargs['instance'], e, sys.exc_info())

    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
      block_device_info)

    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
      raise exception.InvalidLocalStorage(reason=reason, path=source)

  InvalidLocalStorage: cld5b12 is not on local storage: Block migration
  can not be used with shared storage.

  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     dest_check_data)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     dest_check_data=dest_check_data)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     retry=self.retry)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     retry=retry)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     raise result
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     block_device_info)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher     raise exception.InvalidLocalStorage(reason=reason, path=source)
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
  2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
      executor_callback))

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
      executor_callback)

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
      result = func(ctxt, **new_args)

    File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
      payload)

    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
      return f(self, context, *args, **kw)

    File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
      kwargs['instance'], e, sys.exc_info())

    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
      block_device_info)

    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
      raise exception.InvalidLocalStorage(reason=reason, path=source)

  InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
   to caller
  2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n    payload)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n    kwargs[\'instance\'], e, sys.exc_info())\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n    dest_check_data)\n', '  File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n    dest_check_data=dest_check_data)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n    retry=self.retry)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n    timeout=timeout, retry=retry)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n    retry=retry)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n    raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n\n  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n\n  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n\n  File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n    payload)\n\n  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n    return f(self, context, *args, **kw)\n\n  File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n    return function(self, context, *args, **kwargs)\n\n  File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n    kwargs[\'instance\'], e, sys.exc_info())\n\n  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n    return function(self, context, *args, **kwargs)\n\n  File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n    block_device_info)\n\n  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n    raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']

  This check appears to only check the source, and is not actually
  testing to see if the storage is shared between the two nodes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1493512/+subscriptions
References

[Bug 1493512] [NEW] n-cpu misreports shared storage when attempting to block migrate
From: Sean Severson, 2015-09-08