yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1507521] [NEW] Nova Resize is failing for shared storage between Compute node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: palbhan <satyabhan.pal@xxxxxxxxxxxxxx>
Date: Mon, 19 Oct 2015 10:11:50 -0000
Reply-to: Bug 1507521 <1507521@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Nova Version: 2.22.0

I have share nfs storage mounting the /var/lib/nova between two compute
nodes. When I tried to re-sizing the instance using nova resize command,
it is failing and below is the output of log

2015-10-19 05:13:15.582 14325 ERROR oslo_messaging.rpc.dispatcher [req-5cb16661-74ec-4faf-93cd-044e597cc9de d4209dcd86b84fc584f8b3b72bee0c64 da6c9fa9be0046dda47e9bd6caf3908a - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6748, in resize_instance
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 269, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4012, in resize_instance
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6382, in _error_out_instance_on_exception
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher

When I checked the code, I found in /usr/lib/python2.7/dist-
packages/nova/virt/libvirt/driver.py file we checked the share storage
is true or not before migrating.

 def _is_storage_shared_with(self, dest, inst_base):
        # NOTE (rmk): There are two methods of determining whether we are
        #             on the same filesystem: the source and dest IP are the
        #             same, or we create a file on the dest system via SSH
        #             and check whether the source system can also see it.
        shared_storage = (dest == self.get_host_ip_addr())
        if not shared_storage:
            tmp_file = uuid.uuid4().hex + '.tmp'
            tmp_path = os.path.join(inst_base, tmp_file)
            LOG.debug("Temp path is")
            LOG.debug(tmp_path)
            try:
                utils.execute('ssh', dest, 'touch', tmp_path)
                if os.path.exists(tmp_path):
                    shared_storage = True
                    os.unlink(tmp_path)
                else:
                    utils.execute('ssh', dest, 'rm', tmp_path)
            except Exception:
                pass
        return shared_storage

But in case of share storage between compute node  “shared storage =
(dest == self.get_host_ip_addr())” always return false and also ssh
mkdir  return false and Nova resize will fail.I think checking the share
storage using " shared_storage = (dest == self.get_host_ip_addr())" is
not the right way. When I always return true from
_is_storage_shared_with function, the Nova resize works fine.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1507521

Title:
  Nova Resize is failing for shared storage between Compute node

Status in OpenStack Compute (nova):
  New

Bug description:
  Nova Version: 2.22.0

  I have share nfs storage mounting the /var/lib/nova between two
  compute nodes. When I tried to re-sizing the instance using nova
  resize command, it is failing and below is the output of log

  2015-10-19 05:13:15.582 14325 ERROR oslo_messaging.rpc.dispatcher [req-5cb16661-74ec-4faf-93cd-044e597cc9de d4209dcd86b84fc584f8b3b72bee0c64 da6c9fa9be0046dda47e9bd6caf3908a - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
  Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
  Exit code: 255
  Stdout: u''
  Stderr: u'Host key verification failed.\r\n'
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6748, in resize_instance
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 269, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4012, in resize_instance
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6382, in _error_out_instance_on_exception
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher

  When I checked the code, I found in /usr/lib/python2.7/dist-
  packages/nova/virt/libvirt/driver.py file we checked the share storage
  is true or not before migrating.

   def _is_storage_shared_with(self, dest, inst_base):
          # NOTE (rmk): There are two methods of determining whether we are
          #             on the same filesystem: the source and dest IP are the
          #             same, or we create a file on the dest system via SSH
          #             and check whether the source system can also see it.
          shared_storage = (dest == self.get_host_ip_addr())
          if not shared_storage:
              tmp_file = uuid.uuid4().hex + '.tmp'
              tmp_path = os.path.join(inst_base, tmp_file)
              LOG.debug("Temp path is")
              LOG.debug(tmp_path)
              try:
                  utils.execute('ssh', dest, 'touch', tmp_path)
                  if os.path.exists(tmp_path):
                      shared_storage = True
                      os.unlink(tmp_path)
                  else:
                      utils.execute('ssh', dest, 'rm', tmp_path)
              except Exception:
                  pass
          return shared_storage

  But in case of share storage between compute node  “shared storage =
  (dest == self.get_host_ip_addr())” always return false and also ssh
  mkdir  return false and Nova resize will fail.I think checking the
  share storage using " shared_storage = (dest ==
  self.get_host_ip_addr())" is not the right way. When I always return
  true from _is_storage_shared_with function, the Nova resize works
  fine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1507521/+subscriptions
Follow ups

[Bug 1507521] Re: Nova Resize is failing for shared storage between Compute node
From: Launchpad Bug Tracker, 2016-04-06
[Bug 1507521] Re: Nova Resize is failing for shared storage between Compute node
From: palbhan, 2015-10-21
[Bug 1507521] Re: Nova Resize is failing for shared storage between Compute node
From: Markus Zoeller (markus_z), 2015-10-20