yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1507521] Re: Nova Resize is failing for shared storage between Compute node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: palbhan <satyabhan.pal@xxxxxxxxxxxxxx>
Date: Wed, 21 Oct 2015 05:40:00 -0000
Reply-to: Bug 1507521 <1507521@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[...] we checked the share storage is true or not before migrating.
So, did you do a resize which results in a cold-migration (=rebuild) of
the instance to another server?
[palabhan] We used Nova resize command to change the instance flavor which cause instance to move another host.

> I think checking the share storage using " shared_storage =
> (dest == self.get_host_ip_addr())" is not the right way.
What makes you think this way? This is a check if the resize/rebuild
of the instance is happening on the very same host.
[palbhan] if we are checking the resize/rebuild
of the instance is happening on the very same host then it is right but  what happened when we have share storage,  shared between source and destination compute node which have different IP address ?

I think as soon as you did that SSH configuration between your compute
nodes the issue will be gone. Because of this I close this bug report
as invalid. If you think I got you wrong and this is a valid bug, please
reopen it by setting the status back to "New".

[palbhan] why do we need to have SSH configuration when both Compute
node are able to access the share storage without any SSH configuration
and Nova resize works fine  even without SSH configuration when I return
True from _is_storage_shared_with function?




** Changed in: nova
       Status: Invalid => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1507521

Title:
  Nova Resize is failing for shared storage between Compute node

Status in OpenStack Compute (nova):
  New

Bug description:
  Nova Version: 2.22.0

  I have share nfs storage mounting the /var/lib/nova between two
  compute nodes. When I tried to re-sizing the instance using nova
  resize command, it is failing and below is the output of log

  2015-10-19 05:13:15.582 14325 ERROR oslo_messaging.rpc.dispatcher [req-5cb16661-74ec-4faf-93cd-044e597cc9de d4209dcd86b84fc584f8b3b72bee0c64 da6c9fa9be0046dda47e9bd6caf3908a - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
  Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
  Exit code: 255
  Stdout: u''
  Stderr: u'Host key verification failed.\r\n'
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6748, in resize_instance
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 377, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 269, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4012, in resize_instance
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6382, in _error_out_instance_on_exception
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Command: ssh 20.20.20.3 mkdir -p /var/lib/nova/instances/744d6341-023f-49cd-9d93-7bae7eb32653
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
  2015-10-19 05:13:15.582 14325 TRACE oslo_messaging.rpc.dispatcher

  When I checked the code, I found in /usr/lib/python2.7/dist-
  packages/nova/virt/libvirt/driver.py file we checked the share storage
  is true or not before migrating.

   def _is_storage_shared_with(self, dest, inst_base):
          # NOTE (rmk): There are two methods of determining whether we are
          #             on the same filesystem: the source and dest IP are the
          #             same, or we create a file on the dest system via SSH
          #             and check whether the source system can also see it.
          shared_storage = (dest == self.get_host_ip_addr())
          if not shared_storage:
              tmp_file = uuid.uuid4().hex + '.tmp'
              tmp_path = os.path.join(inst_base, tmp_file)
              LOG.debug("Temp path is")
              LOG.debug(tmp_path)
              try:
                  utils.execute('ssh', dest, 'touch', tmp_path)
                  if os.path.exists(tmp_path):
                      shared_storage = True
                      os.unlink(tmp_path)
                  else:
                      utils.execute('ssh', dest, 'rm', tmp_path)
              except Exception:
                  pass
          return shared_storage

  But in case of share storage between compute node  “shared storage =
  (dest == self.get_host_ip_addr())” always return false and also ssh
  mkdir  return false and Nova resize will fail.I think checking the
  share storage using " shared_storage = (dest ==
  self.get_host_ip_addr())" is not the right way. When I always return
  true from _is_storage_shared_with function, the Nova resize works
  fine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1507521/+subscriptions
References

[Bug 1507521] [NEW] Nova Resize is failing for shared storage between Compute node
From: palbhan, 2015-10-19