yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1259389] Re: resize to different host failed by vcenter driver

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tracy Jones <tjones@xxxxxxxxxx>
Date: Fri, 10 Jan 2014 21:05:49 -0000
Reply-to: Bug 1259389 <1259389@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** No longer affects: openstack-vmwareapi-team

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1259389

Title:
  resize to different host failed by vcenter driver

Status in OpenStack Compute (Nova):
  New

Bug description:
  1. I have a environment managed two cluster.  Make sure there is two compute-service for vmware exists, in this case, the vmware compute service is: vmware_nova_cluster01 and vmware_nova_cluster02
  [root@10-1-0-71 nova]# nova-manage service list
  Binary           Host                                 Zone             Status     State Updated_At
  nova-cells       10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:48
  nova-conductor   10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:50
  nova-scheduler   10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:42
  nova-cert        10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:43
  nova-console     10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:51
  nova-consoleauth 10-1-0-71                            internal         enabled    :-)   2013-12-06 06:22:42
  nova-compute     10-1-0-71                            nova             enabled    :-)   2013-12-06 06:22:50
  nova-compute     vmware_nova                          nova             enabled    XXX   2013-11-21 10:03:44
  nova-compute     vmware_nova_cluster01                nova             enabled    :-)   2013-12-06 06:22:43
  nova-compute     vmware_nova_cluster02                nova             enabled    :-)   2013-12-06 06:22:45

  2. deploy a vm to cluster02:
  nova boot --image 4432ca56-8be0-4794-9c22-52a8c7a2bf55 --flavor 1 --availability-zone nova:vmware_nova_cluster02 test
  [root@10-1-0-71 nova]# nova list
  +--------------------------------------+------+--------+------------+-------------+-------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks          |
  +--------------------------------------+------+--------+------------+-------------+-------------------+
  | 127db0b8-97e6-4e66-8550-50b219187134 | test | BUILD  | spawning   | NOSTATE     | network1=10.0.1.2 |
  +--------------------------------------+------+--------+------------+-------------+-------------------+

  3. resize the instance to flavor 2
  POST: http://10.1.0.71:8774/v2/045dd87f67eb40d1b18f6c9498be3bd9/servers/127db0b8-97e6-4e66-8550-50b219187134/action
  {
      "resize" : {
      "flavorRef" :"2"
      }
  }

  4. wait a while, the instance became to verify_resize status, then confirm resize
  [root@10-1-0-71 nova]# nova list
  +--------------------------------------+------+---------------+------------+-------------+-------------------+
  | ID                                   | Name | Status        | Task State | Power State | Networks          |
  +--------------------------------------+------+---------------+------------+-------------+-------------------+
  | 127db0b8-97e6-4e66-8550-50b219187134 | test | VERIFY_RESIZE | None       | Running     | network1=10.0.1.2 |
  +--------------------------------------+------+---------------+------------+-------------+-------------------+
  POST: http://10.1.0.71:8774/v2/045dd87f67eb40d1b18f6c9498be3bd9/servers/127db0b8-97e6-4e66-8550-50b219187134/action
  {
      "confirmResize" : null
  }

  3. Instance become to ERROR
  [root@10-1-0-71 nova]# nova list
  +--------------------------------------+------+--------+------------+-------------+-------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks          |
  +--------------------------------------+------+--------+------------+-------------+-------------------+
  | 127db0b8-97e6-4e66-8550-50b219187134 | test | ERROR  | None       | Running     | network1=10.0.1.2 |
  +--------------------------------------+------+--------+------------+-------------+-------------------+

  4. the error show 
  | fault                                | {u'message': u'NV-3AB798A The resource domain-c17(cluster01) does not exist', u'code': 404, u'created': u'2013-12-06T06:17:58Z'} |

  5. messages in log:
  2013-12-06 00:17:58.892 10072 TRACE nova.compute.manager [instance: 127db0b8-97e6-4e66-8550-50b219187134]
  2013-12-06 00:17:59.035 10072 ERROR nova.openstack.common.rpc.amqp [req-4d2d3892-0b41-494b-b3c8-f94eca801db8 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] Exception during message handling
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     **args)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     payload)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 306, in decorated_function
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 283, in decorated_function
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2775, in confirm_resize
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     do_confirm_resize(context, instance, migration.id)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 248, in inner
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2773, in do_confirm_resize
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     migration=migration)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2799, in _confirm_resize
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     network_info)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 446, in confirm_migration
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     _vmops = self._get_vmops_for_compute_node(instance['node'])
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 548, in _get_vmops_for_compute_node
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     resource = self._get_resource_for_node(nodename)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 540, in _get_resource_for_node
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp     raise exception.NotFound(msg)
  2013-12-06 00:17:59.035 10072 TRACE nova.openstack.common.rpc.amqp NotFound: NV-3AB798A The resource domain-c17(cluster01) does not exist

  6. [root@10-1-0-71 nova]# nova host-list
  +-----------------------+-------------+----------+
  | host_name             | service     | zone     |
  +-----------------------+-------------+----------+
  | 10-1-0-71             | cells       | internal |
  | 10-1-0-71             | conductor   | internal |
  | 10-1-0-71             | scheduler   | internal |
  | 10-1-0-71             | cert        | internal |
  | 10-1-0-71             | console     | internal |
  | 10-1-0-71             | consoleauth | internal |
  | 10-1-0-71             | compute     | nova     |
  | vmware_nova           | compute     | nova     |
  | vmware_nova_cluster01 | compute     | nova     |
  | vmware_nova_cluster02 | compute     | nova     |
  +-----------------------+-------------+----------+
  [root@10-1-0-71 nova]# nova hypervisor-list
  +----+------------------------+
  | ID | Hypervisor hostname    |
  +----+------------------------+
  | 1  | 10-1-0-71              |
  | 7  | domain-c17(cluster01)  |
  | 8  | domain-c382(cluster02) |
  +----+------------------------+

  7. I tried to change nova.conf file to make openstack display same
  host name when use host-list and hypervisor-list,  like:
  domain-c17(cluster01) , however, since there is "(" and ")" in the
  string, seems openstack could not recognize "\(" or "\)", so this
  method failed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1259389/+subscriptions
References

[Bug 1259389] [NEW] resize to different host failed by vcenter driver
From: dingxy, 2013-12-10