yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #07692
[Bug 1263636] [NEW] no correct message prompt when migrate instance without enough cpu by Vcenter dirver
Public bug reported:
1. boot two instances with flavor 13, then there is only 5 vCPU available.
[root@10-1-0-71 nova]# nova flavor-list
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 13 | migrate | 512 | 5 | 0 | | 20 | 1.0 | True |
| 14 | migrate_ram | 51200 | 5 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
2. Then migrate test_1 to same host with command "nova migrate test_1", but instance test_1 became to error status finally
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | RESIZE | resize_migrating | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | ERROR | None | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
3. nova show the instance, see following message:
| fault | {u'message': u'Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk', u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance |
| | block_device_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off |
| | dest, flavor) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off |
| | self._session._wait_for_task(instance[\'uuid\'], vm_clone_task) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task |
| | ret_val = done.wait() |
| | File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait |
| | return hubs.get_hub().switch() |
| | File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch |
| | return self.greenlet.switch() |
| | ', u'created': u'2013-12-23T07:17:47Z'}
4. error in log:
2013-12-23 01:17:46.717 8994 ERROR nova.compute.manager [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NV-D132FDD Setting instance vm_state to ERROR
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] Traceback (most recent call last):
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5063, in _error_out_instance_on_exception
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] yield
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] block_device_info)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] dest, flavor)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] ret_val = done.wait()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return hubs.get_hub().switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return self.greenlet.switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d]
2013-12-23 01:17:47.962 8994 ERROR nova.openstack.common.rpc.amqp [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] Exception during message handling
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp **args)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp payload)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 255, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp pass
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 241, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 306, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 283, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp dest, flavor)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp
2013-12-23 01:17:51.727 8994 WARNING nova.virt.vmwareapi.driver [-] NV-EF000C4 Task [CloneVM_Task] (returnval){
value = "task-8706"
_type = "Task"
} status: error Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:51.728 8994 WARNING nova.virt.vmwareapi.driver [-] NV-7575A72 In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2013-12-23 01:17:51.728 8994 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 935, in _poll_task
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall done.send_exception(excep)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 208, in send_exception
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall return self.send(None, args)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 150, in send
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall AssertionError: Trying to re-send() an already-triggered event.
** Affects: nova
Importance: Undecided
Status: New
** Tags: vmware
** Description changed:
1. boot two instances with flavor 13, then there is only 5 vCPU available.
[root@10-1-0-71 nova]# nova flavor-list
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 13 | migrate | 512 | 5 | 0 | | 20 | 1.0 | True |
| 14 | migrate_ram | 51200 | 5 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
2. Then migrate test_1 to same host with command "nova migrate test_1", but instance test_1 became to error status finally
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | RESIZE | resize_migrating | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | ERROR | None | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
3. nova show the instance, see following message:
| fault | {u'message': u'Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk', u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance |
| | block_device_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off |
| | dest, flavor) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off |
| | self._session._wait_for_task(instance[\'uuid\'], vm_clone_task) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task |
| | ret_val = done.wait() |
| | File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait |
| | return hubs.get_hub().switch() |
| | File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch |
| | return self.greenlet.switch() |
| | ', u'created': u'2013-12-23T07:17:47Z'}
-
4. error in log:
2013-12-23 01:17:46.717 8994 ERROR nova.compute.manager [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NV-D132FDD Setting instance vm_state to ERROR
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] Traceback (most recent call last):
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5063, in _error_out_instance_on_exception
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] yield
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] block_device_info)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] dest, flavor)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] ret_val = done.wait()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return hubs.get_hub().switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return self.greenlet.switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d]
2013-12-23 01:17:47.962 8994 ERROR nova.openstack.common.rpc.amqp [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] Exception during message handling
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp **args)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp payload)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 255, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp pass
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 241, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 306, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 283, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp dest, flavor)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp
2013-12-23 01:17:51.727 8994 WARNING nova.virt.vmwareapi.driver [-] NV-EF000C4 Task [CloneVM_Task] (returnval){
- value = "task-8706"
- _type = "Task"
- } status: error Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
+ value = "task-8706"
+ _type = "Task"
+ } status: error Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:51.728 8994 WARNING nova.virt.vmwareapi.driver [-] NV-7575A72 In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2013-12-23 01:17:51.728 8994 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 935, in _poll_task
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall done.send_exception(excep)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 208, in send_exception
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall return self.send(None, args)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 150, in send
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall AssertionError: Trying to re-send() an already-triggered event.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1263636
Title:
no correct message prompt when migrate instance without enough cpu by
Vcenter dirver
Status in OpenStack Compute (Nova):
New
Bug description:
1. boot two instances with flavor 13, then there is only 5 vCPU available.
[root@10-1-0-71 nova]# nova flavor-list
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 13 | migrate | 512 | 5 | 0 | | 20 | 1.0 | True |
| 14 | migrate_ram | 51200 | 5 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-------------+------------+------------+-----------+------+------------+-------------+-----------+
2. Then migrate test_1 to same host with command "nova migrate test_1", but instance test_1 became to error status finally
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | RESIZE | resize_migrating | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------------+-------------+-------------------+
[root@10-1-0-71 nova]# nova list
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
| 2bffa204-b5d2-4ddf-abd5-15a5db32884d | test_1 | ERROR | None | Running | network1=10.0.1.6 |
+--------------------------------------+--------------+--------+------------+-------------+-------------------+
3. nova show the instance, see following message:
| fault | {u'message': u'Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk', u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance |
| | block_device_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off |
| | dest, flavor) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off |
| | self._session._wait_for_task(instance[\'uuid\'], vm_clone_task) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task |
| | ret_val = done.wait() |
| | File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait |
| | return hubs.get_hub().switch() |
| | File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch |
| | return self.greenlet.switch() |
| | ', u'created': u'2013-12-23T07:17:47Z'}
4. error in log:
2013-12-23 01:17:46.717 8994 ERROR nova.compute.manager [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NV-D132FDD Setting instance vm_state to ERROR
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] Traceback (most recent call last):
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5063, in _error_out_instance_on_exception
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] yield
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] block_device_info)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] dest, flavor)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] ret_val = done.wait()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return hubs.get_hub().switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] return self.greenlet.switch()
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d] NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:46.717 8994 TRACE nova.compute.manager [instance: 2bffa204-b5d2-4ddf-abd5-15a5db32884d]
2013-12-23 01:17:47.962 8994 ERROR nova.openstack.common.rpc.amqp [req-da9f6443-1054-4ad5-93f6-04c95a4d5b27 01dd320eb49d4bdfaa08a9ec021a48d4 045dd87f67eb40d1b18f6c9498be3bd9] Exception during message handling
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp **args)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp payload)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 255, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp pass
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 241, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 306, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 283, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 270, in decorated_function
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3117, in resize_instance
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 443, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp dest, flavor)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1169, in migrate_disk_and_power_off
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 906, in _wait_for_task
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp NovaException: Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:47.962 8994 TRACE nova.openstack.common.rpc.amqp
2013-12-23 01:17:51.727 8994 WARNING nova.virt.vmwareapi.driver [-] NV-EF000C4 Task [CloneVM_Task] (returnval){
value = "task-8706"
_type = "Task"
} status: error Error caused by file /vmfs/volumes/52664587-e6a52545-4f65-3440b5e539d0/2bffa204-b5d2-4ddf-abd5-15a5db32884d/2bffa204-b5d2-4ddf-abd5-15a5db32884d.vmdk
2013-12-23 01:17:51.728 8994 WARNING nova.virt.vmwareapi.driver [-] NV-7575A72 In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2013-12-23 01:17:51.728 8994 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 935, in _poll_task
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall done.send_exception(excep)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 208, in send_exception
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall return self.send(None, args)
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 150, in send
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2013-12-23 01:17:51.728 8994 TRACE nova.openstack.common.loopingcall AssertionError: Trying to re-send() an already-triggered event.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1263636/+subscriptions
Follow ups
References
