yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1180044] Re: nova failures when vCenter has multiple datacenters

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1180044@xxxxxxxxxxxxxxxxxx>
Date: Thu, 23 Jan 2014 17:56:11 -0000
Reply-to: Bug 1180044 <1180044@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/havana
   Importance: Undecided
       Status: New

** Changed in: nova/havana
       Status: New => In Progress

** Changed in: nova/havana
   Importance: Undecided => High

** Changed in: nova/havana
     Assignee: (unassigned) => Gary Kotton (garyk)

** Tags removed: havana-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1180044

Title:
  nova failures when vCenter has multiple datacenters

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) havana series:
  In Progress
Status in The OpenStack VMwareAPI subTeam:
  Fix Committed

Bug description:
  The method at vmops.py _get_datacenter_ref_and_name does not calculate
  datacenter properly.

      def _get_datacenter_ref_and_name(self):
          """Get the datacenter name and the reference."""
          dc_obj = self._session._call_method(vim_util, "get_objects",
                  "Datacenter", ["name"])
          vm_util._cancel_retrieve_if_necessary(self._session, dc_obj)
          return dc_obj.objects[0].obj, dc_obj.objects[0].propSet[0].val

  This will not be correct on systems with more than one datacenter.

  Stack trace from logs:

  ERROR nova.compute.manager [req-9395fe41-cf04-4434-bd77-663e93de1d4a
  foo bar] [instance: 484a42a2-642e-4594-93fe-4f72ddad361f] Error:
  ['Traceback (most recent call last):\n', '  File
  "/opt/stack/nova/nova/compute/manager.py", line 942, in
  _build_instance\n    set_access_ip=set_access_ip)\n', '  File
  "/opt/stack/nova/nova/compute/manager.py", line 1204, in _spawn\n
  LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n',
  '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
  self.gen.next()\n', '  File "/opt/stack/nova/nova/compute/manager.py",
  line 1200, in _spawn\n    block_device_info)\n', '  File
  "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 176, in spawn\n
  block_device_info)\n', '  File
  "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 208, in spawn\n
  _execute_create_vm()\n', '  File
  "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 204, in
  _execute_create_vm\n
  self._session._wait_for_task(instance[\'uuid\'], vm_create_task)\n', '
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 559, in
  _wait_for_task\n    ret_val = done.wait()\n', '  File
  "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116,
  in wait\n    return hubs.get_hub().switch()\n', '  File
  "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
  187, in switch\n    return self.greenlet.switch()\n', 'NovaException:
  A specified parameter was not correct. \nspec.location.folder\n']

  vCenter error is:
  "A specified parameter was not correct. spec.location.folder"

  Work around:
  use only one datacenter, use only one cluster, turn on DRS

  Additional failures:
  2013-07-18 10:59:12.788 DEBUG nova.virt.vmwareapi.vmware_images [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Got image size of 687865856 for the image cde14862-60b8-4360-a145-06585b06577c get_vmdk_size_and_properties /usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmware_images.py:156
  2013-07-18 10:59:12.963 WARNING nova.virt.vmwareapi.network_util [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [(ManagedObjectReference){
     value = "network-1501"
     _type = "Network"
   }, (ManagedObjectReference){
     value = "network-1458"
     _type = "Network"
   }, (ManagedObjectReference){
     value = "network-2085"
     _type = "Network"
   }, (ManagedObjectReference){
     value = "network-1143"
     _type = "Network"
   }]
  2013-07-18 10:59:13.326 DEBUG nova.virt.vmwareapi.vmops [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Creating VM on the ESX host _execute_create_vm /usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py:207
  2013-07-18 10:59:14.258 3145 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:583
  2013-07-18 10:59:14.259 3145 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is 8ef36d061a9341a09d3a5451df798673 multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:586
  2013-07-18 10:59:14.259 3145 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 680b790574c64a9783fd2138c43f5f6d. _add_unique_id /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:337
  2013-07-18 10:59:18.757 3145 WARNING nova.virt.vmwareapi.driver [-] Task [CreateVM_Task] (returnval){
     value = "task-33558"
     _type = "Task"
   } status: error The input arguments had entities that did not belong to the same datacenter.

  2013-07-18 10:59:18.758 ERROR nova.compute.manager [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Instance failed to spawn
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Traceback (most recent call last):
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1103, in _spawn
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] block_device_info)
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 177, in spawn
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] block_device_info)
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 217, in spawn
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] _execute_create_vm()
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 213, in _execute_create_vm
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] self._session._wait_for_task(instance['uuid'], vm_create_task)
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 554, in _wait_for_task
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] ret_val = done.wait()
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] return hubs.get_hub().switch()
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] return self.greenlet.switch()
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] NovaException: The input arguments had entities that did not belong to the same datacenter.
  2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539]

  2013-07-18 10:59:20.029 ERROR nova.compute.manager [req-e8306ffe-
  c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b
  0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-
  fe50f67b1539] Error: ['Traceback (most recent call last):\n', ' File
  "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 848,
  in _run_instance\n set_access_ip=set_access_ip)\n', ' File
  "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1107,
  in _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
  instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line
  24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-
  packages/nova/compute/manager.py", line 1103, in _spawn\n
  block_device_info)\n', ' File "/usr/lib/python2.7/dist-
  packages/nova/virt/vmwareapi/driver.py", line 177, in spawn\n
  block_device_info)\n', ' File "/usr/lib/python2.7/dist-
  packages/nova/virt/vmwareapi/vmops.py", line 217, in spawn\n
  _execute_create_vm()\n', ' File "/usr/lib/python2.7/dist-
  packages/nova/virt/vmwareapi/vmops.py", line 213, in
  _execute_create_vm\n self._session._wait_for_task(instance[\'uuid\'],
  vm_create_task)\n', ' File "/usr/lib/python2.7/dist-
  packages/nova/virt/vmwareapi/driver.py", line 554, in _wait_for_task\n
  ret_val = done.wait()\n', ' File "/usr/lib/python2.7/dist-
  packages/eventlet/event.py", line 116, in wait\n return
  hubs.get_hub().switch()\n', ' File "/usr/lib/python2.7/dist-
  packages/eventlet/hubs/hub.py", line 187, in switch\n return
  self.greenlet.switch()\n', 'NovaException: The input arguments had
  entities that did not belong to the same datacenter.\n']

  2013-07-18 10:59:23.831 3145 WARNING nova.virt.vmwareapi.driver [-] Task [CreateVM_Task] (returnval){
     value = "task-33558"
     _type = "Task"
   } status: error The input arguments had entities that did not belong to the same datacenter.
  2013-07-18 10:59:23.832 3145 WARNING nova.virt.vmwareapi.driver [-] In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
  2013-07-18 10:59:23.833 3145 ERROR nova.utils [-] in fixed duration looping call
  2013-07-18 10:59:23.833 3145 TRACE nova.utils Traceback (most recent call last):
  2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 594, in _inner
  2013-07-18 10:59:23.833 3145 TRACE nova.utils self.f(*self.args, **self.kw)
  2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 580, in _poll_task
  2013-07-18 10:59:23.833 3145 TRACE nova.utils done.send_exception(excep)
  2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 208, in send_exception
  2013-07-18 10:59:23.833 3145 TRACE nova.utils return self.send(None, args)
  2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 150, in send
  2013-07-18 10:59:23.833 3145 TRACE nova.utils assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
  2013-07-18 10:59:23.833 3145 TRACE nova.utils AssertionError: Trying to re-send() an already-triggered event.
  2013-07-18 10:59:23.833 3145 TRACE nova.utils

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1180044/+subscriptions