yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1694543] [NEW] VMware CI is failing with error "VimFaultException: A specified parameter was not correct: Faults: ['InvalidArgument']"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 30 May 2017 21:40:01 -0000
Reply-to: Bug 1694543 <1694543@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://207.189.188.190/logs/47/468147/1/check-vote/ext-nova-
zuul/cbab6d1/n-cpu.log.gz?level=TRACE#_2017-05-26_22_56_46_579

Which resulted in nova-compute auto-disabling the service.

2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall FileNotFoundException: File [mgmt-ds] 192.168.1.6_base/18b6f8c1-a6f3-4d32-9ff9-76403680a8ee was not found
2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall 
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall FileNotFoundException: File [mgmt-ds] 192.168.1.6_base/18b6f8c1-a6f3-4d32-9ff9-76403680a8ee was not found
2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall 
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall VimFaultException: A specified parameter was not correct: 
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall Faults: ['InvalidArgument']
2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall 
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [req-93e2d8e1-a1ea-42de-8103-69196f07fd53 tempest-AggregatesAdminTestJSON-2070234197 tempest-AggregatesAdminTestJSON-2070234197] [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Instance failed to spawn
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Traceback (most recent call last):
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/compute/manager.py", line 2157, in _build_resources
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     yield resources
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/compute/manager.py", line 1963, in _build_and_run_instance
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     block_device_info=block_device_info)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 324, in spawn
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     admin_password, network_info, block_device_info)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 763, in spawn
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     self._fetch_image_if_missing(context, vi)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 617, in _fetch_image_if_missing
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     tmp_dir_loc, tmp_image_ds_loc = image_prepare(vi)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 479, in _prepare_flat_image
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     vi.ii.file_size_in_kb)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 1378, in create_virtual_disk
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     session._wait_for_task(vmdk_create_task)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 546, in _wait_for_task
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return self.wait_for_task(task_ref)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 396, in wait_for_task
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return evt.wait()
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return hubs.get_hub().switch()
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return self.greenlet.switch()
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     self.f(*self.args, **self.kw)
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     raise task_ex
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] VimFaultException: A specified parameter was not correct: 
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Faults: ['InvalidArgument']
2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]

** Affects: nova
     Importance: Undecided
         Status: Confirmed


** Tags: vmware

** Changed in: nova
       Status: New => Confirmed

** Tags added: vmware

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1694543

Title:
  VMware CI is failing with error "VimFaultException: A specified
  parameter was not correct: Faults: ['InvalidArgument']"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://207.189.188.190/logs/47/468147/1/check-vote/ext-nova-
  zuul/cbab6d1/n-cpu.log.gz?level=TRACE#_2017-05-26_22_56_46_579

  Which resulted in nova-compute auto-disabling the service.

  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall FileNotFoundException: File [mgmt-ds] 192.168.1.6_base/18b6f8c1-a6f3-4d32-9ff9-76403680a8ee was not found
  2017-05-26 22:55:27.826 19424 ERROR oslo_vmware.common.loopingcall 
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall FileNotFoundException: File [mgmt-ds] 192.168.1.6_base/18b6f8c1-a6f3-4d32-9ff9-76403680a8ee was not found
  2017-05-26 22:55:28.386 19424 ERROR oslo_vmware.common.loopingcall 
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall     self.f(*self.args, **self.kw)
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall     raise task_ex
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall VimFaultException: A specified parameter was not correct: 
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall Faults: ['InvalidArgument']
  2017-05-26 22:55:28.945 19424 ERROR oslo_vmware.common.loopingcall 
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [req-93e2d8e1-a1ea-42de-8103-69196f07fd53 tempest-AggregatesAdminTestJSON-2070234197 tempest-AggregatesAdminTestJSON-2070234197] [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Instance failed to spawn
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Traceback (most recent call last):
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/compute/manager.py", line 2157, in _build_resources
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     yield resources
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/compute/manager.py", line 1963, in _build_and_run_instance
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     block_device_info=block_device_info)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 324, in spawn
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     admin_password, network_info, block_device_info)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 763, in spawn
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     self._fetch_image_if_missing(context, vi)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 617, in _fetch_image_if_missing
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     tmp_dir_loc, tmp_image_ds_loc = image_prepare(vi)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 479, in _prepare_flat_image
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     vi.ii.file_size_in_kb)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 1378, in create_virtual_disk
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     session._wait_for_task(vmdk_create_task)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 546, in _wait_for_task
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return self.wait_for_task(task_ref)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 396, in wait_for_task
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return evt.wait()
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return hubs.get_hub().switch()
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     return self.greenlet.switch()
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 75, in _inner
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     self.f(*self.args, **self.kw)
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 452, in _poll_task
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]     raise task_ex
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] VimFaultException: A specified parameter was not correct: 
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978] Faults: ['InvalidArgument']
  2017-05-26 22:55:28.946 19424 ERROR nova.compute.manager [instance: 90960b16-aa62-46ba-82d3-842d7439d978]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1694543/+subscriptions
Follow ups

[Bug 1694543] Re: VMware CI is failing with error "VimFaultException: A specified parameter was not correct: Faults: ['InvalidArgument']"
From: Radoslav Gerganov, 2017-10-12