yahoo-eng-team team mailing list archive

[Fabian Wiesel <2053027@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
When trying to start an instance with a local root disk, the VM fails with InvalidArgument.

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* Setup Devstack
* Create Server with boot from volume:
  openstack server create --flavor 1 --image-property --wait test-root-disk
* Create Server with root disk
  openstack server create --flavor 1 --image-property --wait test-root-disk

Expected result
===============
Both instances come up.

Actual result
=============
The first one comes up.
The second one fails with the error
> Instance failed to spawn: oslo_vmware.exceptions.VimFaultException: A specified parameter was not correct: fileType

Further inspection yield, that the file is empty.
The connection has been closed by the nova-compute prematurely on glance side
> Unhandled error: OSError: write error
as well on the vsphere side.

Environment
===========
1. Exact version of OpenStack you are running.

Tested it with nova master (i.e.
35af4b345d997b63f999a090e236d91b78ea4304)

2. Which hypervisor did you use? What's the version of that?

vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

2. Which storage type did you use?

vmdk on NFS 4.0

3. Which networking type did you use?

networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

Logs & Configs
==============

Logs for running on master:
http://openstack-ci-logs.global.cloud.sap/openstack/nova/35af4b345d997b63f999a090e236d91b78ea4304/index.html

Additional Information
========

As the argument is an IterableWithLength (https://github.com/openstack/python-glanceclient/blob/master/glanceclient/common/utils.py#L563C7-L563C25),
we can print debug log message in the ImageReadHandle (https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L765).
When logging  the read bytes and the  and the file size, the length of the iterator matches the file-size stored in glance, while the `StopIteration` is raised right on the first call here:
https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L762

Interestingly, the same code is used in cinder (the working case), and there the iterator is not raising a `StopIteration` before reaching the end of the file.
So, possibly that is a red herring.

The main difference is how Cinder and Nova handle the upload side.
Nova uses a PUT on an individual ESXi host, while Cinder uses a POST operation to the VSphere API.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2053027

Title:
  Nova VMwareapi Fails Image Download

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When trying to start an instance with a local root disk, the VM fails with InvalidArgument.

  Steps to reproduce
  ==================
  A chronological list of steps which will bring off the
  issue you noticed:
  * Setup Devstack
  * Create Server with boot from volume:
    openstack server create --flavor 1 --image-property --wait test-root-disk
  * Create Server with root disk
    openstack server create --flavor 1 --image-property --wait test-root-disk

  Expected result
  ===============
  Both instances come up.

  Actual result
  =============
  The first one comes up.
  The second one fails with the error
  > Instance failed to spawn: oslo_vmware.exceptions.VimFaultException: A specified parameter was not correct: fileType

  Further inspection yield, that the file is empty.
  The connection has been closed by the nova-compute prematurely on glance side
  > Unhandled error: OSError: write error
  as well on the vsphere side.

  Environment
  ===========
  1. Exact version of OpenStack you are running.

  Tested it with nova master (i.e.
  35af4b345d997b63f999a090e236d91b78ea4304)

  2. Which hypervisor did you use? What's the version of that?

  vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

  2. Which storage type did you use?

  vmdk on NFS 4.0

  3. Which networking type did you use?

  networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

  Logs & Configs
  ==============

  Logs for running on master:
  http://openstack-ci-logs.global.cloud.sap/openstack/nova/35af4b345d997b63f999a090e236d91b78ea4304/index.html

  Additional Information
  ========

  As the argument is an IterableWithLength (https://github.com/openstack/python-glanceclient/blob/master/glanceclient/common/utils.py#L563C7-L563C25),
  we can print debug log message in the ImageReadHandle (https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L765).
  When logging  the read bytes and the  and the file size, the length of the iterator matches the file-size stored in glance, while the `StopIteration` is raised right on the first call here:
  https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L762

  Interestingly, the same code is used in cinder (the working case), and there the iterator is not raising a `StopIteration` before reaching the end of the file.
  So, possibly that is a red herring.

  The main difference is how Cinder and Nova handle the upload side.
  Nova uses a PUT on an individual ESXi host, while Cinder uses a POST operation to the VSphere API.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2053027/+subscriptions