yahoo-eng-team team mailing list archive

[OpenStack Infra <2053027@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/909474
Committed: https://opendev.org/openstack/nova/commit/198805c7c516e510f44890c4dec26a5fb8210cff
Submitter: "Zuul (22348)"
Branch:    master

commit 198805c7c516e510f44890c4dec26a5fb8210cff
Author: Fabian Wiesel <fabian.wiesel@xxxxxxx>
Date:   Mon Feb 19 18:41:32 2024 +0100

    Do not close returned image-chunk iterator & get_verifier early
    
    The GlanceClientWrapper._get_verifier method may fail already on the
    metadata, so we better call it early before we open files and start
    downloads, which we then abort uncleanly.
    
    This also likely how the bug #1948706 was triggered in the first place:
    - The file gets opened
    - _get_verifier fails *before* we even iterate over the data
    - glance_utils.IterableWithLength won't close the underlying iterator.
    
    The added close statement, now guarded with `may_close_iterator` is
    likely superfluous.
    
    If we return the image chunk iterator, then we should rather not
    close the underlying iterable, as it will kill the transfer.
    
    Closes-Bug: #2053027
    Change-Id: Ia247af39a96fbed90b027ad30158e66dd2f0bd5e


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2053027

Title:
  Nova VMwareapi Fails Image Download

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  When trying to start an instance with a local root disk, the VM fails with InvalidArgument.

  Steps to reproduce
  ==================
  A chronological list of steps which will bring off the
  issue you noticed:
  * Setup Devstack
  * Create Server with boot from volume:
    source devstack/openrc admin demo
    openstack server create --flavor 1 --boot-from-volume 10 --image-property vmware_disktype=sparse --wait test-bfv-disk
  * Create Server with root disk
    openstack server create --flavor 1 --image-property vmware_disktype=sparse --wait test-root-disk

  Expected result
  ===============
  Both instances come up.

  Actual result
  =============
  The first one comes up.
  The second one fails with the error
  > Instance failed to spawn: oslo_vmware.exceptions.VimFaultException: A specified parameter was not correct: fileType

  Further inspection yield, that the file is empty.
  The connection has been closed by the nova-compute prematurely on glance side
  > Unhandled error: OSError: write error
  as well on the vsphere side.

  Environment
  ===========
  1. Exact version of OpenStack you are running.

  Tested it with nova master (i.e.
  35af4b345d997b63f999a090e236d91b78ea4304)

  2. Which hypervisor did you use? What's the version of that?

  vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

  2. Which storage type did you use?

  vmdk on NFS 4.0

  3. Which networking type did you use?

  networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

  Logs & Configs
  ==============

  Logs for running on master:
  http://openstack-ci-logs.global.cloud.sap/openstack/nova/35af4b345d997b63f999a090e236d91b78ea4304/index.html

  Additional Information
  ========

  As the argument is an IterableWithLength (https://github.com/openstack/python-glanceclient/blob/master/glanceclient/common/utils.py#L563C7-L563C25),
  we can print debug log message in the ImageReadHandle (https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L765).
  When logging  the read bytes and the  and the file size, the length of the iterator matches the file-size stored in glance, while the `StopIteration` is raised right on the first call here:
  https://opendev.org/openstack/oslo.vmware/src/branch/master/oslo_vmware/rw_handles.py#L762

  Interestingly, the same code is used in cinder (the working case), and there the iterator is not raising a `StopIteration` before reaching the end of the file.
  So, possibly that is a red herring.

  The main difference is how Cinder and Nova handle the upload side.
  Nova uses a PUT on an individual ESXi host, while Cinder uses a POST operation to the VSphere API.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2053027/+subscriptions