yahoo-eng-team team mailing list archive

[Fabian Wiesel <1940399@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========

There are two functions in the code, which call (indirectly) RetrievePropertyEx with at most vmware.maximum_objects per result, but do not iterate over the results if there are more, nor does it 
By default the value is 100, which is usually sufficient, but if someone should set the value lower, it will cause the following problems.

Both are in the module nova.virt.vmwarapi.vm_util
1. get_all_cluster_mors
   The function is called to find the cluster reference for named cluster. If there are more clusters than maximum_objects, then there is a chance that the cluster won't be found by the driver despite being configured in the VCenter.

2. get_stats_from_cluster
   The function gets the statistics for all the hosts in a cluster. Since VSphere 7.0U1, the limit is 96 hosts per cluster, and quite likely the limit will increase past the default of 100, leaving the stats inaccurate. On top of it, the function does not call `cancel_retrieval` causing a leak in the VCenter.


Steps to reproduce
==================

Take any VCenter version with more than one cluster and more than one host (I would suggest three or more), and any release of Nova configured to run against the VCenter.
I am not 100% according what the order is by which the clusters are returned, possibly in the order they have been created, or alphabetically.  I would suggest to create first a cluster-a, and then a cluster-b, and add the hosts to cluster-b. That reflects how our clusters are created (chronologically alphabetically sorted).

* Configure additionally
  [vmware]
  maximum_objects=1
* Try to start nova-compute

Expected result
===============
The nova-compute service would start up and get the stats from all the hosts in the configured cluster.

Actual result
=============

nova-compute fails to start with the error message:
> The specified cluster '<clustername>' was not found in vCenter

If you have to clusters, and three hosts, then increasing
maximum_objects to two will get you around that failure, and will
trigger the second problem.

You can verify that by checking the resources of the nova-compute node,
which will report only the resources (CPUs,RAM...) of two of the ESXi-
hosts in the cluster.


Environment
===========
1. Exact version of OpenStack you are running. 
   370830e944 Merge "libvirt: Enable 'vmcoreinfo' feature by default"

   As far as I can see, all Nova releases are affected by this behavior.

2. Which hypervisor did you use?
   VMware VSphere
   What's the version of that?
   7.0.1-17327586

3. Which networking type did you use?
   Neutron with NSX-T (https://github.com/sapcc/networking-nsx-t)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1940399

Title:
  vmware driver fails when setting vmware.maximum_objects to small value

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  There are two functions in the code, which call (indirectly) RetrievePropertyEx with at most vmware.maximum_objects per result, but do not iterate over the results if there are more, nor does it 
  By default the value is 100, which is usually sufficient, but if someone should set the value lower, it will cause the following problems.

  Both are in the module nova.virt.vmwarapi.vm_util
  1. get_all_cluster_mors
     The function is called to find the cluster reference for named cluster. If there are more clusters than maximum_objects, then there is a chance that the cluster won't be found by the driver despite being configured in the VCenter.

  2. get_stats_from_cluster
     The function gets the statistics for all the hosts in a cluster. Since VSphere 7.0U1, the limit is 96 hosts per cluster, and quite likely the limit will increase past the default of 100, leaving the stats inaccurate. On top of it, the function does not call `cancel_retrieval` causing a leak in the VCenter.

  
  Steps to reproduce
  ==================

  Take any VCenter version with more than one cluster and more than one host (I would suggest three or more), and any release of Nova configured to run against the VCenter.
  I am not 100% according what the order is by which the clusters are returned, possibly in the order they have been created, or alphabetically.  I would suggest to create first a cluster-a, and then a cluster-b, and add the hosts to cluster-b. That reflects how our clusters are created (chronologically alphabetically sorted).

  * Configure additionally
    [vmware]
    maximum_objects=1
  * Try to start nova-compute

  Expected result
  ===============
  The nova-compute service would start up and get the stats from all the hosts in the configured cluster.

  Actual result
  =============

  nova-compute fails to start with the error message:
  > The specified cluster '<clustername>' was not found in vCenter

  If you have to clusters, and three hosts, then increasing
  maximum_objects to two will get you around that failure, and will
  trigger the second problem.

  You can verify that by checking the resources of the nova-compute
  node, which will report only the resources (CPUs,RAM...) of two of the
  ESXi-hosts in the cluster.

  
  Environment
  ===========
  1. Exact version of OpenStack you are running. 
     370830e944 Merge "libvirt: Enable 'vmcoreinfo' feature by default"

     As far as I can see, all Nova releases are affected by this
  behavior.

  2. Which hypervisor did you use?
     VMware VSphere
     What's the version of that?
     7.0.1-17327586

  3. Which networking type did you use?
     Neutron with NSX-T (https://github.com/sapcc/networking-nsx-t)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1940399/+subscriptions