yahoo-eng-team team mailing list archive

[Alex Krzos <akrzos@xxxxxxxxxx>]

Public bug reported:

Description of problem:
While attempting to achieve even distribution of tiny instances across 31 compute nodes (Some with 64GiB and some with 128GiB of memory) I tried to use the NumInstancesFilter to limit the number of instance per each compute to 2 instances. (Total limit of 62 instances for this testbed then 31 x 2 = 62)  I then launched 70 guests serially (concurrency of 1) using a rally scenario built for this with persisting rally instances.  67 instances were able to be booted and only 3 failed.

Two hosts ended up with 4 instances, one ended up with 3 instances and
the rest with 2 instances as they should have.


I used the following settings in nova.conf on all controllers (3):
[filter_scheduler]
host_subset_size = 4
max_instances_per_host = 2
enabled_filters = NumInstancesFilter,RetryFilter,RamFilter,ComputeFilter
ram_weight_multiplier = 0

After setting above I had restarted services:
        - openstack-nova-scheduler
        - openstack-nova-api
        - openstack-nova-conductor
        - openstack-nova-novncproxy
        - openstack-nova-consoleauth
        - httpd

*httpd hosts the nova placement api


Version-Release number of selected component (if applicable):
OpenStack Ocata
python-nova-15.0.2-1.el7ost.noarch
openstack-nova-cert-15.0.2-1.el7ost.noarch
openstack-nova-console-15.0.2-1.el7ost.noarch
puppet-nova-10.4.0-3.el7ost.noarch
openstack-nova-novncproxy-15.0.2-1.el7ost.noarch
openstack-nova-placement-api-15.0.2-1.el7ost.noarch
python-novaclient-7.1.0-1.el7ost.noarch
openstack-nova-common-15.0.2-1.el7ost.noarch
openstack-nova-scheduler-15.0.2-1.el7ost.noarch
openstack-nova-conductor-15.0.2-1.el7ost.noarch
openstack-nova-compute-15.0.2-1.el7ost.noarch
openstack-nova-api-15.0.2-1.el7ost.noarch


How reproducible:
Produced above result once.  Unsure if reproduces every single time due to time limitations on testbed.

Steps to Reproduce:
1. Set max_instances_per_host to 2, set enabled_filters to include NumInstancesFilter
2. Restart nova services, attempt to boot > max_instances_per_host*$HOST_COUNT and witness more instances than should be possible
3.

Actual results:
Three hosts had > max_instances_per_host

Expected results:
Only 62 instances to be booted

Additional info:

Perhaps I configured something wrong with Nova?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1685010

Title:
  Able to spawn > max_instances_per_host with NumInstancesFilter

Status in OpenStack Compute (nova):
  New

Bug description:
  Description of problem:
  While attempting to achieve even distribution of tiny instances across 31 compute nodes (Some with 64GiB and some with 128GiB of memory) I tried to use the NumInstancesFilter to limit the number of instance per each compute to 2 instances. (Total limit of 62 instances for this testbed then 31 x 2 = 62)  I then launched 70 guests serially (concurrency of 1) using a rally scenario built for this with persisting rally instances.  67 instances were able to be booted and only 3 failed.

  Two hosts ended up with 4 instances, one ended up with 3 instances and
  the rest with 2 instances as they should have.

  
  I used the following settings in nova.conf on all controllers (3):
  [filter_scheduler]
  host_subset_size = 4
  max_instances_per_host = 2
  enabled_filters = NumInstancesFilter,RetryFilter,RamFilter,ComputeFilter
  ram_weight_multiplier = 0

  After setting above I had restarted services:
          - openstack-nova-scheduler
          - openstack-nova-api
          - openstack-nova-conductor
          - openstack-nova-novncproxy
          - openstack-nova-consoleauth
          - httpd

  *httpd hosts the nova placement api


  Version-Release number of selected component (if applicable):
  OpenStack Ocata
  python-nova-15.0.2-1.el7ost.noarch
  openstack-nova-cert-15.0.2-1.el7ost.noarch
  openstack-nova-console-15.0.2-1.el7ost.noarch
  puppet-nova-10.4.0-3.el7ost.noarch
  openstack-nova-novncproxy-15.0.2-1.el7ost.noarch
  openstack-nova-placement-api-15.0.2-1.el7ost.noarch
  python-novaclient-7.1.0-1.el7ost.noarch
  openstack-nova-common-15.0.2-1.el7ost.noarch
  openstack-nova-scheduler-15.0.2-1.el7ost.noarch
  openstack-nova-conductor-15.0.2-1.el7ost.noarch
  openstack-nova-compute-15.0.2-1.el7ost.noarch
  openstack-nova-api-15.0.2-1.el7ost.noarch

  
  How reproducible:
  Produced above result once.  Unsure if reproduces every single time due to time limitations on testbed.

  Steps to Reproduce:
  1. Set max_instances_per_host to 2, set enabled_filters to include NumInstancesFilter
  2. Restart nova services, attempt to boot > max_instances_per_host*$HOST_COUNT and witness more instances than should be possible
  3.

  Actual results:
  Three hosts had > max_instances_per_host

  Expected results:
  Only 62 instances to be booted

  Additional info:

  Perhaps I configured something wrong with Nova?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1685010/+subscriptions