yahoo-eng-team team mailing list archive

[Supreeth Shivanand <supreeth90@xxxxxxxxx>]

Public bug reported:

Description: nova-scheduler schedules more than max_instances_per_host
tag set for an host aggregate.

Root cause: nova-scheduler has NumInstancesFilter which filters out
hosts exceeding max_instances_per_host instances. But, for that it
relies on host_state.num_instances which is retrieved from stats of
compute_node object. Due to some race condition, this stats of
compute_node object is retrieved as {}, which sets the
host_state.num_instance to 0. And because of this nova-scheduler
schedules more than max_instances_per_host instances thinking the
current host has 0 instances.

Workaround Fix: Changed the NumInstancesFilter to rely on
len(host_state.instances) when host_state.num_instances is 0.

Steps to reproduce:
1. I created 3 heat autoscaling stacks, forced to create instances on 1 host aggregate.
2. Load the cpu on instances of each of these stacks, also set the cooldown value for autoscale to 30 secs(typically some low value so that the we force the race condition).
3. Once the num of instances crosses max_instances_per_host in all the hosts in an host aggregate, instances start ending up in error state(filtered by AggregateNumInstancesFilter).
4. But sometimes(due to some race condition), stats in compute_node objects becomes {} and AggregateNumInstancesFilter doesn't filter those hosts with num instances >= max_instances_per_host.
5. I start seeing more then max_instances_per_host instances in these hosts.

Expected result:
Hosts in Host aggregates having max_instances_per_host tag, should never be scheduled with instances more than max_instances_per_host.

Actual Result:
Hosts in Host aggregates having max_instances_per_host tag, are scheduled with instances more than max_instances_per_host.

Environment:
openstack nova version: Newton release
Hypervisor: Libvirt + KVM

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1740320

Title:
  nova-scheduler does not honor max_instances_per_host set to a host
  aggregate

Status in OpenStack Compute (nova):
  New

Bug description:
  Description: nova-scheduler schedules more than max_instances_per_host
  tag set for an host aggregate.

  Root cause: nova-scheduler has NumInstancesFilter which filters out
  hosts exceeding max_instances_per_host instances. But, for that it
  relies on host_state.num_instances which is retrieved from stats of
  compute_node object. Due to some race condition, this stats of
  compute_node object is retrieved as {}, which sets the
  host_state.num_instance to 0. And because of this nova-scheduler
  schedules more than max_instances_per_host instances thinking the
  current host has 0 instances.

  Workaround Fix: Changed the NumInstancesFilter to rely on
  len(host_state.instances) when host_state.num_instances is 0.

  Steps to reproduce:
  1. I created 3 heat autoscaling stacks, forced to create instances on 1 host aggregate.
  2. Load the cpu on instances of each of these stacks, also set the cooldown value for autoscale to 30 secs(typically some low value so that the we force the race condition).
  3. Once the num of instances crosses max_instances_per_host in all the hosts in an host aggregate, instances start ending up in error state(filtered by AggregateNumInstancesFilter).
  4. But sometimes(due to some race condition), stats in compute_node objects becomes {} and AggregateNumInstancesFilter doesn't filter those hosts with num instances >= max_instances_per_host.
  5. I start seeing more then max_instances_per_host instances in these hosts.

  Expected result:
  Hosts in Host aggregates having max_instances_per_host tag, should never be scheduled with instances more than max_instances_per_host.

  Actual Result:
  Hosts in Host aggregates having max_instances_per_host tag, are scheduled with instances more than max_instances_per_host.

  Environment:
  openstack nova version: Newton release
  Hypervisor: Libvirt + KVM

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1740320/+subscriptions