yahoo-eng-team team mailing list archive

[xhzhf <guoyongxhzhf@xxxxxxx>]

Public bug reported:

Description
===========
we test cellv2 solution and execute creating instances concurrently.
But there is the performance issue due to query instance info in scheduler process 

Steps to reproduce
==================
use rally to execute creating 500 instances concurrently 

Expected result
===============
all instances are created successfully

Actual result
=============
many instances failed due to timeout

Environment
===========
1. Exact version of OpenStack you are running. See the following
openstack-nova-scheduler-16.0.3-1.el7.noarch
openstack-nova-placement-api16.0.3-1.el7.noarch

2. Which hypervisor did you use?
KVM. We have 2000 hosts

2. Which storage type did you use?
Our own storage device

3. Which networking type did you use?
Our own network device

Analysis
==============
In scheduler process, get_host_states_by_uuids calls _get_host_state.
_get_host_state call host_state.update and _get_instance_info
So _get_instance_info will be called at every request and every hosts.
Because cellv2 have disable instance info cache.
So there's a performance issue even we don't use affinity filter policy

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737465

Title:
  [cellv2] the performance issue of cellv2 when creating 500 instances
  concurrently

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  we test cellv2 solution and execute creating instances concurrently.
  But there is the performance issue due to query instance info in scheduler process 

  Steps to reproduce
  ==================
  use rally to execute creating 500 instances concurrently 

  Expected result
  ===============
  all instances are created successfully

  Actual result
  =============
  many instances failed due to timeout

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
  openstack-nova-scheduler-16.0.3-1.el7.noarch
  openstack-nova-placement-api16.0.3-1.el7.noarch

  2. Which hypervisor did you use?
  KVM. We have 2000 hosts

  2. Which storage type did you use?
  Our own storage device

  3. Which networking type did you use?
  Our own network device

  Analysis
  ==============
  In scheduler process, get_host_states_by_uuids calls _get_host_state.
  _get_host_state call host_state.update and _get_instance_info
  So _get_instance_info will be called at every request and every hosts.
  Because cellv2 have disable instance info cache.
  So there's a performance issue even we don't use affinity filter policy

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737465/+subscriptions