yahoo-eng-team team mailing list archive

[Balazs Gibizer <2083246@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Today the nova-compute service does not limit the number of concurrent
evacuation requests. The [compute]max_concurrent_build config only
considered for new VM builds but not for re-builds due to incoming
evacuation.

If the evacuated VMs are on shared storage then there is no heavy IO
operations (no image download / convert) then the
[compute]max_concurrent_disk_ops config could not prevent the overload
either.

At some point nova-compute will start failing to process incoming vif-
plugged events in a timely manner. (In a specific env it happened after
more than 60 concurrent evacuation requests targeting the same node)

The [compute]max_concurrent_build config description does not explicitly
states that it is only counts new build requests and ignores rebuilds.
So I consider it a bug in nova-compute that it is not limiting all the
builds by that config option. As this config option is defaulted to 10
it shows that nova never planned to really support significantly more
than 10 concurrent so failing at 60 concurrent evacuation does not need
t be supported.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: compute evacuate

** Tags added: evacuate

** Tags added: compute

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2083246

Title:
  nova-compute can be overloaded with incoming evacuations

Status in OpenStack Compute (nova):
  New

Bug description:
  Today the nova-compute service does not limit the number of concurrent
  evacuation requests. The [compute]max_concurrent_build config only
  considered for new VM builds but not for re-builds due to incoming
  evacuation.

  If the evacuated VMs are on shared storage then there is no heavy IO
  operations (no image download / convert) then the
  [compute]max_concurrent_disk_ops config could not prevent the overload
  either.

  At some point nova-compute will start failing to process incoming vif-
  plugged events in a timely manner. (In a specific env it happened
  after more than 60 concurrent evacuation requests targeting the same
  node)

  The [compute]max_concurrent_build config description does not
  explicitly states that it is only counts new build requests and
  ignores rebuilds. So I consider it a bug in nova-compute that it is
  not limiting all the builds by that config option. As this config
  option is defaulted to 10 it shows that nova never planned to really
  support significantly more than 10 concurrent so failing at 60
  concurrent evacuation does not need t be supported.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2083246/+subscriptions