yahoo-eng-team team mailing list archive

[Szymon Wróblewski <1926417@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler spawns new dispatch_events thread (within futurist.ThreadPoolExecutor) with each OVO change with handle_event. 
Then dispatch_events is locked with lockutils.synchronized that ensures only one dispatch_events is executed at a time even between different OVO types, all the other threads are waiting doing nothing.
ThreadPoolExecutor has default upper limit of threads spawned equal to processor count * 5 for each _ObjectChangeHandler.

That means on a busy system with large amount of processors and multiple OVOs monitored we could be spawning:
>32 (proc_count) * 5 (ThreadPoolExecutor limit) * 9 (OVOs monitored) * >16 (Neutron API processes) = >23k threads

With that amount of threads spawned it's possible to hit max limit of
threads for user and "RuntimeError: can't start new thread" exception
starts appearing.

** Affects: neutron
     Importance: Undecided
     Assignee: Szymon Wróblewski (bluex)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Szymon Wróblewski (bluex)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926417

Title:
  ObjectChangeHandler excessive thread usage

Status in neutron:
  New

Bug description:
  neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler spawns new dispatch_events thread (within futurist.ThreadPoolExecutor) with each OVO change with handle_event. 
  Then dispatch_events is locked with lockutils.synchronized that ensures only one dispatch_events is executed at a time even between different OVO types, all the other threads are waiting doing nothing.
  ThreadPoolExecutor has default upper limit of threads spawned equal to processor count * 5 for each _ObjectChangeHandler.

  That means on a busy system with large amount of processors and multiple OVOs monitored we could be spawning:
  >32 (proc_count) * 5 (ThreadPoolExecutor limit) * 9 (OVOs monitored) * >16 (Neutron API processes) = >23k threads

  With that amount of threads spawned it's possible to hit max limit of
  threads for user and "RuntimeError: can't start new thread" exception
  starts appearing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926417/+subscriptions