yahoo-eng-team team mailing list archive

[OpenStack Infra <1926417@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/788510
Committed: https://opendev.org/openstack/neutron/commit/9ae22b1fefabe8bf18d5dd26f49821b673bd39d4
Submitter: "Zuul (22348)"
Branch:    master

commit 9ae22b1fefabe8bf18d5dd26f49821b673bd39d4
Author: Szymon Wroblewski <szymon.wroblewski@xxxxxxxxxxxx>
Date:   Mon Apr 26 17:25:42 2021 +0200

    Fix ObjectChangeHandler thread usage
    
    Instead of spawning new thread for each OVO change,
    keep one long running thread fetching changes
    from queue.
    
    Closes-Bug: #1926417
    Change-Id: I390cabeaf6ebbc1c6206fe2cc226ef437462c7fd


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926417

Title:
  ObjectChangeHandler excessive thread usage

Status in neutron:
  Fix Released

Bug description:
  neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler spawns new dispatch_events thread (within futurist.ThreadPoolExecutor) with each OVO change with handle_event. 
  Then dispatch_events is locked with lockutils.synchronized that ensures only one dispatch_events is executed at a time even between different OVO types, all the other threads are waiting doing nothing.
  ThreadPoolExecutor has default upper limit of threads spawned equal to processor count * 5 for each _ObjectChangeHandler.

  That means on a busy system with large amount of processors and multiple OVOs monitored we could be spawning:
  >32 (proc_count) * 5 (ThreadPoolExecutor limit) * 9 (OVOs monitored) * >16 (Neutron API processes) = >23k threads

  With that amount of threads spawned it's possible to hit max limit of
  threads for user and "RuntimeError: can't start new thread" exception
  starts appearing.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926417/+subscriptions