yahoo-eng-team team mailing list archive

[Carl Baldwin <carl@xxxxxxxxxxxxx>]

Interesting.  So, having multiple threads doesn't improve the execution
time at all?  That's different than I remember.

One thing that have some number of threads may do is possibly decrease
the latency to process a new request while some threads are working on
other routers.

** Changed in: neutron
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1526559

Title:
  L3 agent parallel configuration of routers might slow things down

Status in neutron:
  Invalid

Bug description:
  In the L3 agent's _process_routers_loop method, it spawns a GreenPool
  with 8 eventlet threads. Those threads then take updates off the
  agent's queue and process router updates. Router updates are
  serialized by router_id so that two threads don't process the same
  router at any given time.

  In an environment running on a powerful baremetal server, on agent
  restart it was trying to sync roughly 600 routers. Around half were HA
  routers, and half were legacy routers. With the default GreenPool size
  of 8, the result was that the server ground to a halt as CPU usage
  skyrocketed to over 600%. The main offenders were ip, bash, keepalived
  and Python. This was on an environment without rootwrap daemon based
  off stable/juno. It took around 60 seconds to configure a single
  router. Changing the GreenPool size from 8 to 1, caused the agent to:

  1) Configure a router in 30 seconds, a 50% improvement.
  2) Reduce CPU load from 600% to 70%, freeing the machine to do other things.

  I'm filing this bug so that:

  1) Someone can confirm my personal experience in a more controlled way - For example, graph router configuration time and CPU load as a result of GreenPool size.
  2) If my findings are confirmed on master with rootwrap daemon, start considering alternatives like multiprocessing instead of eventlet multithreading, or at the very least optimize the GreenPool size.

  This was on RHEL 7.1:
  kernel-3.10.0-229.11.1.el7, iproute-3.10.0-21.el7

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1526559/+subscriptions