yahoo-eng-team team mailing list archive

[Assaf Muller <amuller@xxxxxxxxxx>]

Public bug reported:

In the L3 agent's _process_routers_loop method, it spawns a GreenPool
with 8 eventlet threads. Those threads then take updates off the agent's
queue and process router updates. Router updates are serialized by
router_id so that two threads don't process the same router at any given
time.

In an environment running on a powerful baremetal server, on agent
restart it was trying to sync roughly 600 routers. Around half were HA
routers, and half were legacy routers. With the default GreenPool size
of 8, the result was that the server ground to a halt as CPU usage
skyrocketed to over 600%. The main offenders were ip, bash, keepalived
and Python. This was on an environment without rootwrap daemon based off
stable/juno. It took around 60 seconds to configure a single router.
Changing the GreenPool size from 8 to 1, caused the agent to:

1) Configure a router in 30 seconds, a 50% improvement.
2) Reduce CPU load from 600% to 70%, freeing the machine to do other things.

I'm filing this bug so that:

1) Someone can confirm my personal experience in a more controlled way - For example, graph router configuration time and CPU load as a result of GreenPool size.
2) If my findings are confirmed on master with rootwrap daemon, start considering alternatives like multiprocessing instead of eventlet multithreading, or at the very least optimize the GreenPool size.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ipam-dhcp loadimpact

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1526559

Title:
  L3 agent parallel configuration of routers might slow things down

Status in neutron:
  New

Bug description:
  In the L3 agent's _process_routers_loop method, it spawns a GreenPool
  with 8 eventlet threads. Those threads then take updates off the
  agent's queue and process router updates. Router updates are
  serialized by router_id so that two threads don't process the same
  router at any given time.

  In an environment running on a powerful baremetal server, on agent
  restart it was trying to sync roughly 600 routers. Around half were HA
  routers, and half were legacy routers. With the default GreenPool size
  of 8, the result was that the server ground to a halt as CPU usage
  skyrocketed to over 600%. The main offenders were ip, bash, keepalived
  and Python. This was on an environment without rootwrap daemon based
  off stable/juno. It took around 60 seconds to configure a single
  router. Changing the GreenPool size from 8 to 1, caused the agent to:

  1) Configure a router in 30 seconds, a 50% improvement.
  2) Reduce CPU load from 600% to 70%, freeing the machine to do other things.

  I'm filing this bug so that:

  1) Someone can confirm my personal experience in a more controlled way - For example, graph router configuration time and CPU load as a result of GreenPool size.
  2) If my findings are confirmed on master with rootwrap daemon, start considering alternatives like multiprocessing instead of eventlet multithreading, or at the very least optimize the GreenPool size.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1526559/+subscriptions