yahoo-eng-team team mailing list archive

[Uday <nagaraj@xxxxxxxxxx>]

Public bug reported:

We have been trying to analyze why under scale creating additional
routers and associating floating IP address with a VM causes the
operation to take longer completion times, and have found that the
programming of iptables seems to be an issue. Particularly 4 functions
(and their use) seem to degrade with large numbers of router instances.

We gathered this data with cprofile on the L3 agent. Tests were run to
collect data on the first router instantiation and on the 40th router
instantiation, with data also being collected at points between 1 and 40
routers. All the following functions showed an increasing trend:

(_find_last_entry)
(_weed_out_removes)
(_weed_out_duplicate_chains)
(_weed_out_duplicate_rules)

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

For first router instantiation and Floating IP associate:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
126    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)

40 run of creating router

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
313    0.001    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)


In particular, the weed out routines seem to do multiple iterations of matching iptables rules with certain rules and then operate on them. The increasing iteration numbers the weed out routines seems to be degrading the performance.

Does someone have information on whether this can be optimized?

Thanks,
-Uday

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1497379

Title:
  iptables programming: Adding router instances takes increasing amount
  of time

Status in neutron:
  New

Bug description:
  We have been trying to analyze why under scale creating additional
  routers and associating floating IP address with a VM causes the
  operation to take longer completion times, and have found that the
  programming of iptables seems to be an issue. Particularly 4 functions
  (and their use) seem to degrade with large numbers of router
  instances.

  We gathered this data with cprofile on the L3 agent. Tests were run to
  collect data on the first router instantiation and on the 40th router
  instantiation, with data also being collected at points between 1 and
  40 routers. All the following functions showed an increasing trend:

  (_find_last_entry)
  (_weed_out_removes)
  (_weed_out_duplicate_chains)
  (_weed_out_duplicate_rules)

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)

  For first router instantiation and Floating IP associate:

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  126    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
  178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
  178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
  178    0.000    0.000    0.000    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)

  40 run of creating router

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  313    0.001    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
  371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
  371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
  371    0.000    0.000    0.004    0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)

  
  In particular, the weed out routines seem to do multiple iterations of matching iptables rules with certain rules and then operate on them. The increasing iteration numbers the weed out routines seems to be degrading the performance.

  Does someone have information on whether this can be optimized?

  Thanks,
  -Uday

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1497379/+subscriptions