yahoo-eng-team team mailing list archive

[Rawlin Peters <rawlin.peters@xxxxxx>]

Public bug reported:

The L3 agent currently issues ARP updates one at a time while processing
a DVR router. Each ARP update creates an external process which has to
call the neutron-rootwrap helper while also "ip netns exec <qrouter
namespace>" -ing each time.

The ip command contains a "-batch <FILENAME>" option which would be able
to batch all of the "ip neigh replace" commands into one external
process per qrouter namespace. This would greatly reduce the amount of
time it takes the L3 agent to update large numbers of ARP entries,
particularly as the number of VMs in a deployment rises.

The benefit of batching ip commands can be seen in this simple bash
example:

$ time for i in {0..50}; do sudo ip netns exec qrouter-bc38451e-0c2f-
4ad2-b76b-daa84066fefb ip a > /dev/null; done

real	    0m2.437s
user    0m0.183s
sys	    0m0.359s
$ for i in {0..50}; do echo a >> /tmp/ip_batch_test; done
$ time sudo ip netns exec qrouter-bc38451e-0c2f-4ad2-b76b-daa84066fefb ip -b /tmp/ip_batch_test > /dev/null

real	        0m0.046s
user        0m0.003s
sys	        0m0.007s

If just 50 arp updates are batched together, there is about a 50x
speedup. Repeating this test with 500 commands showed a speedup of 250x
(disclaimer: this was a rudimentary test just to get a rough estimate of
the performance benefit).

** Affects: neutron
     Importance: Undecided
     Assignee: Rawlin Peters (rawlin-peters)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Rawlin Peters (rawlin-peters)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1511134

Title:
  Batch DVR ARP updates

Status in neutron:
  In Progress

Bug description:
  The L3 agent currently issues ARP updates one at a time while
  processing a DVR router. Each ARP update creates an external process
  which has to call the neutron-rootwrap helper while also "ip netns
  exec <qrouter namespace>" -ing each time.

  The ip command contains a "-batch <FILENAME>" option which would be
  able to batch all of the "ip neigh replace" commands into one external
  process per qrouter namespace. This would greatly reduce the amount of
  time it takes the L3 agent to update large numbers of ARP entries,
  particularly as the number of VMs in a deployment rises.

  The benefit of batching ip commands can be seen in this simple bash
  example:

  $ time for i in {0..50}; do sudo ip netns exec qrouter-bc38451e-0c2f-
  4ad2-b76b-daa84066fefb ip a > /dev/null; done

  real	    0m2.437s
  user    0m0.183s
  sys	    0m0.359s
  $ for i in {0..50}; do echo a >> /tmp/ip_batch_test; done
  $ time sudo ip netns exec qrouter-bc38451e-0c2f-4ad2-b76b-daa84066fefb ip -b /tmp/ip_batch_test > /dev/null

  real	        0m0.046s
  user        0m0.003s
  sys	        0m0.007s

  If just 50 arp updates are batched together, there is about a 50x
  speedup. Repeating this test with 500 commands showed a speedup of
  250x (disclaimer: this was a rudimentary test just to get a rough
  estimate of the performance benefit).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1511134/+subscriptions