yahoo-eng-team team mailing list archive

[Chaoyi Huang <joehuang@xxxxxxxxxx>]

Public bug reported:

https://github.com/osrg/quantum/blob/master/neutron/plugins/ml2/drivers/l2pop/rpc.py

def _notification_fanout(self, context, method, fdb_entries):
        ....
        self.fanout_cast(context,
                         self.make_msg(method, fdb_entries=fdb_entries),
                         topic=self.topic_l2pop_update)

the fanout_cast will publish the message to all L2 agents listening
"l2population" topic.

If there are 1000 agents (it is a small cloud), and all of them are
listening to  "l2population" topic, adding one new port will leads to
1000 sub messages. Generally rabbitMQ can handle 10k messages per
second, and the fanout_cast method will leads to greatly performance
issues, and make the neutron service hard to scale, the concurrency of
VM port request will be very very small.

No matter how many ports in the subnet, the performance is up to the
number of the L2 agents listening the topic.

The way to solve the performance and scalability issue is to make the L2
agent listening a topic related to network, for example, using network
uuid as the topic. If one port is activated in the subnet, only those
agents where there are VMs of the same network should receive the L2-pop
message.  This is parial-mesh, the original design purpose, but not
implemented yet.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l2 l2-pop

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1337717

Title:
  L2-population fanout-cast leads to performance and scalability issue

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  https://github.com/osrg/quantum/blob/master/neutron/plugins/ml2/drivers/l2pop/rpc.py

  def _notification_fanout(self, context, method, fdb_entries):
          ....
          self.fanout_cast(context,
                           self.make_msg(method, fdb_entries=fdb_entries),
                           topic=self.topic_l2pop_update)

  the fanout_cast will publish the message to all L2 agents listening
  "l2population" topic.

  If there are 1000 agents (it is a small cloud), and all of them are
  listening to  "l2population" topic, adding one new port will leads to
  1000 sub messages. Generally rabbitMQ can handle 10k messages per
  second, and the fanout_cast method will leads to greatly performance
  issues, and make the neutron service hard to scale, the concurrency of
  VM port request will be very very small.

  No matter how many ports in the subnet, the performance is up to the
  number of the L2 agents listening the topic.

  The way to solve the performance and scalability issue is to make the
  L2 agent listening a topic related to network, for example, using
  network uuid as the topic. If one port is activated in the subnet,
  only those agents where there are VMs of the same network should
  receive the L2-pop message.  This is parial-mesh, the original design
  purpose, but not implemented yet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1337717/+subscriptions