yahoo-eng-team team mailing list archive

[Aleksandr Shaposhnikov <ashaposhnikov@xxxxxxxxxxxx>]

Public bug reported:

During my testing Openstack with Neutron on scale I found that security group updates for the neutron-ovs-agent are suboptimal. The agent will request the security group rules for all of the ports attached to OVS. The server will then provide the rules for each individual port,
which are almost identical for the ports in the same security group. This becomes extremely large if the security group has a lot of members and the OVS agent has a lot of ports.

So here is some math:
If a security group has 2000 VM’s spread across and 50 compute nodes, the average node will have 40 VMs. If a new VM is launched in the same security group, each compute node will get a notification and request the security group info that will have 2001 entries for each of the ~40 ports on that node. That’s ~80k records that need to be delivered to 50 compute nodes in a short period of time. The only difference between each port’s list of rules is that the port’s fixed_ips are excluded.

I suggest approach when there would be only one response for node that
contains 2000+1 records. The agent would be responsible for taking the
list of rules for the security group and applying it per port by
excluding the rule referencing the port’s address. Besides the
generation of useless information and a lot of work done on neutron-
server side it will significantly decrease load on oslo.messaging and
neutron-server. Right now in my env (25 computes; 1500VMs) a security
groups response could be up to 32Mb for each compute node/ovs-agent.

** Affects: mos
     Importance: Undecided
         Status: New

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: scale

** Also affects: mos
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1432858

Title:
  Suboptimal security groups calculation for nodes

Status in Mirantis OpenStack:
  New
Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  During my testing Openstack with Neutron on scale I found that security group updates for the neutron-ovs-agent are suboptimal. The agent will request the security group rules for all of the ports attached to OVS. The server will then provide the rules for each individual port,
  which are almost identical for the ports in the same security group. This becomes extremely large if the security group has a lot of members and the OVS agent has a lot of ports.

  So here is some math:
  If a security group has 2000 VM’s spread across and 50 compute nodes, the average node will have 40 VMs. If a new VM is launched in the same security group, each compute node will get a notification and request the security group info that will have 2001 entries for each of the ~40 ports on that node. That’s ~80k records that need to be delivered to 50 compute nodes in a short period of time. The only difference between each port’s list of rules is that the port’s fixed_ips are excluded.

  I suggest approach when there would be only one response for node that
  contains 2000+1 records. The agent would be responsible for taking the
  list of rules for the security group and applying it per port by
  excluding the rule referencing the port’s address. Besides the
  generation of useless information and a lot of work done on neutron-
  server side it will significantly decrease load on oslo.messaging and
  neutron-server. Right now in my env (25 computes; 1500VMs) a security
  groups response could be up to 32Mb for each compute node/ovs-agent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mos/+bug/1432858/+subscriptions