yahoo-eng-team team mailing list archive

[Robert van Leeuwen <2045950@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

There is a huge performance issue with the security groups when using the iptables implementation:
If you have a security group with say 500 rules it will take minutes for the RPC server to create the the port configuration.
You will see this when you restart the neutron-linuxbridge-agent on a compute node with an instance with the security group with a lot of rules.
In the agent log you will see "Preparing filters for devices" and this will take minutes for a single port when having a significate amount of rules in the security group.

After some investigation this seems to be cause:

In the commit below stateful functionality was added for iptables
implementation:

https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c

However there is a huge performance impact, in the following function in 
neutron/db/securitygroups_rpc_base.py
 def security_group_info_for_ports

For EACH rule in the security group rule in a a group it will do a database lookup to check what the setting is on the group:
            stateful = self._is_security_group_stateful(context,
                                                        security_group_id)
Which will call:

    def _is_security_group_stateful(self, context, sg_id):
        return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful


So if you have say 500 rules it will go 500 times(!) to the database to check the exact same property on the group object which absolutely tanks performance.

I played around with caching the stateful property for the group (since
it is not even changeable on a security group if there are rules
present) and the function went from taking multiple minutes to about a
second.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045950

Title:
  Security group performance issue for iptables driver due to "stateless
  feature"

Status in neutron:
  New

Bug description:
  There is a huge performance issue with the security groups when using the iptables implementation:
  If you have a security group with say 500 rules it will take minutes for the RPC server to create the the port configuration.
  You will see this when you restart the neutron-linuxbridge-agent on a compute node with an instance with the security group with a lot of rules.
  In the agent log you will see "Preparing filters for devices" and this will take minutes for a single port when having a significate amount of rules in the security group.

  After some investigation this seems to be cause:

  In the commit below stateful functionality was added for iptables
  implementation:

  https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c

  However there is a huge performance impact, in the following function in 
  neutron/db/securitygroups_rpc_base.py
   def security_group_info_for_ports

  For EACH rule in the security group rule in a a group it will do a database lookup to check what the setting is on the group:
              stateful = self._is_security_group_stateful(context,
                                                          security_group_id)
  Which will call:

      def _is_security_group_stateful(self, context, sg_id):
          return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful

  
  So if you have say 500 rules it will go 500 times(!) to the database to check the exact same property on the group object which absolutely tanks performance.

  I played around with caching the stateful property for the group
  (since it is not even changeable on a security group if there are
  rules present) and the function went from taking multiple minutes to
  about a second.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045950/+subscriptions