yahoo-eng-team team mailing list archive

[OpenStack Infra <2045950@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/903707
Committed: https://opendev.org/openstack/neutron/commit/6b6abb9698318a0b5db09f0c4d30a47438a94643
Submitter: "Zuul (22348)"
Branch:    master

commit 6b6abb9698318a0b5db09f0c4d30a47438a94643
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Dec 14 15:45:48 2023 +0000

    Improve the SG RPC callback ``security_group_info_for_ports``
    
    This method populates the SG rules in a dictionary. Each SG rule
    inherits the "stateful" value of the SG. Prior to this patch, each
    SG rule was isuing a database call to retrieve the SG register.
    
    In this patch, the SG "stateful" retrieval is done in one database
    query for all SG. That improves the performance of this method
    reducing the database access to only one single call.
    
    This improvement, as commented in the LP bug, affects to
    ML2/LinuxBridge. ML2/OVS agent uses a cached RPC implementation
    that not requires to perform any RPC call/database query.
    
    Closes-Bug: #2045950
    Change-Id: Iafd0419a1d1eeb25d5589edc2570ebf287450957


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045950

Title:
  Security group performance issue for iptables driver due to "stateless
  feature"

Status in neutron:
  Fix Released

Bug description:
  There is a huge performance issue with the security groups when using the iptables implementation:
  If you have a security group with say 500 rules it will take minutes for the RPC server to create the the port configuration.
  You will see this when you restart the neutron-linuxbridge-agent on a compute node with an instance with the security group with a lot of rules.
  In the agent log you will see "Preparing filters for devices" and this will take minutes for a single port when having a significate amount of rules in the security group.

  After some investigation this seems to be cause:

  In the commit below stateful functionality was added for iptables
  implementation:

  https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c

  However there is a huge performance impact, in the following function in 
  neutron/db/securitygroups_rpc_base.py
   def security_group_info_for_ports

  For EACH rule in the security group rule in a a group it will do a database lookup to check what the setting is on the group:
              stateful = self._is_security_group_stateful(context,
                                                          security_group_id)
  Which will call:

      def _is_security_group_stateful(self, context, sg_id):
          return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful

  
  So if you have say 500 rules it will go 500 times(!) to the database to check the exact same property on the group object which absolutely tanks performance.

  I played around with caching the stateful property for the group
  (since it is not even changeable on a security group if there are
  rules present) and the function went from taking multiple minutes to
  about a second.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045950/+subscriptions