yahoo-eng-team team mailing list archive

[OpenStack Infra <2083682@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/932041
Committed: https://opendev.org/openstack/neutron/commit/adbc3e23b7d2251cc7de088e2a757674a41c2f6a
Submitter: "Zuul (22348)"
Branch:    master

commit adbc3e23b7d2251cc7de088e2a757674a41c2f6a
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Oct 10 08:49:44 2024 +0000

    Optimize the SG rule retrieval
    
    There are some operations where the SG DB object can be used instead of
    the SG OVO. That saves conversion time, including the conversion of the
    SG rule OVOs, that are child resources of the SG OVO.
    
    This optimization applies to the following methods:
    * SecurityGroupDbMixin.get_security_groups
    * SecurityGroupDbMixin.update_security_group (partially)
    
    The Nova query to retrieve the SG list in the "server list" command,
    has been benchmarked. The testing environment had a single SG with
    250 SG rules. Call:
      "GET /networking/v2.0/security-groups?id=81f64aa4-2cea-46db-8fea-cd944f106aab
         &fields=id&fields=name HTTP/1.1"
    
    * Without this patch: around 1.25 seconds
    * With this patch: around 0.025 second (50x improvement).
    
    Closes-bug: #2083682
    Change-Id: Ibd032ea77c5bfbc1fa80b3b3ee9ba7d5c36bb1bc


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2083682

Title:
  Slowness of security groups list API

Status in neutron:
  Fix Released

Bug description:
  Input:
  - OpenStack cluster of 2024.1 release
  - Total number of VMs = 9k
  - Total number of security groups = 6.4k
  - Total number of security groups rules = 122k

  Problem description:

  Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit).
  OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`.
  The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups.

  Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 :
  - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete.
  - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s.

  https://review.opendev.org/c/openstack/neutron/+/929967 is applied to
  neutron server's. It improved the case, but has not resolved it.

  Additional info: Nova uses python-neutronclient library, which in my
  experiments behaves quicker than openstacksdk.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2083682/+subscriptions