yahoo-eng-team team mailing list archive

[Andriy Kurilin <2083682@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Input:
- OpenStack cluster of 2024.1 release
- Total number of VMs = 9k
- Total number of security groups = 6.4k
- Total number of security groups rules = 122k

Problem description:

Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit).
OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`.
The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups.

Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 :
- retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete.
- retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s.

https://review.opendev.org/c/openstack/neutron/+/929967 is applied to
neutron server's. It improved the case, but has not resolved it.

Additional info: Nova uses python-neutronclient library, which in my
experiments behaves quicker than openstacksdk.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  Input:
  - OpenStack cluster of 2024.1 release
  - Total number of VMs = 9k
  - Total number of security groups = 6.4k
  - Total number of security groups rules = 122k
  
  Problem description:
  
  Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit).
  OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`.
  The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups.
  
  Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 :
- - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete. 
+ - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete.
  - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s.
  
  https://review.opendev.org/c/openstack/neutron/+/929967 is applied to
  neutron server's. It improved the case, but has not resolved it.
+ 
+ Additional info: Nova uses python-neutronclient library, which in my
+ experiments behaves quicker than openstacksdk.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2083682

Title:
  Slowness of security groups list API

Status in neutron:
  New

Bug description:
  Input:
  - OpenStack cluster of 2024.1 release
  - Total number of VMs = 9k
  - Total number of security groups = 6.4k
  - Total number of security groups rules = 122k

  Problem description:

  Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit).
  OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`.
  The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups.

  Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 :
  - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete.
  - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s.

  https://review.opendev.org/c/openstack/neutron/+/929967 is applied to
  neutron server's. It improved the case, but has not resolved it.

  Additional info: Nova uses python-neutronclient library, which in my
  experiments behaves quicker than openstacksdk.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2083682/+subscriptions