yahoo-eng-team team mailing list archive

[Dirk Mueller <1842150@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

when doing something like

openstack port list --project=$uuid

in a project that has many (~500) trunk ports defined, the api request takes a very long
tiem to complete (> 60s) and returns an excessively large json document. 

this is due to

neutron/services/trunk/plugin.py

   def _extend_port_trunk_details(port_res, port_db):
        """Add trunk details to a port."""
        if port_db.trunk_port:
            subports = {
                x.port_id: {'segmentation_id': x.segmentation_id,
                            'segmentation_type': x.segmentation_type,
                            'port_id': x.port_id}
                for x in port_db.trunk_port.sub_ports
            }
            core_plugin = directory.get_plugin()
            ports = core_plugin.get_ports(
                context.get_admin_context(), filters={'id': subports})
            for port in ports:
                subports[port['id']]['mac_address'] = port['mac_address']
            trunk_details = {'trunk_id': port_db.trunk_port.id,
                             'sub_ports': [x for x in subports.values()]}
            port_res['trunk_details'] = trunk_details

        return port_res


which is doing a SQL query (via core_plugin.get_ports()) for each trunk port (!) to return its list of subports in the trunk_details section.

This trunk_details section is not actually shown by the openstack
client, so it does so with no good reason. (it might be in --long
output, didn't check).

It isn't entirely clear to me how to solve this, but I would suggest
that we skip the _extend_port*details callouts for each port type when
"port list" is used and *only* do those in "port show $uuid" type
queries (where the overhead of looking up details on a single port is
neglectible). The fanout of those subqueries (which can in return again
fanout to further sql queries) for listings (especially in projects with
10000+ active ports) is excessivly and cloggs database and neutron
service processes for a long amount of time.

Another way would be to add a flag to skip those to the api and make
sure Nova (which triggers those listings) is not using those detailed
lookups (it entirely ignores the result anyway).

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1842150

Title:
  excessive SQL query fanout on port list with many trunk ports

Status in neutron:
  New

Bug description:
  when doing something like

  openstack port list --project=$uuid

  in a project that has many (~500) trunk ports defined, the api request takes a very long
  tiem to complete (> 60s) and returns an excessively large json document. 

  this is due to

  neutron/services/trunk/plugin.py

     def _extend_port_trunk_details(port_res, port_db):
          """Add trunk details to a port."""
          if port_db.trunk_port:
              subports = {
                  x.port_id: {'segmentation_id': x.segmentation_id,
                              'segmentation_type': x.segmentation_type,
                              'port_id': x.port_id}
                  for x in port_db.trunk_port.sub_ports
              }
              core_plugin = directory.get_plugin()
              ports = core_plugin.get_ports(
                  context.get_admin_context(), filters={'id': subports})
              for port in ports:
                  subports[port['id']]['mac_address'] = port['mac_address']
              trunk_details = {'trunk_id': port_db.trunk_port.id,
                               'sub_ports': [x for x in subports.values()]}
              port_res['trunk_details'] = trunk_details

          return port_res

  
  which is doing a SQL query (via core_plugin.get_ports()) for each trunk port (!) to return its list of subports in the trunk_details section.

  This trunk_details section is not actually shown by the openstack
  client, so it does so with no good reason. (it might be in --long
  output, didn't check).

  It isn't entirely clear to me how to solve this, but I would suggest
  that we skip the _extend_port*details callouts for each port type when
  "port list" is used and *only* do those in "port show $uuid" type
  queries (where the overhead of looking up details on a single port is
  neglectible). The fanout of those subqueries (which can in return
  again fanout to further sql queries) for listings (especially in
  projects with 10000+ active ports) is excessivly and cloggs database
  and neutron service processes for a long amount of time.

  Another way would be to add a flag to skip those to the api and make
  sure Nova (which triggers those listings) is not using those detailed
  lookups (it entirely ignores the result anyway).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1842150/+subscriptions