yahoo-eng-team team mailing list archive

[OpenStack Infra <1976439@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/845105
Committed: https://opendev.org/openstack/neutron/commit/ad3f7a8b7d69c2f5612d04b08f7250a21c4488f9
Submitter: "Zuul (22348)"
Branch:    master

commit ad3f7a8b7d69c2f5612d04b08f7250a21c4488f9
Author: liujinxin <hbbfxyxxgcljx@xxxxxxx>
Date:   Wed Jun 8 16:10:55 2022 +0800

    Clean up db residual record from dvr port
    
    Delete the DVR port record on the node when the router is deleted from
    the node. This patch fixes an issue where the DB actions have no context
    writer which will actually update nothing duing the following
    `delete_distributed_port_binding_if_stale`.
    As well as fixing that when the dvr router was deleted from a node,
    only the ml2_distributed_port_bindings were cleaned up and not the
    ml2_port_binding_levels records for the dvr port.
    
    Remove the last VM under one router in one host, the
    ml2_distributed_port_bindings and ml2_port_binding_levels will remain
    the record.  So if VMs under one router had spread on many hosts,
    and router (router ports) still exists, even there is only one VM under it,
    the binding entries will still be equal to amount of hosts where this
    router resided before. The result is a large amount of redundant data
    in the database, even if some nodes are no longer in the cluster,
    there are still records for that node.
    
    Closes-Bug: #1976439
    Change-Id: I320ac2306e0f25ff933d8271203e192486062d61


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1976439

Title:
  The database ml2_port_binding_levels and ml2_distributed_port_bindings
  tables have a lot of redundant data

Status in neutron:
  Fix Released

Bug description:
  ENV: stable/victoria

  In a large scale cloud deployment and we have enabled DVR and have
  some huge virtual routers.When we remove some nodes from the cluster,
  the database ml2_port_binding_levels and ml2_distributed_port_bindings
  tables have a lot of redundant data，when restart neutron agent,
  especially l3 agents, neutron server side will trigger too many slow
  DB query. And this will cause the agent restart time to be too long to
  operate.

  As well, if a cluster has 1000 compute nodes and 5000 routers, after
  running for a long time, the vm under the router is migrated, created
  and deleted many times, which may be scheduled to hundreds of nodes,
  and each time the qrouter is deleted from the node does not delete the
  ml2_port_binding_levels and ml2_distributed_port_bindings table
  entries, finally even though there are only a few vm under this
  router, there are a large number of residual table entries, resulting
  in higher and higher time consumption when neutron-server queries the
  database. Not only the sync_routers interface is getting more and more
  time consuming, but also the port list is getting more and more time
  consuming because the port model contains subqueries for that table.

  Translated with www.DeepL.com/Translator (free version)

  For example, the following xxx nodes have been removed from the
  cluster, and there are no more qrouter on these nodes


  ```
  MariaDB [neutron]> select count(*) from ml2_port_binding_levels;
  +----------+
  | count(*) |
  +----------+
  |   163986 |
  +----------+
  MariaDB [neutron]> select count(*) from ml2_distributed_port_bindings;
  +----------+
  | count(*) |
  +----------+
  |   119797 |
  +----------+
  MariaDB [neutron]> select count(*) from ml2_distributed_port_bindings where host like("%xxx%");
  +----------+
  | count(*) |
  +----------+
  |    78920 |
  +----------+
  MariaDB [neutron]> select count(*) from ml2_port_binding_levels where host like("%xxx%");
  +----------+
  | count(*) |
  +----------+
  |    79482 |
  +----------+

  MariaDB [neutron]> select count(distinct host) from ml2_port_binding_levels where host like("%xxx%");
  +----------------------+
  | count(distinct host) |
  +----------------------+
  |                  385 |
  +----------------------+

  MariaDB [neutron]> select count(*) from routers;
  +----------+
  | count(*) |
  +----------+
  |     7543 |
  +----------+
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1976439/+subscriptions