yahoo-eng-team team mailing list archive

[OpenStack Infra <1946318@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/812805
Committed: https://opendev.org/openstack/neutron/commit/f6c35527698119ee6f73a6a3613c9beebb563840
Submitter: "Zuul (22348)"
Branch:    master

commit f6c35527698119ee6f73a6a3613c9beebb563840
Author: Daniel Alvarez Sanchez <dalvarez@xxxxxxxxxx>
Date:   Thu Oct 7 09:23:21 2021 +0200

    [ovn] Stop monitoring the SB MAC_Binding table to reduce mem footprint
    
    The MAC_Binding table in the SB database may grow indefinitely (due
    to a lack of an aging mechanism of its entries) and eventually
    lead to OOM killers for neutron-server which maintains an in-memory
    copy of the database.
    
    In order to stop monitoring this table, this patch is invoking
    the ovsdb-client tool to remove the entries associated to
    Floating IPs that have just been detached. The execution of this
    tool is really fast as it will just invoke a JSON-RPC transact command
    which doesn't require downloading the database contents.
    
    In a scale test, the memory consumption of neutron-server dropped
    from 75GB to 7GB with this patch.
    
    Closes-Bug: #1946318
    
    Signed-off-by: Daniel Alvarez Sanchez <dalvarez@xxxxxxxxxx>
    Change-Id: Id84bf17953527c415d611bfc198038fb6f811de3


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1946318

Title:
  [ovn] Memory consumption grows over time due to MAC_Binding entries in
  SB database

Status in neutron:
  Fix Released

Bug description:
  MAC_Binding entries are used in OVN as a mechanism to learn MAC
  addresses on logical ports and avoid sending ARP requests to the
  network.

  There is no aging mechanism for these entries [0] and the table can
  grow indefinitely. In environments with for example large (eg. /16)
  external networks; OVN may learn a considerable amount of addresses
  growing the size of the db a lot.

  Today, Neutron monitors this table to workaround the lack of aging mechanism and remove the MAC_Binding entries associated to Floating IPs and each neutron-server worker will keep an in-memory copy of such table increasing its memory footprint to several Gigabytes, eventually leading to OOM killers.
   

  [0] https://mail.openvswitch.org/pipermail/ovs-
  discuss/2019-June/048936.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1946318/+subscriptions