yahoo-eng-team team mailing list archive

[Terry Wilson <1948891@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

As a workaround for saving massive amounts of memory by not monitoring
the MAC_Binding table, we recently switched to using ovsdb-client for
deleting FIP MAC_Binding entries. ovsdb-client will open a new
connection every time it is called. There are some cases where it might
block the worker longer than we would expect.

1) If ovsdb-server is busy and the connection takes a long time, ovsdb-
client will wait up to 2 minutes before giving up. (tested by just
setting iptables to drop packets to ovsdb-server port)

2) If ovsdb-server connects, but just doesn't respond (maybe a cable is
cut), ovsdb-client will wait forever for the response. (tested by just
having nc listen on the ovsdb-server port)

A quick workaround would be to pass --timeout to ovsdb-client, possibly
with the ovsdb_connection_timeout. There is also a daemon mode for
ovsdb-client.

I also have a patch [1] waiting in python-ovs that will allow us to use
the existing Idl connection to craft arbitrary transact operations like
we pass to ovsdb-client. This will lose the overhead of making a new
connection with each FIP delete.

[1]
https://patchwork.ozlabs.org/project/openvswitch/patch/20211013224023.69938-1-twilson@xxxxxxxxxx/

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1948891

Title:
  [ovn] Using ovsdb-client for MAC_Binding could theoretically block
  indefinitely

Status in neutron:
  New

Bug description:
  As a workaround for saving massive amounts of memory by not monitoring
  the MAC_Binding table, we recently switched to using ovsdb-client for
  deleting FIP MAC_Binding entries. ovsdb-client will open a new
  connection every time it is called. There are some cases where it
  might block the worker longer than we would expect.

  1) If ovsdb-server is busy and the connection takes a long time,
  ovsdb-client will wait up to 2 minutes before giving up. (tested by
  just setting iptables to drop packets to ovsdb-server port)

  2) If ovsdb-server connects, but just doesn't respond (maybe a cable
  is cut), ovsdb-client will wait forever for the response. (tested by
  just having nc listen on the ovsdb-server port)

  A quick workaround would be to pass --timeout to ovsdb-client,
  possibly with the ovsdb_connection_timeout. There is also a daemon
  mode for ovsdb-client.

  I also have a patch [1] waiting in python-ovs that will allow us to
  use the existing Idl connection to craft arbitrary transact operations
  like we pass to ovsdb-client. This will lose the overhead of making a
  new connection with each FIP delete.

  [1]
  https://patchwork.ozlabs.org/project/openvswitch/patch/20211013224023.69938-1-twilson@xxxxxxxxxx/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1948891/+subscriptions