yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87534
[Bug 1948891] [NEW] [ovn] Using ovsdb-client for MAC_Binding could theoretically block indefinitely
Public bug reported:
As a workaround for saving massive amounts of memory by not monitoring
the MAC_Binding table, we recently switched to using ovsdb-client for
deleting FIP MAC_Binding entries. ovsdb-client will open a new
connection every time it is called. There are some cases where it might
block the worker longer than we would expect.
1) If ovsdb-server is busy and the connection takes a long time, ovsdb-
client will wait up to 2 minutes before giving up. (tested by just
setting iptables to drop packets to ovsdb-server port)
2) If ovsdb-server connects, but just doesn't respond (maybe a cable is
cut), ovsdb-client will wait forever for the response. (tested by just
having nc listen on the ovsdb-server port)
A quick workaround would be to pass --timeout to ovsdb-client, possibly
with the ovsdb_connection_timeout. There is also a daemon mode for
ovsdb-client.
I also have a patch [1] waiting in python-ovs that will allow us to use
the existing Idl connection to craft arbitrary transact operations like
we pass to ovsdb-client. This will lose the overhead of making a new
connection with each FIP delete.
[1]
https://patchwork.ozlabs.org/project/openvswitch/patch/20211013224023.69938-1-twilson@xxxxxxxxxx/
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1948891
Title:
[ovn] Using ovsdb-client for MAC_Binding could theoretically block
indefinitely
Status in neutron:
New
Bug description:
As a workaround for saving massive amounts of memory by not monitoring
the MAC_Binding table, we recently switched to using ovsdb-client for
deleting FIP MAC_Binding entries. ovsdb-client will open a new
connection every time it is called. There are some cases where it
might block the worker longer than we would expect.
1) If ovsdb-server is busy and the connection takes a long time,
ovsdb-client will wait up to 2 minutes before giving up. (tested by
just setting iptables to drop packets to ovsdb-server port)
2) If ovsdb-server connects, but just doesn't respond (maybe a cable
is cut), ovsdb-client will wait forever for the response. (tested by
just having nc listen on the ovsdb-server port)
A quick workaround would be to pass --timeout to ovsdb-client,
possibly with the ovsdb_connection_timeout. There is also a daemon
mode for ovsdb-client.
I also have a patch [1] waiting in python-ovs that will allow us to
use the existing Idl connection to craft arbitrary transact operations
like we pass to ovsdb-client. This will lose the overhead of making a
new connection with each FIP delete.
[1]
https://patchwork.ozlabs.org/project/openvswitch/patch/20211013224023.69938-1-twilson@xxxxxxxxxx/
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1948891/+subscriptions
Follow ups