yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1914394] [NEW] [OVN] RowNotFound exception while waiting for Chassis metadata networks

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lucas Alvares Gomes <1914394@xxxxxxxxxxxxxxxxxx>
Date: Wed, 03 Feb 2021 10:57:14 -0000
Reply-to: Bug 1914394 <1914394@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In the set_port_status_up() the OVN driver tries waiting for the
metadata to be provisioned (15 seconds) prior to sending the event to
Nova indicating that everything is done (network-vif-plugged). But there
could be a race condition while trying to get that information which
results in a RowNotFound being raise in the waiting loop.

Once that happens, the exception is bubbled up and the OVN driver end up
not sending the event to Nova and the instance will fail to deploy (it
will be stuck in BUILD state until it times out).

Here's a traceback from neutron server (q-svc) when it happens:

Jan 28 10:59:55.066255 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] OVN reports status up for port: 2694a4c0-4ff0-414d-b780-b74da2c91197
Jan 28 10:59:55.068736 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42564) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
Jan 28 10:59:55.110806 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: INFO neutron.wsgi [req-177263b2-82ba-4f7e-84c9-d5b86115b3e7 req-f545cbf1-c1f0-4bc2-9d7e-d0420a9b43ee service neutron] 10.0.1.208,10.0.1.179 "GET /v2.0/floatingips?fixed_ip_address=10.1.0.12&port_id=2694a4c0-4ff0-414d-b780-b74da2c91197 HTTP/1.1" status: 200  len: 217 time: 0.0412221
Jan 28 10:59:55.134847 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42568) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] Unexpected exception in notify_loop: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event Traceback (most recent call last):
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     match.run(event, row, updates)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 392, in run
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self.driver.set_port_status_up(row.name)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 982, in set_port_status_up
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._wait_for_metadata_provisioned_if_needed(port_id)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1116, in _wait_for_metadata_provisioned_if_needed
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     n_utils.wait_until_true(
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/common/utils.py", line 703, in wait_until_true
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     while not predicate():
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in <lambda>
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._sb_ovn.get_chassis_metadata_networks(chassis),
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 795, in get_chassis_metadata_networks
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     chassis = self.lookup('Chassis', chassis_name)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     return self._lookup(table, record)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     row = idlutils.row_by_value(self, rl.table, rl.column, record)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     raise RowNotFound(table=table, col=column, match=match)
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event

** Affects: neutron
     Importance: High
     Assignee: Lucas Alvares Gomes (lucasagomes)
         Status: Confirmed

** Changed in: neutron
   Importance: Undecided => High

** Changed in: neutron
     Assignee: (unassigned) => Lucas Alvares Gomes (lucasagomes)

** Changed in: neutron
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1914394

Title:
  [OVN] RowNotFound exception while waiting for Chassis metadata
  networks

Status in neutron:
  Confirmed

Bug description:
  In the set_port_status_up() the OVN driver tries waiting for the
  metadata to be provisioned (15 seconds) prior to sending the event to
  Nova indicating that everything is done (network-vif-plugged). But
  there could be a race condition while trying to get that information
  which results in a RowNotFound being raise in the waiting loop.

  Once that happens, the exception is bubbled up and the OVN driver end
  up not sending the event to Nova and the instance will fail to deploy
  (it will be stuck in BUILD state until it times out).

  Here's a traceback from neutron server (q-svc) when it happens:

  Jan 28 10:59:55.066255 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] OVN reports status up for port: 2694a4c0-4ff0-414d-b780-b74da2c91197
  Jan 28 10:59:55.068736 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42564) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
  Jan 28 10:59:55.110806 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: INFO neutron.wsgi [req-177263b2-82ba-4f7e-84c9-d5b86115b3e7 req-f545cbf1-c1f0-4bc2-9d7e-d0420a9b43ee service neutron] 10.0.1.208,10.0.1.179 "GET /v2.0/floatingips?fixed_ip_address=10.1.0.12&port_id=2694a4c0-4ff0-414d-b780-b74da2c91197 HTTP/1.1" status: 200  len: 217 time: 0.0412221
  Jan 28 10:59:55.134847 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42568) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] Unexpected exception in notify_loop: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event Traceback (most recent call last):
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     match.run(event, row, updates)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 392, in run
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self.driver.set_port_status_up(row.name)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 982, in set_port_status_up
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._wait_for_metadata_provisioned_if_needed(port_id)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1116, in _wait_for_metadata_provisioned_if_needed
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     n_utils.wait_until_true(
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/common/utils.py", line 703, in wait_until_true
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     while not predicate():
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in <lambda>
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._sb_ovn.get_chassis_metadata_networks(chassis),
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 795, in get_chassis_metadata_networks
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     chassis = self.lookup('Chassis', chassis_name)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     return self._lookup(table, record)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     raise RowNotFound(table=table, col=column, match=match)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1914394/+subscriptions
Follow ups

[Bug 1914394] Re: [OVN] RowNotFound exception while waiting for Chassis metadata networks
From: Brian Haley, 2022-11-08