yahoo-eng-team team mailing list archive

[Brian Haley <1914394@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1914394

Title:
  [OVN] RowNotFound exception while waiting for Chassis metadata
  networks

Status in neutron:
  Fix Released

Bug description:
  In the set_port_status_up() the OVN driver tries waiting for the
  metadata to be provisioned (15 seconds) prior to sending the event to
  Nova indicating that everything is done (network-vif-plugged). But
  there could be a race condition while trying to get that information
  which results in a RowNotFound being raise in the waiting loop.

  Once that happens, the exception is bubbled up and the OVN driver end
  up not sending the event to Nova and the instance will fail to deploy
  (it will be stuck in BUILD state until it times out).

  Here's a traceback from neutron server (q-svc) when it happens:

  Jan 28 10:59:55.066255 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] OVN reports status up for port: 2694a4c0-4ff0-414d-b780-b74da2c91197
  Jan 28 10:59:55.068736 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42564) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
  Jan 28 10:59:55.110806 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: INFO neutron.wsgi [req-177263b2-82ba-4f7e-84c9-d5b86115b3e7 req-f545cbf1-c1f0-4bc2-9d7e-d0420a9b43ee service neutron] 10.0.1.208,10.0.1.179 "GET /v2.0/floatingips?fixed_ip_address=10.1.0.12&port_id=2694a4c0-4ff0-414d-b780-b74da2c91197 HTTP/1.1" status: 200  len: 217 time: 0.0412221
  Jan 28 10:59:55.134847 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42568) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}}
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] Unexpected exception in notify_loop: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event Traceback (most recent call last):
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     match.run(event, row, updates)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 392, in run
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self.driver.set_port_status_up(row.name)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 982, in set_port_status_up
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._wait_for_metadata_provisioned_if_needed(port_id)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1116, in _wait_for_metadata_provisioned_if_needed
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     n_utils.wait_until_true(
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/common/utils.py", line 703, in wait_until_true
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     while not predicate():
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in <lambda>
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     self._sb_ovn.get_chassis_metadata_networks(chassis),
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 795, in get_chassis_metadata_networks
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     chassis = self.lookup('Chassis', chassis_name)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     return self._lookup(table, record)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event     raise RowNotFound(table=table, col=column, match=match)
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0
  Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1914394/+subscriptions