yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2065701] [NEW] set_gateway_mtu fails if one of the lrp is deleted while it's running

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2065701@xxxxxxxxxxxxxxxxxx>
Date: Tue, 14 May 2024 15:50:53 -0000
Reply-to: Bug 2065701 <2065701@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Seen in some downstream jobs running with antelope content and
ovn_emit_need_to_frag = true, tests fails randomly while running
add_router_interface or remove_router_interface calls like:-

ft9.1: setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 206, in setUpClass
    raise value.with_traceback(trace)
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 191, in setUpClass
    cls.setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/api/compute/servers/test_server_actions.py", line 827, in setup_credentials
    super(ServerActionsV293TestJSON, cls).setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/api/compute/base.py", line 75, in setup_credentials
    super(BaseV2ComputeTest, cls).setup_credentials()
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 419, in setup_credentials
    manager = cls.get_client_manager(
  File "/usr/lib/python3.9/site-packages/tempest/test.py", line 764, in get_client_manager
    creds = getattr(cred_provider, credentials_method)()
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 473, in get_primary_creds
    return self.get_project_member_creds()
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 508, in get_project_member_creds
    return self.get_credentials(['member'], scope='project')
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 459, in get_credentials
    network, subnet, router = self._create_network_resources(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 319, in _create_network_resources
    self._add_router_interface(router['id'], subnet['id'])
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 383, in _add_router_interface
    self.routers_admin_client.add_router_interface(router_id,
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/routers_client.py", line 72, in add_router_interface
    return self.update_resource(uri, kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/base.py", line 77, in update_resource
    resp, body = self.put(req_uri, req_post_data)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 372, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 922, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.

Fails with Internal server error:-
192.168.16.2 - - [13/May/2024:15:09:44 +0000] "PUT /v2.0/routers/a76e7487-6fcb-4134-8de4-f08a3eca33cb/add_router_interface HTTP/1.1" 500 150 "-" "python-urllib3/1.26.5"

2024-05-13T15:10:15.685665853+00:00 stdout F 2024-05-13 15:10:15.684 16 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-8b7c93b4-c0d8-4963-b711-cab362fcada4 - - - - - -] Running txn n=1 command(idx=20): LrpSetOptionsCommand(_result=None, entity=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5, options={}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89^[[00m
2024-05-13T15:10:15.686489887+00:00 stdout F 2024-05-13 15:10:15.685 16 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d211c9ca-9694-4d69-9e57-b330a231c30f 9486b7f231584e7ab8f143c840957bf3 6bcd5aaa226d4e69b9e20287ea2d3b11 - - default default] Traceback (most recent call last):
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2024-05-13T15:10:15.686489887+00:00 stdout F     txn.results.put(txn.do_commit())
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
2024-05-13T15:10:15.686489887+00:00 stdout F     command.run_idl(txn)
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 349, in run_idl
2024-05-13T15:10:15.686489887+00:00 stdout F     entity = self.api.lookup(self.table, self.entity)
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 183, in lookup
2024-05-13T15:10:15.686489887+00:00 stdout F     return self._lookup(table, record)
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 234, in _lookup
2024-05-13T15:10:15.686489887+00:00 stdout F     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2024-05-13T15:10:15.686489887+00:00 stdout F     raise RowNotFound(table=table, col=column, match=match)
2024-05-13T15:10:15.686489887+00:00 stdout F ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5
2024-05-13T15:10:15.686489887+00:00 stdout F ^[[00m


This happens as router port is removed while the LrpSetOptionsCommand runs for that router port, the transaction assumes the port exists[1]. Would need to add if-exists=true in LrpSetOptionsCommand/lrp_set_options to handle such cases.

[1]
https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2084-L2088

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2065701

Title:
  set_gateway_mtu fails if one of the lrp is deleted while it's running

Status in neutron:
  New

Bug description:
  Seen in some downstream jobs running with antelope content and
  ovn_emit_need_to_frag = true, tests fails randomly while running
  add_router_interface or remove_router_interface calls like:-

  ft9.1: setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON)testtools.testresult.real._StringException: Traceback (most recent call last):
    File "/usr/lib/python3.9/site-packages/tempest/test.py", line 206, in setUpClass
      raise value.with_traceback(trace)
    File "/usr/lib/python3.9/site-packages/tempest/test.py", line 191, in setUpClass
      cls.setup_credentials()
    File "/usr/lib/python3.9/site-packages/tempest/api/compute/servers/test_server_actions.py", line 827, in setup_credentials
      super(ServerActionsV293TestJSON, cls).setup_credentials()
    File "/usr/lib/python3.9/site-packages/tempest/api/compute/base.py", line 75, in setup_credentials
      super(BaseV2ComputeTest, cls).setup_credentials()
    File "/usr/lib/python3.9/site-packages/tempest/test.py", line 419, in setup_credentials
      manager = cls.get_client_manager(
    File "/usr/lib/python3.9/site-packages/tempest/test.py", line 764, in get_client_manager
      creds = getattr(cred_provider, credentials_method)()
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 473, in get_primary_creds
      return self.get_project_member_creds()
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 508, in get_project_member_creds
      return self.get_credentials(['member'], scope='project')
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 459, in get_credentials
      network, subnet, router = self._create_network_resources(
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 319, in _create_network_resources
      self._add_router_interface(router['id'], subnet['id'])
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/dynamic_creds.py", line 383, in _add_router_interface
      self.routers_admin_client.add_router_interface(router_id,
    File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/routers_client.py", line 72, in add_router_interface
      return self.update_resource(uri, kwargs)
    File "/usr/lib/python3.9/site-packages/tempest/lib/services/network/base.py", line 77, in update_resource
      resp, body = self.put(req_uri, req_post_data)
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 372, in put
      return self.request('PUT', url, extra_headers, headers, body, chunked)
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 742, in request
      self._error_checker(resp, resp_body)
    File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 922, in _error_checker
      raise exceptions.ServerFault(resp_body, resp=resp,
  tempest.lib.exceptions.ServerFault: Got server fault
  Details: Request Failed: internal server error while processing your request.

  Fails with Internal server error:-
  192.168.16.2 - - [13/May/2024:15:09:44 +0000] "PUT /v2.0/routers/a76e7487-6fcb-4134-8de4-f08a3eca33cb/add_router_interface HTTP/1.1" 500 150 "-" "python-urllib3/1.26.5"

  2024-05-13T15:10:15.685665853+00:00 stdout F 2024-05-13 15:10:15.684 16 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-8b7c93b4-c0d8-4963-b711-cab362fcada4 - - - - - -] Running txn n=1 command(idx=20): LrpSetOptionsCommand(_result=None, entity=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5, options={}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89^[[00m
  2024-05-13T15:10:15.686489887+00:00 stdout F 2024-05-13 15:10:15.685 16 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d211c9ca-9694-4d69-9e57-b330a231c30f 9486b7f231584e7ab8f143c840957bf3 6bcd5aaa226d4e69b9e20287ea2d3b11 - - default default] Traceback (most recent call last):
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
  2024-05-13T15:10:15.686489887+00:00 stdout F     txn.results.put(txn.do_commit())
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
  2024-05-13T15:10:15.686489887+00:00 stdout F     command.run_idl(txn)
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 349, in run_idl
  2024-05-13T15:10:15.686489887+00:00 stdout F     entity = self.api.lookup(self.table, self.entity)
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 183, in lookup
  2024-05-13T15:10:15.686489887+00:00 stdout F     return self._lookup(table, record)
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 234, in _lookup
  2024-05-13T15:10:15.686489887+00:00 stdout F     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2024-05-13T15:10:15.686489887+00:00 stdout F   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
  2024-05-13T15:10:15.686489887+00:00 stdout F     raise RowNotFound(table=table, col=column, match=match)
  2024-05-13T15:10:15.686489887+00:00 stdout F ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-9309bd6d-2e46-404e-8c47-fda1adc15cc5
  2024-05-13T15:10:15.686489887+00:00 stdout F ^[[00m

  
  This happens as router port is removed while the LrpSetOptionsCommand runs for that router port, the transaction assumes the port exists[1]. Would need to add if-exists=true in LrpSetOptionsCommand/lrp_set_options to handle such cases.

  [1]
  https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2084-L2088

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2065701/+subscriptions
Follow ups

[Bug 2065701] Re: set_gateway_mtu fails if one of the lrp is deleted while it's running
From: OpenStack Infra, 2024-05-23