yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2087824] Re: random router related test failures since wsgi switch

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2087824@xxxxxxxxxxxxxxxxxx>
Date: Tue, 19 Nov 2024 13:32:45 -0000
Reply-to: Bug 2087824 <2087824@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/934652
Committed: https://opendev.org/openstack/neutron/commit/5828c43c904d938644144bd204a262bd36c3f516
Submitter: "Zuul (22348)"
Branch:    master

commit 5828c43c904d938644144bd204a262bd36c3f516
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Fri Nov 8 22:32:24 2024 +0000

    [OVN] Do not update a deleted Logical_Switch_Port
    
    The method``OVNClient.update_port`` can be called from
    ``update_port_postcommit``; at this point, other concurrent call
    can have deleted the Logical_Switch_Port. This check prevents from
    failing the API call that was issued first.
    
    Closes-Bug: #2087824
    Change-Id: I96152f86595eefa375067ade5a303ffb35c80348


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2087824

Title:
  random router related test failures since wsgi switch

Status in neutron:
  Fix Released

Bug description:
  With couple of recent fixes since wsgi switch we still seeing some
  random failures, this to track fixes of these unfixed issues.

  Example failures:-
  1) test_router_interface_port_update_with_fixed_ip
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eb6/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master/eb6d44f/testr_results.html
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/api/network/base.py", line 227, in delete_router
      cls.routers_client.delete_router(router['id'])
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 52, in delete_router
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 359, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 872, in _error_checker
      raise exceptions.NotFound(resp_body, resp=resp)
  tempest.lib.exceptions.NotFound: Object not found
  Details: {'type': 'RouterNotFound', 'message': 'Router 4506dff7-0d22-40a6-a2ca-e28dbabcb2bc could not be found', 'detail': ''}

  
  2024-11-01 03:32:15,277 113867 WARNING  [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='158.69.64.156', port=443): Read timed out. (read timeout=90)")': /networking/v2.0/routers/4506dff7-0d22-40a6-a2ca-e28dbabcb2bc
  2024-11-01 03:32:15,318 113867 INFO     [tempest.lib.common.rest_client] Request (RoutersTest:_run_cleanups): 404 DELETE https://158.69.64.156/networking/v2.0/routers/4506dff7-0d22-40a6-a2ca-e28dbabcb2bc 90.100s
  2024-11-01 03:32:15,318 113867 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Fri, 01 Nov 2024 03:32:15 GMT', 'server': 'Apache/2.4.52 (Ubuntu)', 'content-type': 'application/json', 'content-length': '135', 'x-openstack-request-id': 'req-55ff8f43-80b7-429b-9673-7068e16ae83c', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.64.156/networking/v2.0/routers/4506dff7-0d22-40a6-a2ca-e28dbabcb2bc'}
          Body: b'{"NeutronError": {"type": "RouterNotFound", "message": "Router 4506dff7-0d22-40a6-a2ca-e28dbabcb2bc could not be found", "detail": ""}}'
  2024-11-01 03:32:15,527 113867 INFO     [tempest.lib.common.rest_client] Request (RoutersTest:_run_cleanups): 204 DELETE https://158.69.64.156/networking/v2.0/subnets/fcfd7df3-1964-48cf-9286-e91dd1059dd0 0.206s

  
  2) test_dhcp_stateful_router
  https://4d23b6e5972044413c22-8c13a3592446d4bae7da2f9bb43833cd.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-full-multinode-ovs-master/7955bb2/testr_results.html
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/api/network/test_dhcp_ipv6.py", line 420, in test_dhcp_stateful_router
      self._clean_network()
    File "/opt/stack/tempest/tempest/api/network/test_dhcp_ipv6.py", line 75, in _clean_network
      self.routers_client.remove_router_interface(port['device_id'],
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 82, in remove_router_interface
      return self.update_resource(uri, kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 77, in update_resource
      resp, body = self.put(req_uri, req_post_data)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 392, in put
      return self.request('PUT', url, extra_headers, headers, body, chunked)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 872, in _error_checker
      raise exceptions.NotFound(resp_body, resp=resp)
  tempest.lib.exceptions.NotFound: Object not found
  Details: {'type': 'RouterInterfaceNotFound', 'message': 'Router 250874bf-6b34-41fd-a864-6482717a2ef8 does not have an interface with id 1e6abe6d-e30c-4f70-aa91-88d8f9340d2d', 'detail': ''}

  2024-10-30 03:34:33,221 117760 WARNING  [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='200.225.47.26', port=443): Read timed out. (read timeout=90)")': /networking/v2.0/routers/250874bf-6b34-41fd-a864-6482717a2ef8/remove_router_interface
  2024-10-30 03:34:33,682 117760 INFO     [tempest.lib.common.rest_client] Request (NetworksTestDHCPv6:test_dhcp_stateful_router): 404 PUT https://200.225.47.26/networking/v2.0/routers/250874bf-6b34-41fd-a864-6482717a2ef8/remove_router_interface 90.509s
  2024-10-30 03:34:33,682 117760 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"port_id": "1e6abe6d-e30c-4f70-aa91-88d8f9340d2d"}
      Response - Headers: {'date': 'Wed, 30 Oct 2024 03:34:33 GMT', 'server': 'Apache/2.4.52 (Ubuntu)', 'content-type': 'application/json', 'content-length': '197', 'x-openstack-request-id': 'req-80a651fe-896f-4622-9d14-f70905e9d949', 'connection': 'close', 'status': '404', 'content-location': 'https://200.225.47.26/networking/v2.0/routers/250874bf-6b34-41fd-a864-6482717a2ef8/remove_router_interface'}
          Body: b'{"NeutronError": {"type": "RouterInterfaceNotFound", "message": "Router 250874bf-6b34-41fd-a864-6482717a2ef8 does not have an interface with id 1e6abe6d-e30c-4f70-aa91-88d8f9340d2d", "detail": ""}}'

  
  3) test_router_remove_interface_in_use_returns_409
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a79/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-mariadb-full/a79c6c7/testr_results.html

  raceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 82, in remove_router_interface
      return self.update_resource(uri, kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 77, in update_resource
      resp, body = self.put(req_uri, req_post_data)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 392, in put
      return self.request('PUT', url, extra_headers, headers, body, chunked)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 872, in _error_checker
      raise exceptions.NotFound(resp_body, resp=resp)
  tempest.lib.exceptions.NotFound: Object not found
  Details: {'type': 'RouterInterfaceNotFoundForSubnet', 'message': 'Router 4baf4e53-30d2-498e-b42c-48af5a1a01ff has no interface on subnet bff49c46-192e-4723-9fea-07f3048ec5dd', 'detail': ''}

  2024-11-11 03:05:46,123 73254 WARNING  [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='158.69.68.177', port=443): Read timed out. (read timeout=90)")': /networking/v2.0/routers/4baf4e53-30d2-498e-b42c-48af5a1a01ff/remove_router_interface
  2024-11-11 03:05:46,448 73254 INFO     [tempest.lib.common.rest_client] Request (RoutersNegativeTest:_run_cleanups): 404 PUT https://158.69.68.177/networking/v2.0/routers/4baf4e53-30d2-498e-b42c-48af5a1a01ff/remove_router_interface 90.346s
  2024-11-11 03:05:46,449 73254 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"subnet_id": "bff49c46-192e-4723-9fea-07f3048ec5dd"}
      Response - Headers: {'date': 'Mon, 11 Nov 2024 03:05:46 GMT', 'server': 'Apache/2.4.52 (Ubuntu)', 'content-type': 'application/json', 'content-length': '198', 'x-openstack-request-id': 'req-d4392599-3d5c-4ecf-aeca-094d9141cad0', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.68.177/networking/v2.0/routers/4baf4e53-30d2-498e-b42c-48af5a1a01ff/remove_router_interface'}
          Body: b'{"NeutronError": {"type": "RouterInterfaceNotFoundForSubnet", "message": "Router 4baf4e53-30d2-498e-b42c-48af5a1a01ff has no interface on subnet bff49c46-192e-4723-9fea-07f3048ec5dd", "detail": ""}}'
  }}}

  Similar one
  https://8f916be78216e964d9a4-88701ef6bf8de11ee3b199ef1b631f6f.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-release-fips/bece4aa/testr_results.html

  2024-11-07 03:55:08,677 71012 INFO     [tempest.lib.common.rest_client] Request (RoutersNegativeTest:_run_cleanups): 500 PUT https://213.32.76.92/networking/v2.0/routers/74cb259a-8e93-4596-823e-3b5cabbcdb34/remove_router_interface 60.018s
  2024-11-07 03:55:08,678 71012 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"subnet_id": "bc0283dc-763d-4d27-9aab-3b8c5f15f636"}
      Response - Headers: {'date': 'Thu, 07 Nov 2024 03:54:08 GMT', 'server': 'Apache/2.4.62 (CentOS Stream) OpenSSL/3.2.2 mod_wsgi/4.7.1 Python/3.9', 'content-length': '527', 'content-type': 'text/html; charset=iso-8859-1', 'connection': 'close', 'status': '500', 'content-location': 'https://213.32.76.92/networking/v2.0/routers/74cb259a-8e93-4596-823e-3b5cabbcdb34/remove_router_interface'}
          Body: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n root@localhost to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n</body></html>\n'
  }}}

  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 82, in remove_router_interface
      return self.update_resource(uri, kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 77, in update_resource
      resp, body = self.put(req_uri, req_post_data)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 392, in put
      return self.request('PUT', url, extra_headers, headers, body, chunked)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 856, in _error_checker
      raise exceptions.UnexpectedContentType(str(resp.status),
  tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided
  Details: 500

  4) test_add_router_interfaces_on_overlapping_subnets_returns_400
  https://8b655475c2159741b8cc-0c9507f3c7942864f4e3074b37139748.ssl.cf1.rackcdn.com/934193/3/check/neutron-ovn-tempest-ipv6-only-ovs-release/e532bf5/testr_results.html
  https://df0cceafb057f67e6c7c-70474520a2e562b88d6ff63e0bb37737.ssl.cf2.rackcdn.com/932181/1/gate/neutron-ovn-tempest-ipv6-only-ovs-release/e413acd/testr_results.html
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 82, in remove_router_interface
      return self.update_resource(uri, kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 77, in update_resource
      resp, body = self.put(req_uri, req_post_data)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 392, in put
      return self.request('PUT', url, extra_headers, headers, body, chunked)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 872, in _error_checker
      raise exceptions.NotFound(resp_body, resp=resp)
  tempest.lib.exceptions.NotFound: Object not found
  Details: {'type': 'RouterInterfaceNotFoundForSubnet', 'message': 'Router cd6d753a-e734-4656-83a9-4a10ccc1c816 has no interface on subnet 574c353a-b8ac-4601-9485-4eaab5c179cd', 'detail': ''}

  2024-11-04 10:00:42,480 75691 WARNING  [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='2001:4802:7805:104:be76:4eff:fe20:2b93', port=443): Read timed out. (read timeout=90)")': /networking/v2.0/routers/cd6d753a-e734-4656-83a9-4a10ccc1c816/remove_router_interface
  2024-11-04 10:00:42,894 75691 INFO     [tempest.lib.common.rest_client] Request (RoutersNegativeTest:_run_cleanups): 404 PUT https://[2001:4802:7805:104:be76:4eff:fe20:2b93]/networking/v2.0/routers/cd6d753a-e734-4656-83a9-4a10ccc1c816/remove_router_interface 90.476s
  2024-11-04 10:00:42,895 75691 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"subnet_id": "574c353a-b8ac-4601-9485-4eaab5c179cd"}
      Response - Headers: {'date': 'Mon, 04 Nov 2024 10:00:42 GMT', 'server': 'Apache/2.4.52 (Ubuntu)', 'content-type': 'application/json', 'content-length': '198', 'x-openstack-request-id': 'req-2a717a0f-c2c6-43c8-98cc-23febc2e3ce4', 'connection': 'close', 'status': '404', 'content-location': 'https://[2001:4802:7805:104:be76:4eff:fe20:2b93]/networking/v2.0/routers/cd6d753a-e734-4656-83a9-4a10ccc1c816/remove_router_interface'}
          Body: b'{"NeutronError": {"type": "RouterInterfaceNotFoundForSubnet", "message": "Router cd6d753a-e734-4656-83a9-4a10ccc1c816 has no interface on subnet 574c353a-b8ac-4601-9485-4eaab5c179cd", "detail": ""}}'
  }}}
  For this there were some related fixes done as part of https://bugs.launchpad.net/neutron/+bug/2083287

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2087824/+subscriptions
References

[Bug 2087824] [NEW] random router related test failures since wsgi switch
From: yatin, 2024-11-11