yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1915271] [NEW] test_create_router_set_gateway_with_fixed_ip fails in dvr-ha job

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <1915271@xxxxxxxxxxxxxxxxxx>
Date: Wed, 10 Feb 2021 13:27:29 -0000
Reply-to: Bug 1915271 <1915271@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

test_create_router_set_gateway_with_fixed_ip periodically fails in neutron-tempest-dvr-ha-multinode-full.
Failure start to happen after new engine facade switch in L3 code.
Failure is due to router failed to be deleted (neutron returns 500), see first line in below traceback.

Traceback:

2021-01-28 13:36:26,139 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:test_create_router_set_gateway_with_fixed_ip): 500 DELETE https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445 10.779s
2021-01-28 13:36:26,139 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '150', 'x-openstack-request-id': 'req-2392442f-a3b0-4c79-a79c-c765c7cab834', 'connection': 'close', 'status': '500', 'content-location': 'https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445'}
        Body: b'{"NeutronError": {"type": "HTTPInternalServerError", "message": "Request Failed: internal server error while processing your request.", "detail": ""}}'
2021-01-28 13:36:26,305 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd 0.165s
2021-01-28 13:36:26,306 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '204', 'x-openstack-request-id': 'req-9eb0af8b-9154-489b-94b8-93d3da458c05', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd'}
        Body: b'{"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.", "detail": ""}}'
2021-01-28 13:36:26,544 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688 0.238s
2021-01-28 13:36:26,545 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '205', 'x-openstack-request-id': 'req-a412a0f0-832c-45f4-9e9d-40deef43cde2', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688'}
        Body: b'{"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.", "detail": ""}}'
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.', 'detail': ''}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/tempest/lib/services/network/networks_client.py", line 52, in delete_network
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'NetworkInUse', 'message': 'Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.', 'detail': ''}
}}}

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 109, in wrapper
    return func(*func_args, **func_kwargs)
  File "/opt/stack/tempest/tempest/api/network/admin/test_routers.py", line 254, in test_create_router_set_gateway_with_fixed_ip
    self.admin_routers_client.delete_router(router['id'])
  File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 52, in delete_router
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 880, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.

** Affects: neutron
     Importance: High
     Assignee: Oleg Bondarev (obondarev)
         Status: Confirmed


** Tags: gate-failure l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1915271

Title:
  test_create_router_set_gateway_with_fixed_ip fails in dvr-ha job

Status in neutron:
  Confirmed

Bug description:
  test_create_router_set_gateway_with_fixed_ip periodically fails in neutron-tempest-dvr-ha-multinode-full.
  Failure start to happen after new engine facade switch in L3 code.
  Failure is due to router failed to be deleted (neutron returns 500), see first line in below traceback.

  Traceback:

  2021-01-28 13:36:26,139 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:test_create_router_set_gateway_with_fixed_ip): 500 DELETE https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445 10.779s
  2021-01-28 13:36:26,139 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '150', 'x-openstack-request-id': 'req-2392442f-a3b0-4c79-a79c-c765c7cab834', 'connection': 'close', 'status': '500', 'content-location': 'https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445'}
          Body: b'{"NeutronError": {"type": "HTTPInternalServerError", "message": "Request Failed: internal server error while processing your request.", "detail": ""}}'
  2021-01-28 13:36:26,305 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd 0.165s
  2021-01-28 13:36:26,306 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '204', 'x-openstack-request-id': 'req-9eb0af8b-9154-489b-94b8-93d3da458c05', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd'}
          Body: b'{"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.", "detail": ""}}'
  2021-01-28 13:36:26,544 81948 INFO     [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688 0.238s
  2021-01-28 13:36:26,545 81948 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '205', 'x-openstack-request-id': 'req-a412a0f0-832c-45f4-9e9d-40deef43cde2', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688'}
          Body: b'{"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.", "detail": ""}}'
  }}}

  traceback-1: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.', 'detail': ''}
  }}}

  traceback-2: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/tempest/tempest/lib/services/network/networks_client.py", line 52, in delete_network
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'NetworkInUse', 'message': 'Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.', 'detail': ''}
  }}}

  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 109, in wrapper
      return func(*func_args, **func_kwargs)
    File "/opt/stack/tempest/tempest/api/network/admin/test_routers.py", line 254, in test_create_router_set_gateway_with_fixed_ip
      self.admin_routers_client.delete_router(router['id'])
    File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 52, in delete_router
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 880, in _error_checker
      raise exceptions.ServerFault(resp_body, resp=resp,
  tempest.lib.exceptions.ServerFault: Got server fault
  Details: Request Failed: internal server error while processing your request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1915271/+subscriptions
Follow ups

[Bug 1915271] Re: test_create_router_set_gateway_with_fixed_ip fails in dvr-ha job
From: Slawek Kaplonski, 2021-02-23