yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1845300] [NEW] FWaaS tempest tests are failing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1845300@xxxxxxxxxxxxxxxxxx>
Date: Wed, 25 Sep 2019 08:53:47 -0000
Reply-to: Bug 1845300 <1845300@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I noticed at least 2 or 3 times that fwaas tempest tests defined in neutron-tempest-plugin repo are failing from time to time.
It seems that failures are due to errors like:

traceback-1: {{{
Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 131, in _try_delete_firewall_group
    self.firewall_groups_client.delete_firewall_group(fwg_id)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 4d900386-2806-4879-bddf-d1f77fbdcb2c is still active.', 'detail': ''}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 3fab4478-ffa8-41bf-a1ce-f0a2d82af518 is being used.', 'detail': ''}
}}}

traceback-3: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 725c2ef0-5fcb-4149-98e8-e52b8437fa4a is being used.', 'detail': ''}
}}}

traceback-4: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule 7b95169e-f975-440e-a44c-d6df440ed3c0 is being used.', 'detail': ''}
}}}

traceback-5: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule d1dd80dd-750e-408b-801c-a71762eda350 is being used.', 'detail': ''}
}}}

Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 342, in test_update_firewall_group
    self.firewall_groups_client.delete_firewall_group(fwg_id)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 4d900386-2806-4879-bddf-d1f77fbdcb2c is still active.', 'detail': ''}


And it happens probably during cleanup phase.

Example of failures:
 http://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_151/652099/38/check/neutron-tempest-plugin-fwaas/151d2d2/testr_results.html.gz
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_868/682059/3/check/neutron-tempest-plugin-fwaas/86846fb/testr_results.html.gz

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: fwaas gate-failure

** Summary changed:

- FWAAS tempest tests are failing
+ FWaaS tempest tests are failing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1845300

Title:
  FWaaS tempest tests are failing

Status in neutron:
  Confirmed

Bug description:
  I noticed at least 2 or 3 times that fwaas tempest tests defined in neutron-tempest-plugin repo are failing from time to time.
  It seems that failures are due to errors like:

  traceback-1: {{{
  Traceback (most recent call last):
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 131, in _try_delete_firewall_group
      self.firewall_groups_client.delete_firewall_group(fwg_id)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 4d900386-2806-4879-bddf-d1f77fbdcb2c is still active.', 'detail': ''}
  }}}

  traceback-2: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 3fab4478-ffa8-41bf-a1ce-f0a2d82af518 is being used.', 'detail': ''}
  }}}

  traceback-3: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 725c2ef0-5fcb-4149-98e8-e52b8437fa4a is being used.', 'detail': ''}
  }}}

  traceback-4: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule 7b95169e-f975-440e-a44c-d6df440ed3c0 is being used.', 'detail': ''}
  }}}

  traceback-5: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule d1dd80dd-750e-408b-801c-a71762eda350 is being used.', 'detail': ''}
  }}}

  Traceback (most recent call last):
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 342, in test_update_firewall_group
      self.firewall_groups_client.delete_firewall_group(fwg_id)
    File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 800, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 4d900386-2806-4879-bddf-d1f77fbdcb2c is still active.', 'detail': ''}

  
  And it happens probably during cleanup phase.

  Example of failures:
   http://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_151/652099/38/check/neutron-tempest-plugin-fwaas/151d2d2/testr_results.html.gz
  https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_868/682059/3/check/neutron-tempest-plugin-fwaas/86846fb/testr_results.html.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1845300/+subscriptions
Follow ups

[Bug 1845300] Re: FWaaS tempest tests are failing
From: Slawek Kaplonski, 2021-12-10