yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2006683] Re: fwaas scenario job fails randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2006683@xxxxxxxxxxxxxxxxxx>
Date: Thu, 16 Feb 2023 00:41:09 -0000
Reply-to: Bug 2006683 <2006683@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron-fwaas/+/873897
Committed: https://opendev.org/openstack/neutron-fwaas/commit/054275813ec96e7f8f3781253251197fd9f26742
Submitter: "Zuul (22348)"
Branch:    master

commit 054275813ec96e7f8f3781253251197fd9f26742
Author: elajkat <lajos.katona@xxxxxxxx>
Date:   Wed Feb 15 11:52:41 2023 +0100

    Add CONTEXT_WRITER to _get_default_fwg_id
    
    Change-Id: I8cc90dc564e9d4fb5637216619ea972bf9a4c417
    Closes-Bug: #2006683


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2006683

Title:
  fwaas scenario job fails randomly

Status in neutron:
  Fix Released

Bug description:
  fwass scenario job fails randomly on random tests

  Example failures:-
  test test_update_firewall_group fails:-
    - https://2c1464b5b351d9aa5e93-14387ee33f74a27c93fba699ca02403e.ssl.cf1.rackcdn.com/869152/3/gate/neutron-tempest-plugin-fwaas/d9df246/testr_results.html
    - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_26d/873060/1/check/neutron-tempest-plugin-fwaas/26d3325/testr_results.html

  traceback-5: {{{
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 330, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 720, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 841, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule ddaf801f-b279-41f4-a793-bd02e37c886e is being used.', 'detail': ''}
  }}}

  Traceback (most recent call last):
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 341, in test_update_firewall_group
      self.firewall_groups_client.delete_firewall_group(fwg_id)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
      return self.delete_resource(uri)
    File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
      resp, body = self.delete(req_uri)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 330, in delete
      return self.request('DELETE', url, extra_headers, headers, body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 720, in request
      self._error_checker(resp, resp_body)
    File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 841, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)
  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 0845629f-cb8a-4796-8bb5-e8df06c6919f is still active.', 'detail': ''}

  
  Fails as:-
  Feb 07 22:44:15.544863 np0033003110 neutron-server[57277]: INFO neutron.api.v2.resource [None req-3d968b8a-c24c-43c1-9d82-ded509555450 tempest-FWaaSv2ExtensionTestJSON-1939451253 tempest-FWaaSv2ExtensionTestJSON-1939451253-project] create failed (client error): There was a conflict when trying to complete your request.
  Feb 07 22:44:15.545878 np0033003110 neutron-server[57277]: INFO neutron.wsgi [None req-3d968b8a-c24c-43c1-9d82-ded509555450 tempest-FWaaSv2ExtensionTestJSON-1939451253 tempest-FWaaSv2ExtensionTestJSON-1939451253-project] 149.202.171.72,149.202.171.72 "POST /networking/v2.0/fwaas/firewall_groups HTTP/1.1" status: 409  len: 378 time: 0.0383568

  
  test_create_show_delete_firewall_group:-
    - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a1d/873060/1/check/neutron-tempest-plugin-fwaas/a1da5c4/testr_results.html
    - https://c4dea4e2ebaa3412b0a2-051d59c98d54ce1c08087cd83140c470.ssl.cf1.rackcdn.com/873060/1/check/neutron-tempest-plugin-fwaas-zed/7ade93d/testr_results.html

  
  Traceback as:-
  Feb 09 01:35:54.898784 np0033019570 neutron-server[57239]: DEBUG neutron_fwaas.db.firewall.v2.firewall_db_v2 [None req-f56695ab-2f6b-4035-8b61-74bcaa43898f tempest-FWaaSv2ExtensionTestJSON-845501349 tempest-FWaaSv2ExtensionTestJSON-845501349-project] Default FWG was concurrently created {{(pid=57239) _ensure_default_firewall_group /opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py:940}}
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [None req-f56695ab-2f6b-4035-8b61-74bcaa43898f tempest-FWaaSv2ExtensionTestJSON-845501349 tempest-FWaaSv2ExtensionTestJSON-845501349-project] create failed: No details.: sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c119661f432d4bc5aaf2308e8a8bd47a' for key 'default_firewall_groups.PRIMARY'")
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: [SQL: INSERT INTO default_firewall_groups (project_id, firewall_group_id) VALUES (%(project_id)s, %(firewall_group_id)s)]
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: [parameters: {'project_id': 'c119661f432d4bc5aaf2308e8a8bd47a', 'firewall_group_id': '2b7267f6-ee45-49c1-aa54-5017e9d38e9c'}]
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/resource.py", line 46, in _count_resource
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     obj_count_getter = getattr(plugins[pname], count_getter_name)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource AttributeError: 'FirewallPluginV2' object has no attribute 'get_firewall_groups_count'
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.dialect.do_execute(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/cursors.py", line 148, in execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result = self._query(query)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/cursors.py", line 310, in _query
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     conn.query(q)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 548, in query
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 775, in _read_query_result
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result.read()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 1156, in read
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 725, in _read_packet
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     packet.raise_for_error()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise errorclass(errno, errval)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1062, "Duplicate entry 'c119661f432d4bc5aaf2308e8a8bd47a' for key 'default_firewall_groups.PRIMARY'")
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource The above exception was the direct cause of the following exception:
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py", line 901, in _ensure_default_firewall_group
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with db_api.CONTEXT_WRITER.using(context):
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     next(self.gen)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1066, in _transaction_scope
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with current._produce_block(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     next(self.gen)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 688, in _session
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.session.flush()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 3386, in flush
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._flush(objects)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 3525, in _flush
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with util.safe_reraise():
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     compat.raise_(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise exception
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 3486, in _flush
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     flush_context.execute()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     rec.execute(self)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     util.preloaded.orm_persistence.save_obj(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     _emit_insert_statements(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     c = connection._execute_20(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return meth(self, args_10style, kwargs_10style, execution_options)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return connection._execute_clauseelement(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     ret = self._execute_context(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._handle_dbapi_exception(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise exception
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.dialect.do_execute(
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/cursors.py", line 148, in execute
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result = self._query(query)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/cursors.py", line 310, in _query
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     conn.query(q)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 548, in query
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 775, in _read_query_result
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result.read()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 1156, in read
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/connections.py", line 725, in _read_packet
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     packet.raise_for_error()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise errorclass(errno, errval)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c119661f432d4bc5aaf2308e8a8bd47a' for key 'default_firewall_groups.PRIMARY'")
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [SQL: INSERT INTO default_firewall_groups (project_id, firewall_group_id) VALUES (%(project_id)s, %(firewall_group_id)s)]
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [parameters: {'project_id': 'c119661f432d4bc5aaf2308e8a8bd47a', 'firewall_group_id': '2b7267f6-ee45-49c1-aa54-5017e9d38e9c'}]
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource (Background on this error at: https://sqlalche.me/e/14/gkpj)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 140, in wrapped
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception() as ectxt:
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 186, in wrapped
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 488, in _create
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     reservation = quota.QUOTAS.make_reservation(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/__init__.py", line 95, in make_reservation
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self.get_driver().make_reservation(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 226, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f_with_retry(*args, **kwargs,
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 140, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception() as ectxt:
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 186, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise self.value
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/quota/driver_nolock.py", line 73, in make_reservation
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     used_and_reserved = self.get_resource_usage(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/quota/driver_nolock.py", line 92, in get_resource_usage
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return tracked_resource.count(context, None, project_id,
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/resource.py", line 160, in count
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self._count_func(context, self.plural_name, project_id)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/quota/resource.py", line 52, in _count_resource
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     obj_list = obj_getter(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_log/helpers.py", line 67, in wrapper
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return fn(*args, **kwargs)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/fwaas_plugin_v2.py", line 355, in get_firewall_groups
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self.driver.get_firewall_groups(context, filters, fields)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/service_drivers/driver_api.py", line 198, in get_firewall_groups
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self.firewall_db.get_firewall_groups(context, filters, fields)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py", line 1094, in get_firewall_groups
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._ensure_default_firewall_group(context, tenant_id)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py", line 941, in _ensure_default_firewall_group
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self._get_default_fwg_id(context, tenant_id)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py", line 863, in _get_default_fwg_id
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     project_id=tenant_id, name=const.DEFAULT_FWG).first()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2823, in first
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self.limit(1)._iter().first()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2907, in _iter
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     result = self.session.execute(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 1711, in execute
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     conn = self._connection_for_bind(bind)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     return self._transaction._connection_for_bind(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     self._assert_active()
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource     raise sa_exc.PendingRollbackError(
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c119661f432d4bc5aaf2308e8a8bd47a' for key 'default_firewall_groups.PRIMARY'")
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [SQL: INSERT INTO default_firewall_groups (project_id, firewall_group_id) VALUES (%(project_id)s, %(firewall_group_id)s)]
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [parameters: {'project_id': 'c119661f432d4bc5aaf2308e8a8bd47a', 'firewall_group_id': '2b7267f6-ee45-49c1-aa54-5017e9d38e9c'}]
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource 
  Feb 09 01:35:54.927897 np0033019570 neutron-server[57239]: INFO neutron.wsgi [None req-f56695ab-2f6b-4035-8b61-74bcaa43898f tempest-FWaaSv2ExtensionTestJSON-845501349 tempest-FWaaSv2ExtensionTestJSON-845501349-project] 158.69.69.103,158.69.69.103 "POST /networking/v2.0/fwaas/firewall_groups HTTP/1.1" status: 500  len: 368 time: 0.0755477

  Builds:-
  - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest-plugin-fwaas&skip=0
  - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest-plugin-fwaas-zed&skip=0

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2006683/+subscriptions
References

[Bug 2006683] [NEW] fwaas scenario job fails randomly
From: yatin, 2023-02-09