yahoo-eng-team team mailing list archive

[yatin <1956344@xxxxxxxxxxxxxxxxxx>]

Closing it as the workaround is in place and recently we have not seen
failures in this test and also the test code have a NOTE added for
future cleanup. If we see again the failure bug can be re opened.

** Changed in: neutron
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1956344

Title:
  Functional test test_gateway_chassis_rebalance is failing
  intermittently

Status in neutron:
  Fix Released

Bug description:
  It may looks like similar to
  https://bugs.launchpad.net/neutron/+bug/1912369 but I noticed recently
  that at least 2 times test
  neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance
  failed due to DB integrity error:

  https://54389876f34a304893e1-7e513b902e81eb926eb2654dfc72839e.ssl.cf1.rackcdn.com/822299/4/check/neutron-
  functional-with-uwsgi/dd33a5d/testr_results.html

  https://fd3bdfefdb4656902679-237e20f2346dac25649a4e5e3b0485b4.ssl.cf1.rackcdn.com/815962/6/check/neutron-
  functional-with-uwsgi/ef35e56/testr_results.html

  Stacktrace:

  
  ft1.1: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      cursor.execute(statement, parameters)
  sqlite3.IntegrityError: FOREIGN KEY constraint failed

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 494, in test_gateway_chassis_rebalance
      router = self._create_router('router%d' % i, gw_info=gw_info)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 49, in _create_router
      return self.l3_plugin.create_router(self.context, router)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/services/ovn_l3/plugin.py", line 166, in create_router
      router = super(OVNL3RouterPlugin, self).create_router(context, router)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 2088, in create_router
      router_dict = super(L3_NAT_db_mixin, self).create_router(context,
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 218, in wrapped
      return method(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped
      setattr(e, '_RETRY_EXCEEDED', True)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
      ectxt.value = e.inner_exc
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped
      LOG.debug("Retry wrapper got retriable exception: %s", e)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 179, in wrapped
      return f(*dup_args, **dup_kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 288, in create_router
      router_db, _unused = db_utils.safe_creation(context, create,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/_utils.py", line 76, in safe_creation
      LOG.error("Cannot clean up created object %(obj)s. "
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/_utils.py", line 69, in safe_creation
      updated_obj, value = create_bindings(obj['id'])
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 264, in _update_gw_for_create_router
      self._update_router_gw_info(context, router_id,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_gwmode_db.py", line 74, in _update_router_gw_info
      super(L3_NAT_dbonly_mixin, self)._update_router_gw_info(
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 541, in _update_router_gw_info
      self._create_gw_port(context, router_id, router, network_id,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 502, in _create_gw_port
      self._create_router_gw_port(context, router,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 351, in _create_router_gw_port
      gw_port = plugin_utils.create_port(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/plugins/utils.py", line 337, in create_port
      return core_plugin.create_port(
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 701, in inner
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 218, in wrapped
      return method(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped
      setattr(e, '_RETRY_EXCEEDED', True)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
      ectxt.value = e.inner_exc
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped
      LOG.debug("Retry wrapper got retriable exception: %s", e)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 179, in wrapped
      return f(*dup_args, **dup_kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1485, in create_port
      result, mech_context = self._create_port_db(context, port)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1459, in _create_port_db
      binding = db.add_port_binding(context, result['id'])
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1010, in wrapper
      return fn(*args, **kwargs)
    File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
      next(self.gen)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
      yield resource
    File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
      next(self.gen)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session
      self.session.flush()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3363, in flush
      self._flush(objects)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3503, in _flush
      transaction.rollback(_capture_exception=True)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
      compat.raise_(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3463, in _flush
      flush_context.execute()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
      rec.execute(self)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
      util.preloaded.orm_persistence.save_obj(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 244, in save_obj
      _emit_insert_statements(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1221, in _emit_insert_statements
      result = connection._execute_20(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
      return meth(self, args_10style, kwargs_10style, execution_options)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
      return connection._execute_clauseelement(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
      ret = self._execute_context(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
      self._handle_dbapi_exception(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      cursor.execute(statement, parameters)
  oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
  [SQL: INSERT INTO ml2_port_bindings (port_id, host, vnic_type, profile, vif_type, vif_details, status) VALUES (?, ?, ?, ?, ?, ?, ?)]
  [parameters: ('07747eb9-1990-4a63-89be-9ee909951329', '', 'normal', '', 'unbound', '', 'ACTIVE')]
  (Background on this error at: https://sqlalche.me/e/14/gkpj)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1956344/+subscriptions