← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #83469

[Bug 1889781] [NEW] Functional tests on Ubuntu 20.04 are timed out

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Probably due to very large number of logs it's timeouting at some point.
We had similar issues in the past and we limited some WARNING messages
to not be logged to workaround the issue.

Example of job's logs:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_117/734304/9/check
/neutron-functional/11781bb/job-output.txt

Most likely problem is caused by logs like below. And those tests are
passing.

2020-07-29 21:56:37.633888 | controller | The above exception was the direct cause of the following exception:
2020-07-29 21:56:37.633897 | controller |
2020-07-29 21:56:37.633909 | controller | Traceback (most recent call last):
2020-07-29 21:56:37.633918 | controller |   File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
2020-07-29 21:56:37.633927 | controller |     self.gen.throw(type, value, traceback)
2020-07-29 21:56:37.633936 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
2020-07-29 21:56:37.633946 | controller |     yield resource
2020-07-29 21:56:37.633956 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_hash_ring_db.py", line 42, in remove_nodes_from_host
2020-07-29 21:56:37.633965 | controller |     context.session.query(ovn_models.OVNHashRing).filter(
2020-07-29 21:56:37.633974 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3894, in delete
2020-07-29 21:56:37.633984 | controller |     delete_op.exec_()
2020-07-29 21:56:37.633994 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec_
2020-07-29 21:56:37.634004 | controller |     self._do_exec()
2020-07-29 21:56:37.634013 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1928, in _do_exec
2020-07-29 21:56:37.634023 | controller |     self._execute_stmt(delete_stmt)
2020-07-29 21:56:37.634032 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt
2020-07-29 21:56:37.634042 | controller |     self.result = self.query._execute_crud(stmt, self.mapper)
2020-07-29 21:56:37.634050 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3536, in _execute_crud
2020-07-29 21:56:37.634059 | controller |     return conn.execute(stmt, self._params)
2020-07-29 21:56:37.634069 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
2020-07-29 21:56:37.634079 | controller |     return meth(self, multiparams, params)
2020-07-29 21:56:37.634088 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2020-07-29 21:56:37.634098 | controller |     return connection._execute_clauseelement(self, multiparams, params)
2020-07-29 21:56:37.634121 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
2020-07-29 21:56:37.634132 | controller |     ret = self._execute_context(
2020-07-29 21:56:37.634142 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
2020-07-29 21:56:37.634150 | controller |     self._handle_dbapi_exception(
2020-07-29 21:56:37.634163 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1509, in _handle_dbapi_exception
2020-07-29 21:56:37.634172 | controller |     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2020-07-29 21:56:37.634187 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2020-07-29 21:56:37.634196 | controller |     raise exception
2020-07-29 21:56:37.634205 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
2020-07-29 21:56:37.634214 | controller |     self.dialect.do_execute(
2020-07-29 21:56:37.634223 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
2020-07-29 21:56:37.634232 | controller |     cursor.execute(statement, parameters)
2020-07-29 21:56:37.634241 | controller | oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: ovn_hash_ring
2020-07-29 21:56:37.634251 | controller | [SQL: DELETE FROM ovn_hash_ring WHERE ovn_hash_ring.hostname = ? AND ovn_hash_ring.group_name = ?]
2020-07-29 21:56:37.634260 | controller | [parameters: ('ubuntu-focal-ovh-gra1-0018961947', 'mechanism_driver')]
2020-07-29 21:56:37.634269 | controller | (Background on this error at: http://sqlalche.me/e/13/e3q8)

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: functional-tests gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1889781

Title:
  Functional tests on Ubuntu 20.04 are timed out

Status in neutron:
  Confirmed

Bug description:
  Probably due to very large number of logs it's timeouting at some
  point. We had similar issues in the past and we limited some WARNING
  messages to not be logged to workaround the issue.

  Example of job's logs:
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_117/734304/9/check
  /neutron-functional/11781bb/job-output.txt

  Most likely problem is caused by logs like below. And those tests are
  passing.

  2020-07-29 21:56:37.633888 | controller | The above exception was the direct cause of the following exception:
  2020-07-29 21:56:37.633897 | controller |
  2020-07-29 21:56:37.633909 | controller | Traceback (most recent call last):
  2020-07-29 21:56:37.633918 | controller |   File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
  2020-07-29 21:56:37.633927 | controller |     self.gen.throw(type, value, traceback)
  2020-07-29 21:56:37.633936 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
  2020-07-29 21:56:37.633946 | controller |     yield resource
  2020-07-29 21:56:37.633956 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_hash_ring_db.py", line 42, in remove_nodes_from_host
  2020-07-29 21:56:37.633965 | controller |     context.session.query(ovn_models.OVNHashRing).filter(
  2020-07-29 21:56:37.633974 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3894, in delete
  2020-07-29 21:56:37.633984 | controller |     delete_op.exec_()
  2020-07-29 21:56:37.633994 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec_
  2020-07-29 21:56:37.634004 | controller |     self._do_exec()
  2020-07-29 21:56:37.634013 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1928, in _do_exec
  2020-07-29 21:56:37.634023 | controller |     self._execute_stmt(delete_stmt)
  2020-07-29 21:56:37.634032 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt
  2020-07-29 21:56:37.634042 | controller |     self.result = self.query._execute_crud(stmt, self.mapper)
  2020-07-29 21:56:37.634050 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3536, in _execute_crud
  2020-07-29 21:56:37.634059 | controller |     return conn.execute(stmt, self._params)
  2020-07-29 21:56:37.634069 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
  2020-07-29 21:56:37.634079 | controller |     return meth(self, multiparams, params)
  2020-07-29 21:56:37.634088 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
  2020-07-29 21:56:37.634098 | controller |     return connection._execute_clauseelement(self, multiparams, params)
  2020-07-29 21:56:37.634121 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
  2020-07-29 21:56:37.634132 | controller |     ret = self._execute_context(
  2020-07-29 21:56:37.634142 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
  2020-07-29 21:56:37.634150 | controller |     self._handle_dbapi_exception(
  2020-07-29 21:56:37.634163 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1509, in _handle_dbapi_exception
  2020-07-29 21:56:37.634172 | controller |     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2020-07-29 21:56:37.634187 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2020-07-29 21:56:37.634196 | controller |     raise exception
  2020-07-29 21:56:37.634205 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
  2020-07-29 21:56:37.634214 | controller |     self.dialect.do_execute(
  2020-07-29 21:56:37.634223 | controller |   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
  2020-07-29 21:56:37.634232 | controller |     cursor.execute(statement, parameters)
  2020-07-29 21:56:37.634241 | controller | oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: ovn_hash_ring
  2020-07-29 21:56:37.634251 | controller | [SQL: DELETE FROM ovn_hash_ring WHERE ovn_hash_ring.hostname = ? AND ovn_hash_ring.group_name = ?]
  2020-07-29 21:56:37.634260 | controller | [parameters: ('ubuntu-focal-ovh-gra1-0018961947', 'mechanism_driver')]
  2020-07-29 21:56:37.634269 | controller | (Background on this error at: http://sqlalche.me/e/13/e3q8)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1889781/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next