yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2065779] Re: neutron-fullstack-with-uwsgi job broken at resource setup

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2065779@xxxxxxxxxxxxxxxxxx>
Date: Wed, 15 May 2024 18:07:49 -0000
Reply-to: Bug 2065779 <2065779@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/919594
Committed: https://opendev.org/openstack/neutron/commit/11255ede9715800e2945e0b318c93a5ac9f1e968
Submitter: "Zuul (22348)"
Branch:    master

commit 11255ede9715800e2945e0b318c93a5ac9f1e968
Author: Takashi Kajinami <kajinamit@xxxxxxxxxxxxxxx>
Date:   Wed May 15 00:12:22 2024 +0900

    Fix the broken neutron gate
    
    There are currently two issues affecting the neutron gate, one
    for pep8 job and the other for fullstack. We need to fix them
    both together since they cannot merge separately.
    
    Bump pylint to the latest version to fix pep8 job - it was
    pinned to an old version which is causing the job to hang.
    
    Do not use str(url) to stringify a URL for subsequent use
    to fix  the fullstack job. The str(url) function in SQLAlchemy
    hides the password. For a URL string that is to be re-used, use
    render_as_string(hide_password=False).
    
    Co-authored-by: Mike Bayer <mike_mp@xxxxxxxxxxxxxxxx>
    
    Closes-Bug: #2065779
    Change-Id: I8d04db91ddc2995fbff05b4c4c48baebcc418522


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2065779

Title:
  neutron-fullstack-with-uwsgi job broken at resource setup

Status in neutron:
  Fix Released

Bug description:
  All tests fails as:-
  ft1.1: neutron.tests.fullstack.test_agent_bandwidth_report.TestAgentBandwidthReport.test_agent_configurations(Open vSwitch agent)testtools.testresult.real._StringException: pythonlogging:'': {{{ERROR [stevedore.extension] Could not load 'oslo_cache.memcache_pool': No module named 'memcache'}}}
  traceback-1: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
      eventlet.sleep(sleep)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
      hub.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
  eventlet.timeout.Timeout: 60 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
      self._setUp()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
      common_utils.wait_until_true(self.server_is_live)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
      raise WaitTimeout(_("Timed out after %d seconds") % timeout)
  neutron.common.utils.WaitTimeout: Timed out after 60 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 207, in setUp
      raise SetupError(details)
  fixtures.fixture.SetupError: {}
  }}}

  traceback-2: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
      eventlet.sleep(sleep)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
      hub.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
  eventlet.timeout.Timeout: 60 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
      self._setUp()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
      common_utils.wait_until_true(self.server_is_live)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
      raise WaitTimeout(_("Timed out after %d seconds") % timeout)
  neutron.common.utils.WaitTimeout: Timed out after 60 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
      self._setUp()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/environment.py", line 423, in _setUp
      self.neutron_server = self.useFixture(
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 256, in useFixture
      fixture.setUp()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 211, in setUp
      raise MultipleExceptions(*errors)
  testtools.runtest.MultipleExceptions: ((<class 'neutron.common.utils.WaitTimeout'>, WaitTimeout('Timed out after 60 seconds'), <traceback object at 0x7f849a0df280>), (<class 'fixtures.fixture.SetupError'>, SetupError({}), <traceback object at 0x7f849a3a9cc0>))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 207, in setUp
      raise SetupError(details)
  fixtures.fixture.SetupError: {}
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
      eventlet.sleep(sleep)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
      hub.switch()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
  eventlet.timeout.Timeout: 60 seconds

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
      self._setUp()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
      common_utils.wait_until_true(self.server_is_live)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
      raise WaitTimeout(_("Timed out after %d seconds") % timeout)
  neutron.common.utils.WaitTimeout: Timed out after 60 seconds

  neutron server fails as:-
  2024-05-14 13:16:19.166 68768 ERROR neutron.service [None req-f40f8906-b901-4249-b984-b87293fbeb92 - - - - - -] Unrecoverable error: please check log for details.: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'openstack_citest'@'localhost' (using password: YES)")
  (Background on this error at: https://sqlalche.me/e/20/e3q8)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service Traceback (most recent call last):
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     self._dbapi_connection = engine.raw_connection()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3304, in raw_connection
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return self.pool.connect()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 449, in connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return _ConnectionFairy._checkout(self)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     fairy = _ConnectionRecord.checkout(pool)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     rec = pool._do_get()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     with util.safe_reraise():
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     raise exc_value.with_traceback(exc_tb)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return self._create_connection()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return _ConnectionRecord(self)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     self.__connect()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     with util.safe_reraise():
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     raise exc_value.with_traceback(exc_tb)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     self.dbapi_connection = connection = pool._invoke_creator(self)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 643, in connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return dialect.connect(*cargs, **cparams)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 617, in connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     return self.loaded_dbapi.connect(*cargs, **cparams)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 358, in __init__
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     self.connect()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 664, in connect
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     self._request_authentication()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 976, in _request_authentication
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     auth_packet = _auth.caching_sha2_password_auth(self, auth_packet)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/_auth.py", line 267, in caching_sha2_password_auth
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     pkt = _roundtrip(conn, data)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/_auth.py", line 120, in _roundtrip
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     pkt = conn._read_packet()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet 
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     packet.raise_for_error()
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     err.raise_mysql_exception(self._data)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     raise errorclass(errno, errval)
  2024-05-14 13:16:19.166 68768 ERROR neutron.service pymysql.err.OperationalError: (1045, "Access denied for user 'openstack_citest'@'localhost' (using password: YES)")
  2024-05-14 13:16:19.166 68768 ERROR neutron.service     

  
  Example failure:- https://688891352003d7aa1ebf-eb9cf67f730d66b65f59031307d31060.ssl.cf1.rackcdn.com/919602/2/check/neutron-fullstack-with-uwsgi/94787d6/testr_results.html

  Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-
  fullstack-with-uwsgi&project=openstack/neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2065779/+subscriptions
References

[Bug 2065779] [NEW] neutron-fullstack-with-uwsgi job broken at resource setup
From: yatin, 2024-05-15