yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2015847] [NEW] Functional job db missconfiguration

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <2015847@xxxxxxxxxxxxxxxxxx>
Date: Tue, 11 Apr 2023 08:33:26 -0000
Reply-to: Bug 2015847 <2015847@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

There is (probably) some issue with configuration of the functional
tests job neutron-functional-with-sqlalchemy-master which runs in
periodic queue. It is failing every day with same error:

2023-04-11 03:57:17.753314 | controller | neutron.tests.functional.db.test_migrations.TestWalkMigrationsPostgreSQL.test_walk_versions
2023-04-11 03:57:17.753325 | controller | -------------------------------------------------------------------------------------------
2023-04-11 03:57:17.753336 | controller |
2023-04-11 03:57:17.753346 | controller | Captured traceback:
2023-04-11 03:57:17.753357 | controller | ~~~~~~~~~~~~~~~~~~~
2023-04-11 03:57:17.753368 | controller |     Traceback (most recent call last):
2023-04-11 03:57:17.753379 | controller |
2023-04-11 03:57:17.753390 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-04-11 03:57:17.753401 | controller |     self._dbapi_connection = engine.raw_connection()
2023-04-11 03:57:17.753412 | controller |
2023-04-11 03:57:17.753429 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
2023-04-11 03:57:17.753441 | controller |     return self.pool.connect()
2023-04-11 03:57:17.753452 | controller |
2023-04-11 03:57:17.753463 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
2023-04-11 03:57:17.753474 | controller |     return _ConnectionFairy._checkout(self)
2023-04-11 03:57:17.753492 | controller |
2023-04-11 03:57:17.753504 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
2023-04-11 03:57:17.753514 | controller |     fairy = _ConnectionRecord.checkout(pool)
2023-04-11 03:57:17.753525 | controller |
2023-04-11 03:57:17.753536 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
2023-04-11 03:57:17.753548 | controller |     rec = pool._do_get()
2023-04-11 03:57:17.753558 | controller |
2023-04-11 03:57:17.753570 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
2023-04-11 03:57:17.753581 | controller |     with util.safe_reraise():
2023-04-11 03:57:17.753591 | controller |
2023-04-11 03:57:17.753602 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.753613 | controller |     raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.753624 | controller |
2023-04-11 03:57:17.753635 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
2023-04-11 03:57:17.753645 | controller |     return self._create_connection()
2023-04-11 03:57:17.753656 | controller |
2023-04-11 03:57:17.753667 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
2023-04-11 03:57:17.753678 | controller |     return _ConnectionRecord(self)
2023-04-11 03:57:17.753689 | controller |
2023-04-11 03:57:17.753700 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
2023-04-11 03:57:17.753711 | controller |     self.__connect()
2023-04-11 03:57:17.753722 | controller |
2023-04-11 03:57:17.753733 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
2023-04-11 03:57:17.753743 | controller |     with util.safe_reraise():
2023-04-11 03:57:17.753754 | controller |
2023-04-11 03:57:17.753769 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.753781 | controller |     raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.753792 | controller |
2023-04-11 03:57:17.753803 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
2023-04-11 03:57:17.753814 | controller |     self.dbapi_connection = connection = pool._invoke_creator(self)
2023-04-11 03:57:17.753825 | controller |
2023-04-11 03:57:17.753836 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
2023-04-11 03:57:17.753847 | controller |     return dialect.connect(*cargs, **cparams)
2023-04-11 03:57:17.753858 | controller |
2023-04-11 03:57:17.753868 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
2023-04-11 03:57:17.753879 | controller |     return self.loaded_dbapi.connect(*cargs, **cparams)
2023-04-11 03:57:17.753890 | controller |
2023-04-11 03:57:17.753901 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
2023-04-11 03:57:17.753912 | controller |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2023-04-11 03:57:17.753923 | controller |
2023-04-11 03:57:17.753934 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
2023-04-11 03:57:17.753945 | controller |     state = conn.poll()
2023-04-11 03:57:17.753962 | controller |
2023-04-11 03:57:17.753974 | controller |     psycopg2.OperationalError: FATAL:  password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.753985 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.753997 | controller |
2023-04-11 03:57:17.754007 | controller |
2023-04-11 03:57:17.754018 | controller |
2023-04-11 03:57:17.754029 | controller | The above exception was the direct cause of the following exception:
2023-04-11 03:57:17.754040 | controller |
2023-04-11 03:57:17.754051 | controller |
2023-04-11 03:57:17.754062 | controller |     Traceback (most recent call last):
2023-04-11 03:57:17.754073 | controller |
2023-04-11 03:57:17.754084 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
2023-04-11 03:57:17.754095 | controller |     return f(self, *args, **kwargs)
2023-04-11 03:57:17.754105 | controller |
2023-04-11 03:57:17.754116 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/db/test_migrations.py", line 576, in test_walk_versions
2023-04-11 03:57:17.754128 | controller |     migration.do_alembic_command(config, 'upgrade', dest)
2023-04-11 03:57:17.754138 | controller |
2023-04-11 03:57:17.754149 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/cli.py", line 90, in do_alembic_command
2023-04-11 03:57:17.754160 | controller |     getattr(alembic_command, cmd)(config, *args, **kwargs)
2023-04-11 03:57:17.754171 | controller |
2023-04-11 03:57:17.754188 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/command.py", line 378, in upgrade
2023-04-11 03:57:17.754200 | controller |     script.run_env()
2023-04-11 03:57:17.754211 | controller |
2023-04-11 03:57:17.754222 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
2023-04-11 03:57:17.754233 | controller |     util.load_python_file(self.dir, "env.py")
2023-04-11 03:57:17.754244 | controller |
2023-04-11 03:57:17.754255 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
2023-04-11 03:57:17.754266 | controller |     module = load_module_py(module_id, path)
2023-04-11 03:57:17.754277 | controller |
2023-04-11 03:57:17.754288 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
2023-04-11 03:57:17.754299 | controller |     spec.loader.exec_module(module)  # type: ignore
2023-04-11 03:57:17.754310 | controller |
2023-04-11 03:57:17.754321 | controller |       File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2023-04-11 03:57:17.754332 | controller |
2023-04-11 03:57:17.754343 | controller |       File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2023-04-11 03:57:17.754354 | controller |
2023-04-11 03:57:17.754365 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
2023-04-11 03:57:17.754376 | controller |     run_migrations_online()
2023-04-11 03:57:17.754387 | controller |
2023-04-11 03:57:17.754398 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online
2023-04-11 03:57:17.754409 | controller |     with DBConnection(neutron_config.database.connection, connection) as conn:
2023-04-11 03:57:17.754420 | controller |
2023-04-11 03:57:17.754431 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/connection.py", line 32, in __enter__
2023-04-11 03:57:17.754442 | controller |     self.engine = session.create_engine(self.connection_url)
2023-04-11 03:57:17.754453 | controller |
2023-04-11 03:57:17.754464 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/removals.py", line 260, in wrapper
2023-04-11 03:57:17.754475 | controller |     return f(*args, **kwargs)
2023-04-11 03:57:17.754485 | controller |
2023-04-11 03:57:17.754496 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/renames.py", line 41, in decorator
2023-04-11 03:57:17.754517 | controller |     return wrapped(*args, **kwargs)
2023-04-11 03:57:17.754529 | controller |
2023-04-11 03:57:17.754540 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 257, in create_engine
2023-04-11 03:57:17.754551 | controller |     test_conn = _test_connection(engine, max_retries, retry_interval)
2023-04-11 03:57:17.754562 | controller |
2023-04-11 03:57:17.754573 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 453, in _test_connection
2023-04-11 03:57:17.754584 | controller |     return engine.connect()
2023-04-11 03:57:17.754595 | controller |
2023-04-11 03:57:17.754606 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3251, in connect
2023-04-11 03:57:17.754617 | controller |     return self._connection_cls(self)
2023-04-11 03:57:17.754628 | controller |
2023-04-11 03:57:17.754639 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
2023-04-11 03:57:17.754650 | controller |     Connection._handle_dbapi_exception_noconnection(
2023-04-11 03:57:17.754660 | controller |
2023-04-11 03:57:17.754671 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2410, in _handle_dbapi_exception_noconnection
2023-04-11 03:57:17.754682 | controller |     raise newraise.with_traceback(exc_info[2]) from e
2023-04-11 03:57:17.754693 | controller |
2023-04-11 03:57:17.754704 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-04-11 03:57:17.754715 | controller |     self._dbapi_connection = engine.raw_connection()
2023-04-11 03:57:17.754726 | controller |
2023-04-11 03:57:17.754737 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
2023-04-11 03:57:17.754748 | controller |     return self.pool.connect()
2023-04-11 03:57:17.754764 | controller |
2023-04-11 03:57:17.754776 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
2023-04-11 03:57:17.754787 | controller |     return _ConnectionFairy._checkout(self)
2023-04-11 03:57:17.754798 | controller |
2023-04-11 03:57:17.754809 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
2023-04-11 03:57:17.754820 | controller |     fairy = _ConnectionRecord.checkout(pool)
2023-04-11 03:57:17.754831 | controller |
2023-04-11 03:57:17.754842 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
2023-04-11 03:57:17.754853 | controller |     rec = pool._do_get()
2023-04-11 03:57:17.754864 | controller |
2023-04-11 03:57:17.754881 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
2023-04-11 03:57:17.754893 | controller |     with util.safe_reraise():
2023-04-11 03:57:17.754905 | controller |
2023-04-11 03:57:17.754916 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.754926 | controller |     raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.754938 | controller |
2023-04-11 03:57:17.754949 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
2023-04-11 03:57:17.754960 | controller |     return self._create_connection()
2023-04-11 03:57:17.754971 | controller |
2023-04-11 03:57:17.754982 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
2023-04-11 03:57:17.754999 | controller |     return _ConnectionRecord(self)
2023-04-11 03:57:17.755011 | controller |
2023-04-11 03:57:17.755022 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
2023-04-11 03:57:17.755033 | controller |     self.__connect()
2023-04-11 03:57:17.755044 | controller |
2023-04-11 03:57:17.755055 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
2023-04-11 03:57:17.755066 | controller |     with util.safe_reraise():
2023-04-11 03:57:17.755076 | controller |
2023-04-11 03:57:17.755091 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-04-11 03:57:17.755103 | controller |     raise exc_value.with_traceback(exc_tb)
2023-04-11 03:57:17.755114 | controller |
2023-04-11 03:57:17.755125 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
2023-04-11 03:57:17.755136 | controller |     self.dbapi_connection = connection = pool._invoke_creator(self)
2023-04-11 03:57:17.755147 | controller |
2023-04-11 03:57:17.755158 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
2023-04-11 03:57:17.755169 | controller |     return dialect.connect(*cargs, **cparams)
2023-04-11 03:57:17.755180 | controller |
2023-04-11 03:57:17.755191 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
2023-04-11 03:57:17.755202 | controller |     return self.loaded_dbapi.connect(*cargs, **cparams)
2023-04-11 03:57:17.755213 | controller |
2023-04-11 03:57:17.755224 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
2023-04-11 03:57:17.755235 | controller |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2023-04-11 03:57:17.755246 | controller |
2023-04-11 03:57:17.755257 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
2023-04-11 03:57:17.755268 | controller |     state = conn.poll()
2023-04-11 03:57:17.755279 | controller |
2023-04-11 03:57:17.755290 | controller |     sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.755301 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "openstack_citest"
2023-04-11 03:57:17.755311 | controller |
2023-04-11 03:57:17.755322 | controller | (Background on this error at: https://sqlalche.me/e/20/e3q8)

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: functional-tests

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2015847

Title:
  Functional job db missconfiguration

Status in neutron:
  Confirmed

Bug description:
  There is (probably) some issue with configuration of the functional
  tests job neutron-functional-with-sqlalchemy-master which runs in
  periodic queue. It is failing every day with same error:

  2023-04-11 03:57:17.753314 | controller | neutron.tests.functional.db.test_migrations.TestWalkMigrationsPostgreSQL.test_walk_versions
  2023-04-11 03:57:17.753325 | controller | -------------------------------------------------------------------------------------------
  2023-04-11 03:57:17.753336 | controller |
  2023-04-11 03:57:17.753346 | controller | Captured traceback:
  2023-04-11 03:57:17.753357 | controller | ~~~~~~~~~~~~~~~~~~~
  2023-04-11 03:57:17.753368 | controller |     Traceback (most recent call last):
  2023-04-11 03:57:17.753379 | controller |
  2023-04-11 03:57:17.753390 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
  2023-04-11 03:57:17.753401 | controller |     self._dbapi_connection = engine.raw_connection()
  2023-04-11 03:57:17.753412 | controller |
  2023-04-11 03:57:17.753429 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
  2023-04-11 03:57:17.753441 | controller |     return self.pool.connect()
  2023-04-11 03:57:17.753452 | controller |
  2023-04-11 03:57:17.753463 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
  2023-04-11 03:57:17.753474 | controller |     return _ConnectionFairy._checkout(self)
  2023-04-11 03:57:17.753492 | controller |
  2023-04-11 03:57:17.753504 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
  2023-04-11 03:57:17.753514 | controller |     fairy = _ConnectionRecord.checkout(pool)
  2023-04-11 03:57:17.753525 | controller |
  2023-04-11 03:57:17.753536 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
  2023-04-11 03:57:17.753548 | controller |     rec = pool._do_get()
  2023-04-11 03:57:17.753558 | controller |
  2023-04-11 03:57:17.753570 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
  2023-04-11 03:57:17.753581 | controller |     with util.safe_reraise():
  2023-04-11 03:57:17.753591 | controller |
  2023-04-11 03:57:17.753602 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  2023-04-11 03:57:17.753613 | controller |     raise exc_value.with_traceback(exc_tb)
  2023-04-11 03:57:17.753624 | controller |
  2023-04-11 03:57:17.753635 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
  2023-04-11 03:57:17.753645 | controller |     return self._create_connection()
  2023-04-11 03:57:17.753656 | controller |
  2023-04-11 03:57:17.753667 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
  2023-04-11 03:57:17.753678 | controller |     return _ConnectionRecord(self)
  2023-04-11 03:57:17.753689 | controller |
  2023-04-11 03:57:17.753700 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
  2023-04-11 03:57:17.753711 | controller |     self.__connect()
  2023-04-11 03:57:17.753722 | controller |
  2023-04-11 03:57:17.753733 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
  2023-04-11 03:57:17.753743 | controller |     with util.safe_reraise():
  2023-04-11 03:57:17.753754 | controller |
  2023-04-11 03:57:17.753769 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  2023-04-11 03:57:17.753781 | controller |     raise exc_value.with_traceback(exc_tb)
  2023-04-11 03:57:17.753792 | controller |
  2023-04-11 03:57:17.753803 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
  2023-04-11 03:57:17.753814 | controller |     self.dbapi_connection = connection = pool._invoke_creator(self)
  2023-04-11 03:57:17.753825 | controller |
  2023-04-11 03:57:17.753836 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
  2023-04-11 03:57:17.753847 | controller |     return dialect.connect(*cargs, **cparams)
  2023-04-11 03:57:17.753858 | controller |
  2023-04-11 03:57:17.753868 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
  2023-04-11 03:57:17.753879 | controller |     return self.loaded_dbapi.connect(*cargs, **cparams)
  2023-04-11 03:57:17.753890 | controller |
  2023-04-11 03:57:17.753901 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
  2023-04-11 03:57:17.753912 | controller |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
  2023-04-11 03:57:17.753923 | controller |
  2023-04-11 03:57:17.753934 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
  2023-04-11 03:57:17.753945 | controller |     state = conn.poll()
  2023-04-11 03:57:17.753962 | controller |
  2023-04-11 03:57:17.753974 | controller |     psycopg2.OperationalError: FATAL:  password authentication failed for user "openstack_citest"
  2023-04-11 03:57:17.753985 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "openstack_citest"
  2023-04-11 03:57:17.753997 | controller |
  2023-04-11 03:57:17.754007 | controller |
  2023-04-11 03:57:17.754018 | controller |
  2023-04-11 03:57:17.754029 | controller | The above exception was the direct cause of the following exception:
  2023-04-11 03:57:17.754040 | controller |
  2023-04-11 03:57:17.754051 | controller |
  2023-04-11 03:57:17.754062 | controller |     Traceback (most recent call last):
  2023-04-11 03:57:17.754073 | controller |
  2023-04-11 03:57:17.754084 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
  2023-04-11 03:57:17.754095 | controller |     return f(self, *args, **kwargs)
  2023-04-11 03:57:17.754105 | controller |
  2023-04-11 03:57:17.754116 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/db/test_migrations.py", line 576, in test_walk_versions
  2023-04-11 03:57:17.754128 | controller |     migration.do_alembic_command(config, 'upgrade', dest)
  2023-04-11 03:57:17.754138 | controller |
  2023-04-11 03:57:17.754149 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/cli.py", line 90, in do_alembic_command
  2023-04-11 03:57:17.754160 | controller |     getattr(alembic_command, cmd)(config, *args, **kwargs)
  2023-04-11 03:57:17.754171 | controller |
  2023-04-11 03:57:17.754188 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/command.py", line 378, in upgrade
  2023-04-11 03:57:17.754200 | controller |     script.run_env()
  2023-04-11 03:57:17.754211 | controller |
  2023-04-11 03:57:17.754222 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
  2023-04-11 03:57:17.754233 | controller |     util.load_python_file(self.dir, "env.py")
  2023-04-11 03:57:17.754244 | controller |
  2023-04-11 03:57:17.754255 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
  2023-04-11 03:57:17.754266 | controller |     module = load_module_py(module_id, path)
  2023-04-11 03:57:17.754277 | controller |
  2023-04-11 03:57:17.754288 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
  2023-04-11 03:57:17.754299 | controller |     spec.loader.exec_module(module)  # type: ignore
  2023-04-11 03:57:17.754310 | controller |
  2023-04-11 03:57:17.754321 | controller |       File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  2023-04-11 03:57:17.754332 | controller |
  2023-04-11 03:57:17.754343 | controller |       File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  2023-04-11 03:57:17.754354 | controller |
  2023-04-11 03:57:17.754365 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
  2023-04-11 03:57:17.754376 | controller |     run_migrations_online()
  2023-04-11 03:57:17.754387 | controller |
  2023-04-11 03:57:17.754398 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/alembic_migrations/env.py", line 106, in run_migrations_online
  2023-04-11 03:57:17.754409 | controller |     with DBConnection(neutron_config.database.connection, connection) as conn:
  2023-04-11 03:57:17.754420 | controller |
  2023-04-11 03:57:17.754431 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/migration/connection.py", line 32, in __enter__
  2023-04-11 03:57:17.754442 | controller |     self.engine = session.create_engine(self.connection_url)
  2023-04-11 03:57:17.754453 | controller |
  2023-04-11 03:57:17.754464 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/removals.py", line 260, in wrapper
  2023-04-11 03:57:17.754475 | controller |     return f(*args, **kwargs)
  2023-04-11 03:57:17.754485 | controller |
  2023-04-11 03:57:17.754496 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/debtcollector/renames.py", line 41, in decorator
  2023-04-11 03:57:17.754517 | controller |     return wrapped(*args, **kwargs)
  2023-04-11 03:57:17.754529 | controller |
  2023-04-11 03:57:17.754540 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 257, in create_engine
  2023-04-11 03:57:17.754551 | controller |     test_conn = _test_connection(engine, max_retries, retry_interval)
  2023-04-11 03:57:17.754562 | controller |
  2023-04-11 03:57:17.754573 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 453, in _test_connection
  2023-04-11 03:57:17.754584 | controller |     return engine.connect()
  2023-04-11 03:57:17.754595 | controller |
  2023-04-11 03:57:17.754606 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3251, in connect
  2023-04-11 03:57:17.754617 | controller |     return self._connection_cls(self)
  2023-04-11 03:57:17.754628 | controller |
  2023-04-11 03:57:17.754639 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
  2023-04-11 03:57:17.754650 | controller |     Connection._handle_dbapi_exception_noconnection(
  2023-04-11 03:57:17.754660 | controller |
  2023-04-11 03:57:17.754671 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2410, in _handle_dbapi_exception_noconnection
  2023-04-11 03:57:17.754682 | controller |     raise newraise.with_traceback(exc_info[2]) from e
  2023-04-11 03:57:17.754693 | controller |
  2023-04-11 03:57:17.754704 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
  2023-04-11 03:57:17.754715 | controller |     self._dbapi_connection = engine.raw_connection()
  2023-04-11 03:57:17.754726 | controller |
  2023-04-11 03:57:17.754737 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
  2023-04-11 03:57:17.754748 | controller |     return self.pool.connect()
  2023-04-11 03:57:17.754764 | controller |
  2023-04-11 03:57:17.754776 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 455, in connect
  2023-04-11 03:57:17.754787 | controller |     return _ConnectionFairy._checkout(self)
  2023-04-11 03:57:17.754798 | controller |
  2023-04-11 03:57:17.754809 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
  2023-04-11 03:57:17.754820 | controller |     fairy = _ConnectionRecord.checkout(pool)
  2023-04-11 03:57:17.754831 | controller |
  2023-04-11 03:57:17.754842 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
  2023-04-11 03:57:17.754853 | controller |     rec = pool._do_get()
  2023-04-11 03:57:17.754864 | controller |
  2023-04-11 03:57:17.754881 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
  2023-04-11 03:57:17.754893 | controller |     with util.safe_reraise():
  2023-04-11 03:57:17.754905 | controller |
  2023-04-11 03:57:17.754916 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  2023-04-11 03:57:17.754926 | controller |     raise exc_value.with_traceback(exc_tb)
  2023-04-11 03:57:17.754938 | controller |
  2023-04-11 03:57:17.754949 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
  2023-04-11 03:57:17.754960 | controller |     return self._create_connection()
  2023-04-11 03:57:17.754971 | controller |
  2023-04-11 03:57:17.754982 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
  2023-04-11 03:57:17.754999 | controller |     return _ConnectionRecord(self)
  2023-04-11 03:57:17.755011 | controller |
  2023-04-11 03:57:17.755022 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
  2023-04-11 03:57:17.755033 | controller |     self.__connect()
  2023-04-11 03:57:17.755044 | controller |
  2023-04-11 03:57:17.755055 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 905, in __connect
  2023-04-11 03:57:17.755066 | controller |     with util.safe_reraise():
  2023-04-11 03:57:17.755076 | controller |
  2023-04-11 03:57:17.755091 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
  2023-04-11 03:57:17.755103 | controller |     raise exc_value.with_traceback(exc_tb)
  2023-04-11 03:57:17.755114 | controller |
  2023-04-11 03:57:17.755125 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
  2023-04-11 03:57:17.755136 | controller |     self.dbapi_connection = connection = pool._invoke_creator(self)
  2023-04-11 03:57:17.755147 | controller |
  2023-04-11 03:57:17.755158 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 636, in connect
  2023-04-11 03:57:17.755169 | controller |     return dialect.connect(*cargs, **cparams)
  2023-04-11 03:57:17.755180 | controller |
  2023-04-11 03:57:17.755191 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 579, in connect
  2023-04-11 03:57:17.755202 | controller |     return self.loaded_dbapi.connect(*cargs, **cparams)
  2023-04-11 03:57:17.755213 | controller |
  2023-04-11 03:57:17.755224 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
  2023-04-11 03:57:17.755235 | controller |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
  2023-04-11 03:57:17.755246 | controller |
  2023-04-11 03:57:17.755257 | controller |       File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
  2023-04-11 03:57:17.755268 | controller |     state = conn.poll()
  2023-04-11 03:57:17.755279 | controller |
  2023-04-11 03:57:17.755290 | controller |     sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "openstack_citest"
  2023-04-11 03:57:17.755301 | controller | connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "openstack_citest"
  2023-04-11 03:57:17.755311 | controller |
  2023-04-11 03:57:17.755322 | controller | (Background on this error at: https://sqlalche.me/e/20/e3q8)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2015847/+subscriptions
Follow ups

[Bug 2015847] Re: [sqlalchemy-20][FT] Functional job db missconfiguration with SQLAlchemy2.0
From: OpenStack Infra, 2023-05-04