yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2066221] [NEW] [dalmatian] skip level grenade jobs fails at keystone-manage db_sync

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2066221@xxxxxxxxxxxxxxxxxx>
Date: Tue, 21 May 2024 06:17:10 -0000
Reply-to: Bug 2066221 <2066221@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Fails like:-
2024-05-20 03:18:02.465 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 :   /usr/local/bin/keystone-manage db_sync
2024-05-20 03:18:05.597 | [00;36mINFO dbcounter [[00;36m-[00;36m] [01;35m[00;36mRegistered counter for database keystone[00m[00m
2024-05-20 03:18:05.613 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: NotImplementedError
2024-05-20 03:18:05.613 | ERROR keystone [01;35m[00mTraceback (most recent call last):
2024-05-20 03:18:05.613 | ERROR keystone [01;35m[00m  File "/usr/local/bin/keystone-manage", line 10, in <module>
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    sys.exit(main())
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/manage.py", line 40, in main
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    cli.main(argv=sys.argv, developer_config_file=developer_config)
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 1463, in main
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    CONF.command.cmd_class.main()
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 413, in main
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    upgrades.offline_sync_database_to_version(
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 304, in offline_sync_database_to_version
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    _db_sync(engine=engine)
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 195, in _db_sync
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    with sql.session_for_write() as session:
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return next(self.gen)
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    with current._produce_block(
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return next(self.gen)
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 641, in _session
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self.session = self.factory._create_session(
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 404, in _create_session
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self._start()
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 493, in _start
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self._setup_for_connection(
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 519, in _setup_for_connection
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    engine = engines.create_engine(
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/debtcollector/renames.py", line 41, in decorator
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return wrapped(*args, **kwargs)
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py", line 194, in create_engine
2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    engine = sqlalchemy.create_engine(url, **engine_args)
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "<string>", line 2, in create_engine
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/deprecations.py", line 281, in warned
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    return fn(*args, **kwargs)  # type: ignore[no-any-return]
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/create.py", line 548, in create_engine
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    u, plugins, kwargs = u._instantiate_plugins(kwargs)
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/url.py", line 739, in _instantiate_plugins
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    new_u = plugin.update_url(u)
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/interfaces.py", line 2870, in update_url
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    raise NotImplementedError()
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00mNotImplementedError
2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m[00m
2024-05-20 03:18:06.079 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 :   die 61 'DB sync error'
2024-05-20 03:18:06.082 | + functions-common:die:285                 :   local exitcode=1
2024-05-20 03:18:06.085 | [Call Trace]
2024-05-20 03:18:06.085 | /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61:die
2024-05-20 03:18:06.089 | [ERROR] /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61 DB sync error
2024-05-20 03:18:07.092 | 1 die /opt/stack/new/devstack/functions-common
2024-05-20 03:18:07.092 | 61 main /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
2024-05-20 03:18:07.092 | Exit code: 1
2024-05-20 03:18:09.512 | + inc/upgrade:upgrade_service:65           :   die 65 'Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh'
2024-05-20 03:18:09.514 | + /opt/stack/new/devstack/functions-common:die:285 :   local exitcode=1
2024-05-20 03:18:09.517 | [Call Trace]
2024-05-20 03:18:09.517 | ./grenade.sh:347:upgrade_service
2024-05-20 03:18:09.517 | /opt/stack/new/grenade/inc/upgrade:65:die
2024-05-20 03:18:09.521 | [ERROR] /opt/stack/new/grenade/inc/upgrade:65 Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
2024-05-20 03:18:10.523 | 1 die /opt/stack/new/devstack/functions-common
2024-05-20 03:18:10.523 | 65 upgrade_service /opt/stack/new/grenade/inc/upgrade
2024-05-20 03:18:10.523 | 347 main ./grenade.sh
2024-05-20 03:18:10.523 | Exit code: 1


Example failure:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ae/periodic/opendev.org/openstack/neutron/master/neutron-ovn-grenade-multinode-skip-level/3aee59a/controller/logs/grenade.sh_log.txt
Example builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode-skip-level&job_name=neutron-ovs-grenade-multinode-skip-level&project=openstack%2Fneutron&branch=master&skip=0

Timing wise could be related to sqlalchemy update to 2.0
Also jobs upgrading from 2023.1 , this also needs to be fixed to upgrade from 2023.2

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2066221

Title:
  [dalmatian] skip level grenade jobs fails at keystone-manage db_sync

Status in neutron:
  New

Bug description:
  Fails like:-
  2024-05-20 03:18:02.465 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 :   /usr/local/bin/keystone-manage db_sync
  2024-05-20 03:18:05.597 | [00;36mINFO dbcounter [[00;36m-[00;36m] [01;35m[00;36mRegistered counter for database keystone[00m[00m
  2024-05-20 03:18:05.613 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: NotImplementedError
  2024-05-20 03:18:05.613 | ERROR keystone [01;35m[00mTraceback (most recent call last):
  2024-05-20 03:18:05.613 | ERROR keystone [01;35m[00m  File "/usr/local/bin/keystone-manage", line 10, in <module>
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    sys.exit(main())
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/manage.py", line 40, in main
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    cli.main(argv=sys.argv, developer_config_file=developer_config)
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 1463, in main
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    CONF.command.cmd_class.main()
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/cmd/cli.py", line 413, in main
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    upgrades.offline_sync_database_to_version(
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 304, in offline_sync_database_to_version
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    _db_sync(engine=engine)
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/opt/stack/new/keystone/keystone/common/sql/upgrades.py", line 195, in _db_sync
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    with sql.session_for_write() as session:
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return next(self.gen)
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    with current._produce_block(
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return next(self.gen)
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 641, in _session
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self.session = self.factory._create_session(
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 404, in _create_session
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self._start()
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 493, in _start
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    self._setup_for_connection(
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 519, in _setup_for_connection
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    engine = engines.create_engine(
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/debtcollector/renames.py", line 41, in decorator
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    return wrapped(*args, **kwargs)
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/engines.py", line 194, in create_engine
  2024-05-20 03:18:05.614 | ERROR keystone [01;35m[00m    engine = sqlalchemy.create_engine(url, **engine_args)
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "<string>", line 2, in create_engine
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/util/deprecations.py", line 281, in warned
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/create.py", line 548, in create_engine
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    u, plugins, kwargs = u._instantiate_plugins(kwargs)
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/url.py", line 739, in _instantiate_plugins
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    new_u = plugin.update_url(u)
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/interfaces.py", line 2870, in update_url
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m    raise NotImplementedError()
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00mNotImplementedError
  2024-05-20 03:18:05.615 | ERROR keystone [01;35m[00m[00m
  2024-05-20 03:18:06.079 | + /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:main:61 :   die 61 'DB sync error'
  2024-05-20 03:18:06.082 | + functions-common:die:285                 :   local exitcode=1
  2024-05-20 03:18:06.085 | [Call Trace]
  2024-05-20 03:18:06.085 | /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61:die
  2024-05-20 03:18:06.089 | [ERROR] /opt/stack/new/grenade/projects/10_keystone/upgrade.sh:61 DB sync error
  2024-05-20 03:18:07.092 | 1 die /opt/stack/new/devstack/functions-common
  2024-05-20 03:18:07.092 | 61 main /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
  2024-05-20 03:18:07.092 | Exit code: 1
  2024-05-20 03:18:09.512 | + inc/upgrade:upgrade_service:65           :   die 65 'Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh'
  2024-05-20 03:18:09.514 | + /opt/stack/new/devstack/functions-common:die:285 :   local exitcode=1
  2024-05-20 03:18:09.517 | [Call Trace]
  2024-05-20 03:18:09.517 | ./grenade.sh:347:upgrade_service
  2024-05-20 03:18:09.517 | /opt/stack/new/grenade/inc/upgrade:65:die
  2024-05-20 03:18:09.521 | [ERROR] /opt/stack/new/grenade/inc/upgrade:65 Failure in /opt/stack/new/grenade/projects/10_keystone/upgrade.sh
  2024-05-20 03:18:10.523 | 1 die /opt/stack/new/devstack/functions-common
  2024-05-20 03:18:10.523 | 65 upgrade_service /opt/stack/new/grenade/inc/upgrade
  2024-05-20 03:18:10.523 | 347 main ./grenade.sh
  2024-05-20 03:18:10.523 | Exit code: 1

  
  Example failure:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ae/periodic/opendev.org/openstack/neutron/master/neutron-ovn-grenade-multinode-skip-level/3aee59a/controller/logs/grenade.sh_log.txt
  Example builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode-skip-level&job_name=neutron-ovs-grenade-multinode-skip-level&project=openstack%2Fneutron&branch=master&skip=0

  Timing wise could be related to sqlalchemy update to 2.0
  Also jobs upgrading from 2023.1 , this also needs to be fixed to upgrade from 2023.2

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2066221/+subscriptions