yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2069704] [NEW] rally jobs broken on unmaintantained/yoga and older branches

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bence Romsics <2069704@xxxxxxxxxxxxxxxxxx>
Date: Tue, 18 Jun 2024 09:01:01 -0000
Reply-to: Bug 2069704 <2069704@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

affected branches: unmaintantained/yoga and older
affected jobs: neutron-ovs-rally-task (voting), neutron-ovn-rally-task (non-voting)

typical error message:

2024-06-16 01:35:40.555933 | controller | ++ /opt/stack/rally-openstack/devstack/lib/rally:init_rally:124 :   rally --config-file /etc/rally/rally.conf deployment create --name devstack --filename /tmp/tmp.5bKViViucY
2024-06-16 01:35:44.306252 | controller | Command failed, please check log for more info
2024-06-16 01:35:44.314415 | controller | 2024-06-16 01:35:44.306 82074 CRITICAL rally [-] Unhandled error: sqlalchemy.exc.InvalidRequestError: No transaction is begun.
2024-06-16 01:35:44.314470 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally Traceback (most recent call last):
2024-06-16 01:35:44.314524 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/bin/rally", line 8, in <module>
2024-06-16 01:35:44.314559 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     sys.exit(main())
2024-06-16 01:35:44.314629 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/main.py", line 40, in main
2024-06-16 01:35:44.314657 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return cliutils.run(sys.argv, categories)
2024-06-16 01:35:44.314702 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/cliutils.py", line 669, in run
2024-06-16 01:35:44.314748 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     ret = fn(*fn_args, **fn_kwargs)
2024-06-16 01:35:44.314795 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/plugins/__init__.py", line 59, in wrapper
2024-06-16 01:35:44.314841 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return func(*args, **kwargs)
2024-06-16 01:35:44.314887 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/commands/deployment.py", line 109, in create
2024-06-16 01:35:44.314930 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     deployment = api.deployment.create(config=config, name=name)
2024-06-16 01:35:44.314995 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/api.py", line 107, in create
2024-06-16 01:35:44.315021 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return self._create(config, name).to_dict()
2024-06-16 01:35:44.315096 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/api.py", line 89, in _create
2024-06-16 01:35:44.315140 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     deployment = objects.Deployment(name=name, config=config,
2024-06-16 01:35:44.315212 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/common/objects/deploy.py", line 76, in __init__
2024-06-16 01:35:44.315256 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self._env = env_mgr.EnvManager.create(
2024-06-16 01:35:44.315368 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/plugins/__init__.py", line 59, in wrapper
2024-06-16 01:35:44.315413 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return func(*args, **kwargs)
2024-06-16 01:35:44.315455 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/env/env_mgr.py", line 374, in create
2024-06-16 01:35:44.315497 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self = cls._validate_and_create_env(name, spec)
2024-06-16 01:35:44.315563 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/env/env_mgr.py", line 276, in _validate_and_create_env
2024-06-16 01:35:44.315608 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return cls(db.env_create(name, STATUS.INIT, description, extras,
2024-06-16 01:35:44.315684 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/common/db/api.py", line 135, in wrapper
2024-06-16 01:35:44.315728 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     result = f(session, *args, **kwargs)
2024-06-16 01:35:44.315769 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2024-06-16 01:35:44.315811 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     next(self.gen)
2024-06-16 01:35:44.315875 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
2024-06-16 01:35:44.315901 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     yield resource
2024-06-16 01:35:44.315977 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2024-06-16 01:35:44.316003 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     next(self.gen)                                                                                                                                         
2024-06-16 01:35:44.316071 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session
2024-06-16 01:35:44.316124 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self.session.rollback()
2024-06-16 01:35:44.316146 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-06-16 01:35:44.316194 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self.force_reraise()
2024-06-16 01:35:44.316233 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-06-16 01:35:44.316280 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     raise self.value
2024-06-16 01:35:44.316319 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session
2024-06-16 01:35:44.316362 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self._end_session_transaction(self.session)
2024-06-16 01:35:44.316403 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction
2024-06-16 01:35:44.316444 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     session.commit()
2024-06-16 01:35:44.316495 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1429, in commit
2024-06-16 01:35:44.316536 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     raise sa_exc.InvalidRequestError("No transaction is begun.")
2024-06-16 01:35:44.316590 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally sqlalchemy.exc.InvalidRequestError: No transaction is begun.
2024-06-16 01:35:44.316614 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally ^[[00m
2024-06-16 01:35:44.686439 | controller | + /opt/stack/rally-openstack/devstack/lib/rally:init_rally:1 :   exit_trap

example job failures:

https://zuul.opendev.org/t/openstack/build/f31271a9d9d94aeeb47250878d21538f
https://zuul.opendev.org/t/openstack/build/89bd8f7b92ad4bea8925143fc8451ee9
https://zuul.opendev.org/t/openstack/build/f6ae550750a24076b8e73f8dc707d75d

** Affects: neutron
     Importance: Low
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2069704

Title:
  rally jobs broken on unmaintantained/yoga and older branches

Status in neutron:
  New

Bug description:
  affected branches: unmaintantained/yoga and older
  affected jobs: neutron-ovs-rally-task (voting), neutron-ovn-rally-task (non-voting)

  typical error message:

  2024-06-16 01:35:40.555933 | controller | ++ /opt/stack/rally-openstack/devstack/lib/rally:init_rally:124 :   rally --config-file /etc/rally/rally.conf deployment create --name devstack --filename /tmp/tmp.5bKViViucY
  2024-06-16 01:35:44.306252 | controller | Command failed, please check log for more info
  2024-06-16 01:35:44.314415 | controller | 2024-06-16 01:35:44.306 82074 CRITICAL rally [-] Unhandled error: sqlalchemy.exc.InvalidRequestError: No transaction is begun.
  2024-06-16 01:35:44.314470 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally Traceback (most recent call last):
  2024-06-16 01:35:44.314524 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/bin/rally", line 8, in <module>
  2024-06-16 01:35:44.314559 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     sys.exit(main())
  2024-06-16 01:35:44.314629 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/main.py", line 40, in main
  2024-06-16 01:35:44.314657 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return cliutils.run(sys.argv, categories)
  2024-06-16 01:35:44.314702 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/cliutils.py", line 669, in run
  2024-06-16 01:35:44.314748 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     ret = fn(*fn_args, **fn_kwargs)
  2024-06-16 01:35:44.314795 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/plugins/__init__.py", line 59, in wrapper
  2024-06-16 01:35:44.314841 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return func(*args, **kwargs)
  2024-06-16 01:35:44.314887 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/cli/commands/deployment.py", line 109, in create
  2024-06-16 01:35:44.314930 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     deployment = api.deployment.create(config=config, name=name)
  2024-06-16 01:35:44.314995 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/api.py", line 107, in create
  2024-06-16 01:35:44.315021 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return self._create(config, name).to_dict()
  2024-06-16 01:35:44.315096 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/api.py", line 89, in _create
  2024-06-16 01:35:44.315140 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     deployment = objects.Deployment(name=name, config=config,
  2024-06-16 01:35:44.315212 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/common/objects/deploy.py", line 76, in __init__
  2024-06-16 01:35:44.315256 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self._env = env_mgr.EnvManager.create(
  2024-06-16 01:35:44.315368 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/plugins/__init__.py", line 59, in wrapper
  2024-06-16 01:35:44.315413 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return func(*args, **kwargs)
  2024-06-16 01:35:44.315455 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/env/env_mgr.py", line 374, in create
  2024-06-16 01:35:44.315497 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self = cls._validate_and_create_env(name, spec)
  2024-06-16 01:35:44.315563 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/env/env_mgr.py", line 276, in _validate_and_create_env
  2024-06-16 01:35:44.315608 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     return cls(db.env_create(name, STATUS.INIT, description, extras,
  2024-06-16 01:35:44.315684 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/rally/common/db/api.py", line 135, in wrapper
  2024-06-16 01:35:44.315728 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     result = f(session, *args, **kwargs)
  2024-06-16 01:35:44.315769 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  2024-06-16 01:35:44.315811 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     next(self.gen)
  2024-06-16 01:35:44.315875 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope
  2024-06-16 01:35:44.315901 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     yield resource
  2024-06-16 01:35:44.315977 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  2024-06-16 01:35:44.316003 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     next(self.gen)                                                                                                                                         
  2024-06-16 01:35:44.316071 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session
  2024-06-16 01:35:44.316124 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self.session.rollback()
  2024-06-16 01:35:44.316146 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-06-16 01:35:44.316194 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self.force_reraise()
  2024-06-16 01:35:44.316233 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-06-16 01:35:44.316280 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     raise self.value
  2024-06-16 01:35:44.316319 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session
  2024-06-16 01:35:44.316362 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     self._end_session_transaction(self.session)
  2024-06-16 01:35:44.316403 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction
  2024-06-16 01:35:44.316444 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     session.commit()
  2024-06-16 01:35:44.316495 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1429, in commit
  2024-06-16 01:35:44.316536 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally     raise sa_exc.InvalidRequestError("No transaction is begun.")
  2024-06-16 01:35:44.316590 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally sqlalchemy.exc.InvalidRequestError: No transaction is begun.
  2024-06-16 01:35:44.316614 | controller | 2024-06-16 01:35:44.306 82074 ERROR rally ^[[00m
  2024-06-16 01:35:44.686439 | controller | + /opt/stack/rally-openstack/devstack/lib/rally:init_rally:1 :   exit_trap

  example job failures:

  https://zuul.opendev.org/t/openstack/build/f31271a9d9d94aeeb47250878d21538f
  https://zuul.opendev.org/t/openstack/build/89bd8f7b92ad4bea8925143fc8451ee9
  https://zuul.opendev.org/t/openstack/build/f6ae550750a24076b8e73f8dc707d75d

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2069704/+subscriptions
Follow ups

[Bug 2069704] Re: rally jobs broken on unmaintantained/yoga and older branches
From: Bernard Cafarelli, 2024-06-20