yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #67210
[Bug 1715163] [NEW] StaleDataError in _reconfigure_ha_resources while running tempest tests
Public bug reported:
When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure.
(note: Link to q-svc log file [1])
Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop
Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources
Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.schedule_router(context, router_id)
Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router
Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, context, router, candidates=candidates)
Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule
Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager plugin, context, router_id, candidates=candidates)
Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router
Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager if not plugin.router_supports_scheduling(context, router_id):
Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling
Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return self.l3_driver_controller.uses_scheduler(context, router_id)
Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler
Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return (self.get_provider_for_router(context, router_id).
Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router
Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, [router_id]).get(router_id)
Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids
Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, resource_id=resource_ids)
Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager with context.session.begin(subtransactions=True):
Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin
Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, nested=nested)
Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__
Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._take_snapshot()
Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot
Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.session.flush()
Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._flush(objects)
Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager transaction.rollback(_capture_exception=True)
Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager compat.reraise(exc_type, exc_value, exc_tb)
Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager flush_context.execute()
Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager rec.execute(self)
Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager uow
Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager mapper, table, update)
Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager (table.description, len(records), rows))
Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager
[1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz
** Affects: neutron
Importance: Undecided
Assignee: venkata anil (anil-venkata)
Status: New
** Changed in: neutron
Assignee: (unassigned) => venkata anil (anil-venkata)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1715163
Title:
StaleDataError in _reconfigure_ha_resources while running tempest
tests
Status in neutron:
New
Bug description:
When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure.
(note: Link to q-svc log file [1])
Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop
Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources
Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.schedule_router(context, router_id)
Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router
Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, context, router, candidates=candidates)
Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule
Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager plugin, context, router_id, candidates=candidates)
Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router
Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager if not plugin.router_supports_scheduling(context, router_id):
Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling
Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return self.l3_driver_controller.uses_scheduler(context, router_id)
Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler
Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return (self.get_provider_for_router(context, router_id).
Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router
Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, [router_id]).get(router_id)
Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids
Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, resource_id=resource_ids)
Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager with context.session.begin(subtransactions=True):
Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin
Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, nested=nested)
Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__
Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._take_snapshot()
Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot
Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.session.flush()
Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._flush(objects)
Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager transaction.rollback(_capture_exception=True)
Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager compat.reraise(exc_type, exc_value, exc_tb)
Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager flush_context.execute()
Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager rec.execute(self)
Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager uow
Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager mapper, table, update)
Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager (table.description, len(records), rows))
Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager
[1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1715163/+subscriptions
Follow ups