← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1715163] [NEW] StaleDataError in _reconfigure_ha_resources while running tempest tests

 

Public bug reported:

When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure.
(note: Link to q-svc log file [1])

Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop
Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources
Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.schedule_router(context, router_id)
Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router
Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, context, router, candidates=candidates)
Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule
Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     plugin, context, router_id, candidates=candidates)
Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router
Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     if not plugin.router_supports_scheduling(context, router_id):
Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling
Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return self.l3_driver_controller.uses_scheduler(context, router_id)
Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler
Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return (self.get_provider_for_router(context, router_id).
Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router
Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, [router_id]).get(router_id)
Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids
Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, resource_id=resource_ids)
Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     with context.session.begin(subtransactions=True):
Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin
Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, nested=nested)
Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__
Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._take_snapshot()
Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot
Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.session.flush()
Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._flush(objects)
Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     transaction.rollback(_capture_exception=True)
Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     compat.reraise(exc_type, exc_value, exc_tb)
Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     flush_context.execute()
Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     rec.execute(self)
Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     uow
Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     mapper, table, update)
Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     (table.description, len(records), rows))
Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager 


[1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz

** Affects: neutron
     Importance: Undecided
     Assignee: venkata anil (anil-venkata)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => venkata anil (anil-venkata)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1715163

Title:
  StaleDataError in _reconfigure_ha_resources while running tempest
  tests

Status in neutron:
  New

Bug description:
  When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure.
  (note: Link to q-svc log file [1])

  Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop
  Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources
  Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.schedule_router(context, router_id)
  Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router
  Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, context, router, candidates=candidates)
  Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule
  Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     plugin, context, router_id, candidates=candidates)
  Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router
  Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     if not plugin.router_supports_scheduling(context, router_id):
  Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling
  Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return self.l3_driver_controller.uses_scheduler(context, router_id)
  Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler
  Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return (self.get_provider_for_router(context, router_id).
  Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router
  Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, [router_id]).get(router_id)
  Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids
  Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, resource_id=resource_ids)
  Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
  Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     with context.session.begin(subtransactions=True):
  Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin
  Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, nested=nested)
  Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__
  Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._take_snapshot()
  Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot
  Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.session.flush()
  Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
  Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._flush(objects)
  Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
  Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     transaction.rollback(_capture_exception=True)
  Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     compat.reraise(exc_type, exc_value, exc_tb)
  Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
  Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     flush_context.execute()
  Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
  Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     rec.execute(self)
  Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
  Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     uow
  Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
  Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     mapper, table, update)
  Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
  Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     (table.description, len(records), rows))
  Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager 

  
  [1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1715163/+subscriptions


Follow ups