yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1715163] Re: StaleDataError in _reconfigure_ha_resources while running tempest tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1715163@xxxxxxxxxxxxxxxxxx>
Date: Thu, 05 Oct 2017 01:17:08 -0000
Reply-to: Bug 1715163 <1715163@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/507856
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7910c3a1c090ed7a46a32b371ecf73157ddba88f
Submitter: Jenkins
Branch:    master

commit 7910c3a1c090ed7a46a32b371ecf73157ddba88f
Author: venkata anil <anilvenkata@xxxxxxxxxx>
Date:   Wed Sep 27 13:27:48 2017 +0000

    Refactor DVR HA migarations DB operations
    
    when a router is migrated from/to HA, router interface device_owner
    in DB is updated in the AFTER_UPDATE event handler. Because of this
    we see StaleDataError Trace as described in the bug report.
    Instead, we follow DVR approach and update it in PRECOMMIT_UPDATE
    handler, which only does DB changes. Here we update device_owner only
    between legacy and HA router modes as migration involving DVR is
    already handled in DVR code.
    
    Also update distributed flag after all DB operations. Otherwise existing
    code may affect router unbinding during retry as distributed flag
    already updated before retry.
    
    Testing is covered with scenario tempest test i.e
    DVR HA migrations tests from gate job
    "legacy-tempest-dsvm-neutron-dvr-multinode-scenario".
    
    Closes-Bug: 1715163
    Change-Id: I6e5783c44c77ec07371921e900ed45efd031ebdb


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1715163

Title:
  StaleDataError in _reconfigure_ha_resources while running tempest
  tests

Status in neutron:
  Fix Released

Bug description:
  When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure.
  (note: Link to q-svc log file [1])

  Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop
  Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources
  Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.schedule_router(context, router_id)
  Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router
  Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, context, router, candidates=candidates)
  Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule
  Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     plugin, context, router_id, candidates=candidates)
  Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router
  Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     if not plugin.router_supports_scheduling(context, router_id):
  Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling
  Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return self.l3_driver_controller.uses_scheduler(context, router_id)
  Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler
  Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     return (self.get_provider_for_router(context, router_id).
  Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router
  Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, [router_id]).get(router_id)
  Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids
  Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     context, resource_id=resource_ids)
  Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
  Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     with context.session.begin(subtransactions=True):
  Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin
  Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self, nested=nested)
  Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__
  Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._take_snapshot()
  Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot
  Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self.session.flush()
  Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush
  Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     self._flush(objects)
  Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush
  Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     transaction.rollback(_capture_exception=True)
  Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     compat.reraise(exc_type, exc_value, exc_tb)
  Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush
  Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     flush_context.execute()
  Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
  Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     rec.execute(self)
  Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
  Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     uow
  Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
  Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     mapper, table, update)
  Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
  Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager     (table.description, len(records), rows))
  Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager 

  
  [1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1715163/+subscriptions
References

[Bug 1715163] [NEW] StaleDataError in _reconfigure_ha_resources while running tempest tests
From: venkata anil, 2017-09-05