yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1611627] [NEW] revision plugin throwing objectdeletederror

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1611627@xxxxxxxxxxxxxxxxxx>
Date: Wed, 10 Aug 2016 06:42:28 -0000
Reply-to: Bug 1611627 <1611627@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If there is an expired object in the session that got concurrently
deleted by another session, the revision plugin may throw an
ObjectDeletedError when it goes to bump the revision and references
expired attributes (causing a DB lookup that returns nothing).


2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource [req-c1b008ba-a6a7-4ac0-bb71-baf27253e098 tempest-NetworksTestDHCPv6-1396555541 -] create failed: No details.
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 397, in create
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.force_reraise()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 510, in _create
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     obj = do_create(body)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 492, in do_create
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.force_reraise()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 485, in do_create
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 613, in inner
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 977, in create_subnet
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result, mech_context = self._create_subnet_db(context, subnet)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 965, in _create_subnet_db
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result = super(Ml2Plugin, self).create_subnet(context, subnet)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 723, in create_subnet
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self._create_subnet(context, subnet, subnetpool_id)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 623, in _create_subnet
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     subnet, ipam_subnet)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 350, in add_auto_addrs_on_network_ports
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     context.session.add(allocated)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.rollback()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.commit()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._prepare_impl()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.session.flush()
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._flush(objects)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2046, in _flush
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.dispatch.before_flush(self, flush_context, objects)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 218, in __call__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     fn(*args, **kw)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 52, in bump_revisions
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._bump_related_revisions(session, obj)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 56, in _bump_related_revisions
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     related_obj = self._find_related_obj(session, obj, revises_col)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 100, in _find_related_obj
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     if getattr(session_obj, pk.name) == rel_id:
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self.impl.get(instance_state(instance), dict_)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 578, in get
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     value = state._load_expired(state, passive)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.manager.deferred_scalar_loader(self, toload)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 669, in load_scalar_attributes
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     raise orm_exc.ObjectDeletedError(state)
2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource ObjectDeletedError: Instance '<Port at 0x7f994e799d50>' has been deleted, or its row is otherwise not present.

** Affects: neutron
     Importance: Critical
     Assignee: Kevin Benton (kevinbenton)
         Status: New


** Tags: gate-failure

** Changed in: neutron
   Importance: Undecided => Critical

** Changed in: neutron
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1611627

Title:
  revision plugin throwing objectdeletederror

Status in neutron:
  New

Bug description:
  If there is an expired object in the session that got concurrently
  deleted by another session, the revision plugin may throw an
  ObjectDeletedError when it goes to bump the revision and references
  expired attributes (causing a DB lookup that returns nothing).

  
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource [req-c1b008ba-a6a7-4ac0-bb71-baf27253e098 tempest-NetworksTestDHCPv6-1396555541 -] create failed: No details.
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 397, in create
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 510, in _create
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     obj = do_create(body)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 492, in do_create
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 485, in do_create
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 613, in inner
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 977, in create_subnet
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result, mech_context = self._create_subnet_db(context, subnet)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 965, in _create_subnet_db
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     result = super(Ml2Plugin, self).create_subnet(context, subnet)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 723, in create_subnet
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self._create_subnet(context, subnet, subnetpool_id)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 623, in _create_subnet
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     subnet, ipam_subnet)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 350, in add_auto_addrs_on_network_ports
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     context.session.add(allocated)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.rollback()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.commit()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._prepare_impl()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.session.flush()
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._flush(objects)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2046, in _flush
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.dispatch.before_flush(self, flush_context, objects)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 218, in __call__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     fn(*args, **kw)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 52, in bump_revisions
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self._bump_related_revisions(session, obj)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 56, in _bump_related_revisions
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     related_obj = self._find_related_obj(session, obj, revises_col)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/services/revisions/revision_plugin.py", line 100, in _find_related_obj
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     if getattr(session_obj, pk.name) == rel_id:
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     return self.impl.get(instance_state(instance), dict_)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 578, in get
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     value = state._load_expired(state, passive)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     self.manager.deferred_scalar_loader(self, toload)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 669, in load_scalar_attributes
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource     raise orm_exc.ObjectDeletedError(state)
  2016-08-09 23:57:02.093 17791 ERROR neutron.api.v2.resource ObjectDeletedError: Instance '<Port at 0x7f994e799d50>' has been deleted, or its row is otherwise not present.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1611627/+subscriptions
Follow ups

[Bug 1611627] Re: revision plugin throwing objectdeletederror
From: OpenStack Infra, 2016-08-12