yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1611533] [NEW] ml2 transaction_guard broke out of tree plugins

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: YAMAMOTO Takashi <yamamoto@xxxxxxxxxxxx>
Date: Tue, 09 Aug 2016 22:13:31 -0000
Reply-to: Bug 1611533 <1611533@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

recent change [1] broke l3 plugin for networking-midonet.

[1] I9924600c57648f7eccaa5abb6979419d9547a2ff

l3 plugins for networking-odl and dragonflow seem to have similar code
and would be affected too.

eg.
http://logs.openstack.org/87/199387/36/check/gate-tempest-dsvm-networking-midonet-ml2/ceb0331/logs/q-svc.txt.gz?level=TRACE

2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource [req-af588d02-2944-411f-aa22-eafca4fdabeb tempest-TestSecurityGroupsBasicOps-509565194 -] remove_router_interface failed: No details.
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     self.force_reraise()
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 217, in _handle_action
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/networking-midonet/midonet/neutron/services/l3/l3_midonet.py", line 190, in remove_router_interface
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, interface_info)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1756, in remove_router_interface
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, interface_info)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 924, in remove_router_interface
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, subnet_id, device_owner)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 901, in _remove_interface_by_subnet
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     l3_port_check=False)
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 611, in inner
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     raise RuntimeError(_("Method cannot be called within a "
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource RuntimeError: Method cannot be called within a transaction.
2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource 
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource [req-c9ae4bf8-2baf-4327-be58-bb3006b4d9c9 tempest-TestSecurityGroupsBasicOps-2112515119 -] delete failed: No details.
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 522, in delete
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     self.force_reraise()
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 544, in _delete
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/networking-midonet/midonet/neutron/services/l3/l3_midonet.py", line 220, in delete_floatingip
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     super(MidonetL3ServicePlugin, self).delete_floatingip(context, id)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1779, in delete_floatingip
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     floating_ip = self._delete_floatingip(context, id)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1301, in _delete_floatingip
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     l3_port_check=False)
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 611, in inner
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     raise RuntimeError(_("Method cannot be called within a "
2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource RuntimeError: Method cannot be called within a transaction.

** Affects: networking-midonet
     Importance: Critical
     Assignee: YAMAMOTO Takashi (yamamoto)
         Status: New

** Affects: neutron
     Importance: Undecided
     Assignee: YAMAMOTO Takashi (yamamoto)
         Status: In Progress


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1611533

Title:
  ml2 transaction_guard broke out of tree plugins

Status in networking-midonet:
  New
Status in neutron:
  In Progress

Bug description:
  recent change [1] broke l3 plugin for networking-midonet.

  [1] I9924600c57648f7eccaa5abb6979419d9547a2ff

  l3 plugins for networking-odl and dragonflow seem to have similar code
  and would be affected too.

  eg.
  http://logs.openstack.org/87/199387/36/check/gate-tempest-dsvm-networking-midonet-ml2/ceb0331/logs/q-svc.txt.gz?level=TRACE

  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource [req-af588d02-2944-411f-aa22-eafca4fdabeb tempest-TestSecurityGroupsBasicOps-509565194 -] remove_router_interface failed: No details.
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 217, in _handle_action
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/networking-midonet/midonet/neutron/services/l3/l3_midonet.py", line 190, in remove_router_interface
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, interface_info)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1756, in remove_router_interface
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, interface_info)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 924, in remove_router_interface
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     context, router_id, subnet_id, device_owner)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 901, in _remove_interface_by_subnet
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     l3_port_check=False)
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 611, in inner
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource     raise RuntimeError(_("Method cannot be called within a "
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource RuntimeError: Method cannot be called within a transaction.
  2016-08-09 12:31:57.844 25876 ERROR neutron.api.v2.resource 
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource [req-c9ae4bf8-2baf-4327-be58-bb3006b4d9c9 tempest-TestSecurityGroupsBasicOps-2112515119 -] delete failed: No details.
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 522, in delete
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 544, in _delete
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/networking-midonet/midonet/neutron/services/l3/l3_midonet.py", line 220, in delete_floatingip
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     super(MidonetL3ServicePlugin, self).delete_floatingip(context, id)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1779, in delete_floatingip
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     floating_ip = self._delete_floatingip(context, id)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1301, in _delete_floatingip
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     l3_port_check=False)
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 611, in inner
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource     raise RuntimeError(_("Method cannot be called within a "
  2016-08-09 12:31:58.293 25876 ERROR neutron.api.v2.resource RuntimeError: Method cannot be called within a transaction.

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-midonet/+bug/1611533/+subscriptions
Follow ups

[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: Isaku Yamahata, 2017-08-18
[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: Omer Anson, 2016-08-31
[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: Armando Migliaccio, 2016-08-22
[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: YAMAMOTO Takashi, 2016-08-22
[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: OpenStack Infra, 2016-08-12
[Bug 1611533] Re: ml2 transaction_guard broke out of tree plugins
From: YAMAMOTO Takashi, 2016-08-09