yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622616] [NEW] delete_subnet update_port appears racey with ipam

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1622616@xxxxxxxxxxxxxxxxxx>
Date: Mon, 12 Sep 2016 13:43:43 -0000
Reply-to: Bug 1622616 <1622616@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

failure spotted in a patch on a delete_subnet call:


2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     traceback.format_exc())
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "port %s"), port_id)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.update_port(context, port_id, data)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     port)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port=new_port)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port.get('mac_address'))
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_mac)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     changes.remove)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "external system for %s"), addresses)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ipam_subnet.deallocate(ip['ip_address'])
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ip_address=address)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource 


http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452

** Affects: neutron
     Importance: High
         Status: New


** Tags: l3-ipam-dhcp

** Changed in: neutron
    Milestone: None => newton-rc1

** Changed in: neutron
   Importance: Undecided => High

** Tags added: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1622616

Title:
  delete_subnet update_port appears racey with ipam

Status in neutron:
  New

Bug description:
  failure spotted in a patch on a delete_subnet call:

  
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     traceback.format_exc())
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "port %s"), port_id)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.update_port(context, port_id, data)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port=new_port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port.get('mac_address'))
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_mac)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     changes.remove)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "external system for %s"), addresses)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ipam_subnet.deallocate(ip['ip_address'])
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ip_address=address)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource 


  http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1622616/+subscriptions
Follow ups

[Bug 1622616] Re: delete_subnet update_port appears racey with ipam
From: OpenStack Infra, 2016-09-24
[Bug 1622616] Re: delete_subnet update_port appears racey with ipam
From: Kevin Benton, 2016-09-14
[Bug 1622616] Re: delete_subnet update_port appears racey with ipam
From: OpenStack Infra, 2016-09-14