yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622616] Re: delete_subnet update_port appears racey with ipam

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1622616@xxxxxxxxxxxxxxxxxx>
Date: Sat, 24 Sep 2016 14:05:22 -0000
Reply-to: Bug 1622616 <1622616@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/373536
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=873b5ac837220c11e204c6610782f8b86c90bf03
Submitter: Jenkins
Branch:    master

commit 873b5ac837220c11e204c6610782f8b86c90bf03
Author: Armando Migliaccio <armamig@xxxxxxxxx>
Date:   Tue Sep 20 14:23:40 2016 -0700

    Retry port update on IpAddressAllocationNotFound
    
    If a port update and a subnet delete interleave, there is a
    chance that the IPAM update operation raises this exception.
    Rather than throwing that up to the user under some sort of
    conflict, bubble up a retry instead; that should bring things
    back to sanity.
    
    Closes-bug: #1622616
    
    Change-Id: Ia8cac09349d4cb722737bdf0bec6c54b9e77f31d


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1622616

Title:
  delete_subnet update_port appears racey with ipam

Status in neutron:
  Fix Released

Bug description:
  failure spotted in a patch on a delete_subnet call:

  
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     traceback.format_exc())
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "port %s"), port_id)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.update_port(context, port_id, data)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port=new_port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port.get('mac_address'))
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_mac)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     changes.remove)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "external system for %s"), addresses)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ipam_subnet.deallocate(ip['ip_address'])
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ip_address=address)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource 


  http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1622616/+subscriptions
References

[Bug 1622616] [NEW] delete_subnet update_port appears racey with ipam
From: Kevin Benton, 2016-09-12