yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622616] Re: delete_subnet update_port appears racey with ipam

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1622616@xxxxxxxxxxxxxxxxxx>
Date: Wed, 14 Sep 2016 01:46:31 -0000
Reply-to: Bug 1622616 <1622616@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/369051
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f07c07b16fb0858c45f6cef135a8d8c07a16c505
Submitter: Jenkins
Branch:    master

commit f07c07b16fb0858c45f6cef135a8d8c07a16c505
Author: Carl Baldwin <carl.baldwin@xxxxxxx>
Date:   Mon Sep 12 15:31:08 2016 -0600

    Don't allocate IP on port update when existing subnet specified
    
    If a port update specifies only a subnet_id for a fixed_ip then we
    want to look at existing fixed_ips to see if that subnet_id is already
    there. This avoids allocating a new IP address on the subnet and
    deallocating the old one.
    
    Without some special care, this breaks the code path for prefix
    delegation. One could argue that PD needs reworking. However, as a
    stop-gap measure, we still run the old code path if the address is an
    EUI-64 address. This allows PD to continue to work as it was
    originally written and it doesn't do any harm because allocating
    EUI-64 addresses is repeatable.
    
    This commit removes a test case from the DNS integration tests. The
    test was specifically testing that DNS records we updated in the case
    where a subnet id was passed to re-allocate a fixed_ip. Since the use
    case no longer works, the test doesn't make sense.
    
    Change-Id: I887cd23cf65a1df9bd1d59ab897a1ecd005a588c
    Closes-Bug: #1622616


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1622616

Title:
  delete_subnet update_port appears racey with ipam

Status in neutron:
  Fix Released

Bug description:
  failure spotted in a patch on a delete_subnet call:

  
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     traceback.format_exc())
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "port %s"), port_id)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.update_port(context, port_id, data)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     return f(self, context, *args, **kwargs)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port=new_port)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_port.get('mac_address'))
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     new_mac)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     changes.remove)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     "external system for %s"), addresses)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ipam_subnet.deallocate(ip['ip_address'])
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource     ip_address=address)
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55
  2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource 


  http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1622616/+subscriptions
References

[Bug 1622616] [NEW] delete_subnet update_port appears racey with ipam
From: Kevin Benton, 2016-09-12