yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1555387] Re: ipam driver not using retryrequest correctly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1555387@xxxxxxxxxxxxxxxxxx>
Date: Fri, 11 Mar 2016 02:27:06 -0000
Reply-to: Bug 1555387 <1555387@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/290922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2dc9d67f6e0fbec8272dc1dfdb594d05378536d5
Submitter: Jenkins
Branch:    master

commit 2dc9d67f6e0fbec8272dc1dfdb594d05378536d5
Author: Kevin Benton <kevin@xxxxxxxxxx>
Date:   Wed Mar 9 16:08:06 2016 -0800

    Construct exceptions before passing to retryrequest
    
    The IP allocation failure constructions weren't being constructed
    before being passed to RetryRequest so they would fail to be reraised
    after the retries were exhausted.
    
    This patch just constructs them and adjusts a few of the unit tests
    to execute the full retry life-cycle.
    
    Closes-Bug: #1555387
    Change-Id: I1c1ea488886ee3f4790eaa53911ea5845f327334


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1555387

Title:
  ipam driver not using retryrequest correctly

Status in neutron:
  Fix Released

Bug description:
  The IPAM driver is not passing a constructed exception into
  RetryRequest so the reraise on retry exhaustion fails:


  
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource [req-6e0e721f-12df-4132-81ef-f1b96875cdd7 c_rally_3c9bcbbd_m5PryrFI -] create failed
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 410, in create
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource   File "<string>", line 2, in reraise
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
  2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api [req-d357eb93-048c-425b-b6de-f860be1d767a c_rally_3c9bcbbd_IA8UUZUc -] DB exceeded retry limit.
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api Traceback (most recent call last):
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     return f(*args, **kwargs)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/api/v2/base.py", line 521, in _create
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     obj = do_create(body)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/api/v2/base.py", line 503, in do_create
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     request.context, reservation.reservation_id)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/api/v2/base.py", line 496, in do_create
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     return obj_creator(request.context, **kwargs)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 463, in create_router
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.delete_router(context, router_dict['id'])
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 457, in create_router
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     context, router_db)[1]
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 443, in _create_ha_interfaces_and_ensure_network
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     creator, dep_getter, dep_creator, dep_id_attr)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/common/utils.py", line 707, in create_object_with_dependency
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     ctx.reraise = False
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/common/utils.py", line 696, in create_object_with_dependency
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     result = creator(dependency)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 394, in _create_ha_interfaces
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     l3_port_check=False)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 388, in _create_ha_interfaces
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     router.tenant_id)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 375, in add_ha_port
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     deletion, content)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/common_db_mixin.py", line 55, in safe_creation
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     obj = create_fn()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/plugins/common/utils.py", line 161, in create_port
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     return core_plugin.create_port(context, {'port': port_data})
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1123, in create_port
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1094, in _create_port_db
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     result = super(Ml2Plugin, self).create_port(context, port)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1237, in create_port
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     port_id)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 179, in allocate_ips_for_port_and_store
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     revert_on_fail=False)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 163, in allocate_ips_for_port_and_store
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port_copy)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 235, in _allocate_ips_for_port
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in _ipam_allocate_ips
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     "external system for %s"), addresses)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     self.force_reraise()
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in _ipam_allocate_ips
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     context, ipam_driver, port, ip_list)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in _ipam_allocate_single_ip
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     port, subnet),
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in _ipam_try_allocate_ip
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     return ipam_subnet.allocate(ip_request)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 350, in allocate
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     auto_generated)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 219, in _allocate_specific_ip
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     session, db_range, first_ip=first_ip, last_ip=last_ip)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api   File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 167, in update_range
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api     raise db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api RetryRequest
  2016-03-09 16:06:10.210 98260 ERROR oslo_db.api
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource [req-d357eb93-048c-425b-b6de-f860be1d767a c_rally_3c9bcbbd_IA8UUZUc -] create failed
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 410, in create
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource   File "<string>", line 2, in reraise
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
  2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1555387/+subscriptions
References

[Bug 1555387] [NEW] ipam driver not using retryrequest correctly
From: Kevin Benton, 2016-03-10