yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1543094] Re: [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on create_router call

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1543094@xxxxxxxxxxxxxxxxxx>
Date: Thu, 09 Jun 2016 22:55:32 -0000
Reply-to: Bug 1543094 <1543094@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/292207
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=dcb2a931b5b84fb7aa41f08b37a5148bf6e987bc
Submitter: Jenkins
Branch:    master

commit dcb2a931b5b84fb7aa41f08b37a5148bf6e987bc
Author: Ryan Tidwell <ryan.tidwell@xxxxxxx>
Date:   Fri Apr 8 14:21:03 2016 -0700

    Compute IPAvailabilityRanges in memory during IP allocation
    
    This patch computes IP availability in memory without locking on
    IPAvailabilityRanges. IP availability is generated in memory, and
    to avoid contention an IP address is selected by randomly
    selecting from within the first 10 available IP addresses on a
    subnet. Raises IPAddressGenerationFailure if unable to allocate an
    IP address from within the window.
    
    Change-Id: I52e4485e832cbe6798de6b4afb6a7cfd88db11e2
    Depends-On: I84195b0eb63b7ca6a4e00becbe09e579ff8b718e
    Closes-Bug: #1543094


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1543094

Title:
  [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on
  create_router call

Status in neutron:
  Fix Released

Bug description:
  Observed errors "DB exceeded retry limit" [1] in cases where pluggable ipam is enabled, observed on master branch.
  Each time retest is done different tests are failed, so looks like concurency issue.
  4  errors 'DB exceeded retry limit' are observed in [1].

  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] DB exceeded retry limit.
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api Traceback (most recent call last):
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return f(*args, **kwargs)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 519, in _create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     obj = do_create(body)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 501, in do_create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     request.context, reservation.reservation_id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 494, in do_create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return obj_creator(request.context, **kwargs)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 411, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self).create_router(context, router)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 200, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self.delete_router(context, router_db.id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 196, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     gw_info, router=router_db)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_gwmode_db.py", line 69, in _update_router_gw_info
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, router_id, info, router=router)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 429, in _update_router_gw_info
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 185, in _create_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 399, in _create_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     new_network_id, ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 310, in _create_router_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context.elevated(), {'port': port_data})
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/common/utils.py", line 149, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return core_plugin.create_port(context, {'port': port_data})
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1069, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1045, in _create_port_db
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result = super(Ml2Plugin, self).create_port(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1193, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port_id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     revert_on_fail=False)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 156, in allocate_ips_for_port_and_store
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 228, in _allocate_ips_for_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in _ipam_allocate_ips
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     "external system for %s"), addresses)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in _ipam_allocate_ips
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, ipam_driver, port, ip_list)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in _ipam_allocate_single_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port, subnet),
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in _ipam_try_allocate_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return ipam_subnet.allocate(ip_request)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 350, in allocate
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     auto_generated)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 219, in _allocate_specific_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     session, db_range, first_ip=first_ip, last_ip=last_ip)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 167, in update_range
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     raise db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api RetryRequest
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api 
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] create failed
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 408, in create
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "<string>", line 2, in reraise
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource 

  
  [1] http://logs.openstack.org/23/181023/54/check/gate-neutron-dsvm-api/054071e/logs/screen-q-svc.txt.gz#_2016-02-04_11_55_59_944

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1543094/+subscriptions
References

[Bug 1543094] [NEW] [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on create_router call
From: Pavel Bondar, 2016-02-08