yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1543094] [NEW] [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on create_router call

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pavel Bondar <pbondar@xxxxxxxxxxxx>
Date: Mon, 08 Feb 2016 12:35:48 -0000
Reply-to: Bug 1543094 <1543094@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Observed errors "DB exceeded retry limit" [1] in cases where pluggable ipam is enabled, observed on master branch.
Each time retest is done different tests are failed, so looks like concurency issue.
4  errors 'DB exceeded retry limit' are observed in [1].

2016-02-04 11:55:59.944 15476 ERROR oslo_db.api [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] DB exceeded retry limit.
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api Traceback (most recent call last):
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return f(*args, **kwargs)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 519, in _create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     obj = do_create(body)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 501, in do_create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     request.context, reservation.reservation_id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 494, in do_create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return obj_creator(request.context, **kwargs)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 411, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self).create_router(context, router)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 200, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self.delete_router(context, router_db.id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 196, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     gw_info, router=router_db)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_gwmode_db.py", line 69, in _update_router_gw_info
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, router_id, info, router=router)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 429, in _update_router_gw_info
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 185, in _create_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 399, in _create_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     new_network_id, ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 310, in _create_router_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context.elevated(), {'port': port_data})
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/common/utils.py", line 149, in create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return core_plugin.create_port(context, {'port': port_data})
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1069, in create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1045, in _create_port_db
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result = super(Ml2Plugin, self).create_port(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1193, in create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port_id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     revert_on_fail=False)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 156, in allocate_ips_for_port_and_store
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 228, in _allocate_ips_for_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in _ipam_allocate_ips
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     "external system for %s"), addresses)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in _ipam_allocate_ips
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, ipam_driver, port, ip_list)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in _ipam_allocate_single_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port, subnet),
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in _ipam_try_allocate_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return ipam_subnet.allocate(ip_request)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 350, in allocate
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     auto_generated)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 219, in _allocate_specific_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     session, db_range, first_ip=first_ip, last_ip=last_ip)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 167, in update_range
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     raise db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api RetryRequest
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api 
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] create failed
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 408, in create
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "<string>", line 2, in reraise
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource 


[1] http://logs.openstack.org/23/181023/54/check/gate-neutron-dsvm-api/054071e/logs/screen-q-svc.txt.gz#_2016-02-04_11_55_59_944

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1543094

Title:
  [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on
  create_router call

Status in neutron:
  New

Bug description:
  Observed errors "DB exceeded retry limit" [1] in cases where pluggable ipam is enabled, observed on master branch.
  Each time retest is done different tests are failed, so looks like concurency issue.
  4  errors 'DB exceeded retry limit' are observed in [1].

  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] DB exceeded retry limit.
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api Traceback (most recent call last):
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return f(*args, **kwargs)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 519, in _create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     obj = do_create(body)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 501, in do_create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     request.context, reservation.reservation_id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 494, in do_create
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return obj_creator(request.context, **kwargs)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 411, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self).create_router(context, router)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 200, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     self.delete_router(context, router_db.id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 196, in create_router
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     gw_info, router=router_db)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_gwmode_db.py", line 69, in _update_router_gw_info
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, router_id, info, router=router)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 429, in _update_router_gw_info
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 185, in _create_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 399, in _create_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     new_network_id, ext_ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 310, in _create_router_gw_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context.elevated(), {'port': port_data})
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/common/utils.py", line 149, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return core_plugin.create_port(context, {'port': port_data})
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1069, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1045, in _create_port_db
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     result = super(Ml2Plugin, self).create_port(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1193, in create_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port_id)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     revert_on_fail=False)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 156, in allocate_ips_for_port_and_store
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 228, in _allocate_ips_for_port
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in _ipam_allocate_ips
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     "external system for %s"), addresses)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in _ipam_allocate_ips
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     context, ipam_driver, port, ip_list)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in _ipam_allocate_single_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     port, subnet),
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in _ipam_try_allocate_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     return ipam_subnet.allocate(ip_request)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 350, in allocate
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     auto_generated)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 219, in _allocate_specific_ip
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     session, db_range, first_ip=first_ip, last_ip=last_ip)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api   File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 167, in update_range
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api     raise db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api RetryRequest
  2016-02-04 11:55:59.944 15476 ERROR oslo_db.api 
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource [req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] create failed
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 408, in create
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource   File "<string>", line 2, in reraise
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
  2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource 

  
  [1] http://logs.openstack.org/23/181023/54/check/gate-neutron-dsvm-api/054071e/logs/screen-q-svc.txt.gz#_2016-02-04_11_55_59_944

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1543094/+subscriptions
Follow ups

[Bug 1543094] Re: [Pluggable IPAM] DB exceeded retry limit (RetryRequest) on create_router call
From: OpenStack Infra, 2016-06-09