yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1725587] [NEW] rally create_and_list_floating_ips task fails with duplicate IpamAllocation

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Huang Cheng <1725587@xxxxxxxxxxxxxxxxxx>
Date: Sat, 21 Oct 2017 07:20:58 -0000
Reply-to: Bug 1725587 <1725587@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

RDO OpenStack Ocata

* rally result

Total durations
Action	Min (sec)	Median (sec)	90%ile (sec)	95%ile (sec)	Max (sec)	Avg (sec)	Success	Count
neutron.create_floating_ip	4.2	18.946	71.561	91.803	131.663	29.769	94.2%	500
-> neutron.list_networks	0.991	1.837	10.769	11.56	12.955	3.366	94.2%	500
neutron.list_floating_ips	0.162	0.95	1.672	1.884	3.009	1.007	100.0%	471
total	4.396	20.196	71.896	92.886	131.663	30.717	94.2%	500
-> duration	4.396	20.196	71.896	92.886	131.663	30.717	94.2%	500
-> idle_duration	0	0	0	0	0	0	94.2%	500

html report here: https://pastebin.com/9bLJVhfS

* neutron-server log

2017-10-21 10:50:50.445 20020 ERROR oslo_db.api [req-f4c8db2c-d80d-475a-b408-6eafd6701b62 07a6aba4b1a44dbc9453ddddbfbe4a65 bb0f198d854c440d8f7c0beb27eaae2b - - -] DB exceeded retry limit.
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api Traceback (most recent call last):
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return f(*args, **kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     traceback.format_exc())
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return f(*dup_args, **dup_kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1192, in create_port
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1163, in _create_port_db
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     port_db = self.create_port_db(context, port)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1221, in create_port_db
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     context, port, port_id)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 191, in allocate_ips_for_port_and_store
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     revert_on_fail=False)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port_copy)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 227, in _allocate_ips_for_port
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 147, in _ipam_allocate_ips
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     "external system for %s"), addresses)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 121, in _ipam_allocate_ips
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ip_address, subnet_id = ipam_allocator.allocate(ip_request)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/subnet_alloc.py", line 232, in allocate
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return ipam_subnet.allocate(address_request), subnet_id
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 210, in allocate
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ip_address)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 127, in create_allocation
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ipam_subnet_id=self._ipam_subnet_id).create()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     res = func(self, *args, **kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 582, in create
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     object_class=self.__class__, db_exception=db_exc)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api NeutronDbObjectDuplicateEntry: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 111.0.0.204-f9f8245b-6920-4a51-a21b-76e0ab8e2984
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1725587

Title:
  rally create_and_list_floating_ips task fails with duplicate
  IpamAllocation

Status in neutron:
  New

Bug description:
  Description
  ===========

  RDO OpenStack Ocata

  * rally result

  Total durations
  Action	Min (sec)	Median (sec)	90%ile (sec)	95%ile (sec)	Max (sec)	Avg (sec)	Success	Count
  neutron.create_floating_ip	4.2	18.946	71.561	91.803	131.663	29.769	94.2%	500
  -> neutron.list_networks	0.991	1.837	10.769	11.56	12.955	3.366	94.2%	500
  neutron.list_floating_ips	0.162	0.95	1.672	1.884	3.009	1.007	100.0%	471
  total	4.396	20.196	71.896	92.886	131.663	30.717	94.2%	500
  -> duration	4.396	20.196	71.896	92.886	131.663	30.717	94.2%	500
  -> idle_duration	0	0	0	0	0	0	94.2%	500

  html report here: https://pastebin.com/9bLJVhfS

  * neutron-server log

  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api [req-f4c8db2c-d80d-475a-b408-6eafd6701b62 07a6aba4b1a44dbc9453ddddbfbe4a65 bb0f198d854c440d8f7c0beb27eaae2b - - -] DB exceeded retry limit.
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api Traceback (most recent call last):
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return f(*args, **kwargs)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     traceback.format_exc())
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return f(*dup_args, **dup_kwargs)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1192, in create_port
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     result, mech_context = self._create_port_db(context, port)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1163, in _create_port_db
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     port_db = self.create_port_db(context, port)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1221, in create_port_db
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     context, port, port_id)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 191, in allocate_ips_for_port_and_store
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     revert_on_fail=False)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ips = self._allocate_ips_for_port(context, port_copy)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 227, in _allocate_ips_for_port
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 147, in _ipam_allocate_ips
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     "external system for %s"), addresses)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     self.force_reraise()
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 121, in _ipam_allocate_ips
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ip_address, subnet_id = ipam_allocator.allocate(ip_request)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/subnet_alloc.py", line 232, in allocate
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     return ipam_subnet.allocate(address_request), subnet_id
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 210, in allocate
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ip_address)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 127, in create_allocation
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     ipam_subnet_id=self._ipam_subnet_id).create()
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     res = func(self, *args, **kwargs)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api   File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 582, in create
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api     object_class=self.__class__, db_exception=db_exc)
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api NeutronDbObjectDuplicateEntry: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 111.0.0.204-f9f8245b-6920-4a51-a21b-76e0ab8e2984
  2017-10-21 10:50:50.445 20020 ERROR oslo_db.api

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1725587/+subscriptions
Follow ups

[Bug 1725587] Re: rally create_and_list_floating_ips task fails with duplicate IpamAllocation
From: Brian Haley, 2023-01-20