yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1642203] Re: [ stable/newton] Port creation failed with error "DBDuplicateEntry: (pymysql.err.IntegrityError)" on 100 concurrency

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1642203@xxxxxxxxxxxxxxxxxx>
Date: Tue, 22 Nov 2016 11:12:24 -0000
Reply-to: Bug 1642203 <1642203@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Incomplete => Triaged

** Changed in: neutron
   Importance: Undecided => Medium

** Project changed: neutron => vmware-nsx

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1642203

Title:
  [ stable/newton] Port creation failed with error "DBDuplicateEntry:
  (pymysql.err.IntegrityError)" on 100 concurrency

Status in vmware-nsx:
  Triaged

Bug description:
  [stable/newton] Port creation failed with error "DBDuplicateEntry:
  (pymysql.err.IntegrityError)" on 100 concurrency

  With stable/newton,executed below scenario with 100 concurrency and
  500 iterations:

   -Create network, subnet
   -Create port (device_owner->compute:None)

  For about 40 iterations, the scenario is failing with below error

  "DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate
  entry '70.1.1.4-7005b2ac-9431-41f3-a59a-f6d19589b03c' for key
  'PRIMARY'") [SQL: u'INSERT INTO ipamallocations (ip_address, status,
  ipam_subnet_id) VALUES (%(ip_address)s, %(status)s,
  %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED',
  'ip_address': '70.1.1.4', 'ipam_subnet_id': u'7005b2ac-9431-41f3-a59a-
  f6d19589b03c'}]"

  I have attached the neutron.conf for your reference.

  Neutron logs:

  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource [req-9cd01b4e-a770-4309-bd8c-987062cf4dde c_rally_94c739d9_QlP59G01 -] create failed: No details.
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 430, in create
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     traceback.format_exc())
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 543, in _create
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     obj = do_create(body)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 525, in do_create
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 518, in do_create
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1769, in create_port
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     neutron_db = super(NsxV3Plugin, self).create_port(context, port)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 159, in wrapped
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1138, in create_port
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     db_port = self.create_port_db(context, port)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1173, in create_port_db
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     context, port, port_id)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 191, in allocate_ips_for_port_and_store
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     revert_on_fail=False)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     ips = self._allocate_ips_for_port(context, port_copy)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 235, in _allocate_ips_for_port
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return self._ipam_allocate_ips(context, ipam_driver, p, ips)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 147, in _ipam_allocate_ips
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     "external system for %s"), addresses)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 121, in _ipam_allocate_ips
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     ip_address, subnet_id = ipam_allocator.allocate(ip_request)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/ipam/subnet_alloc.py", line 232, in allocate
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return ipam_subnet.allocate(address_request), subnet_id
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 210, in allocate
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.subnet_manager.create_allocation(session, ip_address)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.rollback()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.commit()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self._prepare_impl()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self.session.flush()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self._flush(objects)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137, in _flush
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     transaction.rollback(_capture_exception=True)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2101, in _flush
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     flush_context.execute()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     rec.execute(self)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     uow
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     mapper, table, insert)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 767, in _emit_insert_statements
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     execute(statement, multiparams)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return meth(self, multiparams, params)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     return connection._execute_clauseelement(self, multiparams, params)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     compiled_sql, distilled_params
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     context)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     context)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 167, in execute
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     result = self._query(query)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 323, in _query
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     conn.query(q)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 836, in query
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1020, in _read_query_result
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     result.read()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1303, in read
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 982, in _read_packet
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     packet.check_error()
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     _check_mysql_exception(errinfo)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource     raise errorclass(errno, errorvalue)
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '70.1.1.4-7005b2ac-9431-41f3-a59a-f6d19589b03c' for key 'PRIMARY'") [SQL: u'INSERT INTO ipamallocations (ip_address, status, ipam_subnet_id) VALUES (%(ip_address)s, %(status)s, %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED', 'ip_address': '70.1.1.4', 'ipam_subnet_id': u'7005b2ac-9431-41f3-a59a-f6d19589b03c'}]
  2016-11-16 18:32:28.715 5697 ERROR neutron.api.v2.resource

To manage notifications about this bug go to:
https://bugs.launchpad.net/vmware-nsx/+bug/1642203/+subscriptions
References

[Bug 1642203] [NEW] [ stable/newton] Port creation failed with error "DBDuplicateEntry: (pymysql.err.IntegrityError)" on 100 concurrency
From: Sujai, 2016-11-16