yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1337690] [NEW] Mysql/Eventlet deadlock when creating a port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Magesh GV <magesh.gv@xxxxxxx>
Date: Fri, 04 Jul 2014 04:06:01 -0000
Reply-to: Bug 1337690 <1337690@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Another instance of the Mysql/Eventlet deadlock which is occuring while
trying to  bind the network to a DHCP agent during port create

2014-07-02 22:17:43.322 31258 ERROR neutron.api.v2.resource [-] create failed
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 407, in create
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     obj)})
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 386, in notify
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     notifier_method)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 268, in _send_dhcp_notification
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._dhcp_agent_notifier.notify(context, data, methodname)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 153, in notify
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._notification(context, methodname, data, network_id)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 71, in _notification
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     chosen_agents = plugin.schedule_network(adminContext, network)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 207, in schedule_network
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self, context, created_network)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/scheduler/dhcp_agent_scheduler.py", line 83, in schedule
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._schedule_bind_network(context, agent, network['id'])
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.rollback()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 444, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.commit()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 354, in commit
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._prepare_impl()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.session.flush()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 524, in _wrap
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     return f(*args, **kwargs)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 718, in flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     return super(Session, self).flush(*args, **kwargs)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1818, in flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._flush(objects)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1936, in _flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     transaction.rollback(_capture_exception=True)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1900, in _flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     flush_context.execute()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     rec.execute(self)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     uow
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     table, insert)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 541, in _emit_insert_statements
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     execute(statement, multiparams)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     params)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compiled_sql, distilled_params
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     context)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     exc_info
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 195, in raise_from_cause
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     context)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     cursor.execute(statement, parameters)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO networkdhcpagentbindings (network_id, dhcp_agent_id) VALUES (%s, %s)' ('20a4c19b-7931-4815-8eb8-337ab158785f', '130e492d-8e74-46bf-a6b2-47e857f592c4')

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1337690

Title:
  Mysql/Eventlet deadlock when creating a port

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Another instance of the Mysql/Eventlet deadlock which is occuring
  while trying to  bind the network to a DHCP agent during port create

  2014-07-02 22:17:43.322 31258 ERROR neutron.api.v2.resource [-] create failed
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 407, in create
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     obj)})
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 386, in notify
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     notifier_method)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 268, in _send_dhcp_notification
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._dhcp_agent_notifier.notify(context, data, methodname)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 153, in notify
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._notification(context, methodname, data, network_id)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 71, in _notification
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     chosen_agents = plugin.schedule_network(adminContext, network)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 207, in schedule_network
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self, context, created_network)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/scheduler/dhcp_agent_scheduler.py", line 83, in schedule
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._schedule_bind_network(context, agent, network['id'])
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in __exit__
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.rollback()
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 444, in __exit__
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.commit()
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 354, in commit
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._prepare_impl()
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.session.flush()
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 524, in _wrap
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     return f(*args, **kwargs)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 718, in flush
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     return super(Session, self).flush(*args, **kwargs)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1818, in flush
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self._flush(objects)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1936, in _flush
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     transaction.rollback(_capture_exception=True)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1900, in _flush
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     flush_context.execute()
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     rec.execute(self)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     uow
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     table, insert)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 541, in _emit_insert_statements
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     execute(statement, multiparams)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     params)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     compiled_sql, distilled_params
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     context)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     exc_info
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 195, in raise_from_cause
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     context)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     cursor.execute(statement, parameters)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
  2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO networkdhcpagentbindings (network_id, dhcp_agent_id) VALUES (%s, %s)' ('20a4c19b-7931-4815-8eb8-337ab158785f', '130e492d-8e74-46bf-a6b2-47e857f592c4')

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1337690/+subscriptions
Follow ups

[Bug 1337690] Re: Mysql/Eventlet deadlock when creating a port
From: Launchpad Bug Tracker, 2014-09-28
[Bug 1337690] [NEW] Mysql/Eventlet deadlock when creating a port
From: Magesh GV, 2014-07-04
References

[Bug 1337690] [NEW] Mysql/Eventlet deadlock when creating a port
From: Magesh GV, 2014-07-04