yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1311970] [NEW] Concurrent delete-update raises database lock in linuxbridge-agent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Li Ma <skywalker.nick@xxxxxxxxx>
Date: Thu, 24 Apr 2014 03:21:56 -0000
Reply-to: Bug 1311970 <1311970@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I concurrently deleted more than 100 instances and then linuxbridge-
agent threw exception as below:

2014-04-22 20:35:53.435 494 DEBUG
neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-
ffc712fc-80af-4837-a068-6e1a076e4ebc None] port_removed failed for
tap2552f6fa-0a: Remote error: OperationalError (OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction') 'UPDATE ports
SET status=%s WHERE ports.id = %s' ('DOWN', '2552f6fa-
0a43-4193-b018-2c971fe4b0d7')

[u'Traceback (most recent call last):\n', u'
File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_zmq.py", line 314, in _get_response\n    data.get(\'namespace\'), **data[\'args\'])\n', u'
File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 178, in update_device_down\n    q_const.PORT_STATUS_DOWN)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 747, in update_port_status\n    original_port[\'network_id\'])\n', u'
File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 410, in get_network\n    result = super(Ml2Plugin, self).get_network(context, id, None)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 971, in get_network\n    network = self._get_network(context, id)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 245, in _get_network\n    network = self._get_by_id(context, models_v2.Network, id)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 144, in _get_by_id\n    return query.filter(model.id == id).one()\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one\n    ret = list(self)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2226, in __iter__\n    self.session._autoflush()\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1012, in _autoflush\n    self.flush()\n', u'
File "/usr/lib/python2.6/site-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 597, in _wrap\n    return f(*args, **kwargs)\n', u'
File "/usr/lib/python2.6/site-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 836, in flush\n    return super(Session, self).flush(*args, **kwargs)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1583, in flush\n    self._flush(objects)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1654, in _flush\n    flush_context.execute()\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute\n    rec.execute(self)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute\n    uow\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj\n    mapper, table, update)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements\n    execute(statement, params)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n    params)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n    context)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n    context)\n', u'
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n    cursor.execute(statement, parameters)\n', u'
File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute\n    self.errorhandler(self, exc, value)\n', u'
File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', u"OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE ports SET status=%s WHERE ports.id = %s' ('DOWN', '2552f6fa-0a43-4193-b018-2c971fe4b0d7')\n"]. treat_devices_removed /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:961

2014-04-22 20:35:53.436 494 ERROR
neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-
ffc712fc-80af-4837-a068-6e1a076e4ebc None] Error in agent loop. Devices
info: {'current': set(['tap6d4d95da-16', 'tap47a80904-8e',
'tapeca417e9-c6', 'tape372251c-f3', 'tapabed7b2b-6f']), 'removed': set
(['tap2552f6fa-0a']), 'added': set([])}

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1311970

Title:
  Concurrent delete-update raises database lock in linuxbridge-agent

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  I concurrently deleted more than 100 instances and then linuxbridge-
  agent threw exception as below:

  2014-04-22 20:35:53.435 494 DEBUG
  neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-
  ffc712fc-80af-4837-a068-6e1a076e4ebc None] port_removed failed for
  tap2552f6fa-0a: Remote error: OperationalError (OperationalError)
  (1205, 'Lock wait timeout exceeded; try restarting transaction')
  'UPDATE ports SET status=%s WHERE ports.id = %s' ('DOWN', '2552f6fa-
  0a43-4193-b018-2c971fe4b0d7')

  [u'Traceback (most recent call last):\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_zmq.py", line 314, in _get_response\n    data.get(\'namespace\'), **data[\'args\'])\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 178, in update_device_down\n    q_const.PORT_STATUS_DOWN)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 747, in update_port_status\n    original_port[\'network_id\'])\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 410, in get_network\n    result = super(Ml2Plugin, self).get_network(context, id, None)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 971, in get_network\n    network = self._get_network(context, id)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 245, in _get_network\n    network = self._get_by_id(context, models_v2.Network, id)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 144, in _get_by_id\n    return query.filter(model.id == id).one()\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one\n    ret = list(self)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2226, in __iter__\n    self.session._autoflush()\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1012, in _autoflush\n    self.flush()\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 597, in _wrap\n    return f(*args, **kwargs)\n', u'
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 836, in flush\n    return super(Session, self).flush(*args, **kwargs)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1583, in flush\n    self._flush(objects)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1654, in _flush\n    flush_context.execute()\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute\n    rec.execute(self)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute\n    uow\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj\n    mapper, table, update)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements\n    execute(statement, params)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n    params)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n    context)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n    context)\n', u'
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n    cursor.execute(statement, parameters)\n', u'
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute\n    self.errorhandler(self, exc, value)\n', u'
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', u"OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE ports SET status=%s WHERE ports.id = %s' ('DOWN', '2552f6fa-0a43-4193-b018-2c971fe4b0d7')\n"]. treat_devices_removed /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:961

  2014-04-22 20:35:53.436 494 ERROR
  neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-
  ffc712fc-80af-4837-a068-6e1a076e4ebc None] Error in agent loop.
  Devices info: {'current': set(['tap6d4d95da-16', 'tap47a80904-8e',
  'tapeca417e9-c6', 'tape372251c-f3', 'tapabed7b2b-6f']), 'removed': set
  (['tap2552f6fa-0a']), 'added': set([])}

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1311970/+subscriptions
Follow ups

[Bug 1311970] [NEW] Concurrent delete-update raises database lock in linuxbridge-agent
From: Li Ma, 2014-04-24
References

[Bug 1311970] [NEW] Concurrent delete-update raises database lock in linuxbridge-agent
From: Li Ma, 2014-04-24