yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1422504] Re: floating ip delete deadlock

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1422504@xxxxxxxxxxxxxxxxxx>
Date: Sat, 14 Nov 2015 10:33:58 -0000
Reply-to: Bug 1422504 <1422504@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron/juno
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1422504

Title:
  floating ip delete deadlock

Status in neutron:
  Fix Released
Status in neutron juno series:
  New
Status in neutron kilo series:
  New

Bug description:
  rdo juno:

  2015-02-16 13:54:11.772 3612 ERROR neutron.api.v2.resource [req-5c6e13d3-56d6-476b-a961-e767aea637e5 None] delete failed
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 87, in resource
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 476, in delete
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 183, in delete_floatingip
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     self).delete_floatingip(context, id)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1178, in delete_floatingip
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     router_id = self._delete_floatingip(context, id)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 840, in _delete_floatingip
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     l3_port_check=False)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 984, in delete_port
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     port_db, binding = db.get_locked_port_and_binding(session, id)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py", line 141, in get_locked_port_and_binding
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     with_lockmode('update').
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2369, in one
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     ret = list(self)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2411, in __iter__
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     self.session._autoflush()
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1198, in _autoflush
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     self.flush()
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     self._flush(objects)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     transaction.rollback(_capture_exception=True)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     compat.reraise(exc_type, exc_value, exc_tb)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     flush_context.execute()
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     rec.execute(self)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 555, in execute
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     uow
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 117, in delete_obj
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     cached_connections, mapper, table, delete)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 694, in _emit_delete_statements
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     c = connection.execute(statement, del_objects)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     return meth(self, multiparams, params)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     return connection._execute_clauseelement(self, multiparams, params)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     compiled_sql, distilled_params
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     context)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     context)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     cursor.execute(statement, parameters)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'DELETE FROM floatingips WHERE floatingips.id = %s' ('1423b851-2a7e-471f-b790-c35c7dade6cf',)
  2015-02-16 13:54:11.772 3612 TRACE neutron.api.v2.resource
  2015-02-16 13:54:11.784 3612 INFO neutron.wsgi [req-5c6e13d3-56d6-476b-a961-e767aea637e5 None] 172.20.96.65 - - [16/Feb/2015 13:54:11] "DELETE //v2.0/floatingips/1423b851-2a7e-471f-b790-c35c7dade6cf.json HTTP/1.1" 500 359 4.735289

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1422504/+subscriptions
References

[Bug 1422504] [NEW] floating ip delete deadlock
From: Kevin Fox, 2015-02-16