yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1822382] Re: DBDeadlock for INSERT INTO resourcedeltas

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Michael Johnson <johnsomor@xxxxxxxxx>
Date: Fri, 29 Mar 2019 21:44:41 -0000
Reply-to: Bug 1822382 <1822382@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Looking at this deeper, it appears neutron did properly retry this DB
action and the instance connection issue may be unrelated. Marking this
invalid.

** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1822382

Title:
  DBDeadlock for INSERT INTO resourcedeltas

Status in neutron:
  Invalid

Bug description:
  Recently we started seeing instances fail to become reachable in the
  Octavia tempest jobs. This is intermittent, but recurring. This may be
  related to other DBDeadlock bugs recently reported for quotas, but
  since the SQL is different here I am reporting it.

  This is on Master/Train.

  Summary of the error in q-svc:

  Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340 neutron-
  server[11470]: ERROR oslo_db.sqlalchemy.exc_filters
  oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213,
  'Deadlock found when trying to get lock; try restarting transaction')
  [SQL: 'INSERT INTO resourcedeltas (resource, reservation_id, amount)
  VALUES (%(resource)s, %(reservation_id)s, %(amount)s)'] [parameters:
  {'reservation_id': '4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource':
  'port', 'amount': 1}] (Background on this error at:
  http://sqlalche.me/e/2j85)

  Full traceback from q-svc for once occurrence:

  Mar 29 20:04:12.790909 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters [req-56d6484e-b182-4dbd-8bb9-8db4ceb3c38a req-ddb65494-cdaf-4dec-ab19-84efbede0da7 admin admin] DBAPIError exception wrapped from (pymysql.err.InternalError) (1305, 'SAVEPOINT sa_savepoint_9 does not exist') [SQL: 'ROLLBACK TO SAVEPOINT sa_savepoint_9'] (Background on this error at: http://sqlalche.me/e/2j85): pymysql.err.InternalError: (1305, 'SAVEPOINT sa_savepoint_9 does not exist')
  Mar 29 20:04:12.791441 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  Mar 29 20:04:12.791889 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  Mar 29 20:04:12.792380 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     context)
  Mar 29 20:04:12.792860 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
  Mar 29 20:04:12.793296 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  Mar 29 20:04:12.793872 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in execute
  Mar 29 20:04:12.794320 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  Mar 29 20:04:12.794743 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
  Mar 29 20:04:12.795219 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  Mar 29 20:04:12.795668 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in query
  Mar 29 20:04:12.796102 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Mar 29 20:04:12.796505 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
  Mar 29 20:04:12.796904 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  Mar 29 20:04:12.797336 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in read
  Mar 29 20:04:12.797730 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  Mar 29 20:04:12.798022 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in _read_packet
  Mar 29 20:04:12.798305 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
  Mar 29 20:04:12.798600 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in check_error
  Mar 29 20:04:12.798894 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
  Mar 29 20:04:12.799224 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  Mar 29 20:04:12.799516 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
  Mar 29 20:04:12.799802 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters pymysql.err.InternalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')
  Mar 29 20:04:12.800094 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters 
  Mar 29 20:04:12.800509 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters The above exception was the direct cause of the following exception:
  Mar 29 20:04:12.800944 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters 
  Mar 29 20:04:12.801332 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  Mar 29 20:04:12.801636 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2333, in _flush
  Mar 29 20:04:12.801922 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     flush_context.execute()
  Mar 29 20:04:12.802210 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
  Mar 29 20:04:12.802555 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     rec.execute(self)
  Mar 29 20:04:12.802851 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
  Mar 29 20:04:12.803240 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     uow
  Mar 29 20:04:12.803792 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
  Mar 29 20:04:12.804108 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     mapper, table, insert)
  Mar 29 20:04:12.804435 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 830, in _emit_insert_statements
  Mar 29 20:04:12.804743 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     execute(statement, multiparams)
  Mar 29 20:04:12.805056 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
  Mar 29 20:04:12.805337 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     return meth(self, multiparams, params)
  Mar 29 20:04:12.805697 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
  Mar 29 20:04:12.806036 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     return connection._execute_clauseelement(self, multiparams, params)
  Mar 29 20:04:12.806449 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  Mar 29 20:04:12.806739 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     compiled_sql, distilled_params
  Mar 29 20:04:12.807148 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  Mar 29 20:04:12.807480 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     context)
  Mar 29 20:04:12.807773 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
  Mar 29 20:04:12.808062 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     util.raise_from_cause(newraise, exc_info)
  Mar 29 20:04:12.808388 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  Mar 29 20:04:12.808732 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  Mar 29 20:04:12.809104 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
  Mar 29 20:04:12.809390 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     raise value.with_traceback(tb)
  Mar 29 20:04:12.809730 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  Mar 29 20:04:12.810057 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     context)
  Mar 29 20:04:12.810334 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
  Mar 29 20:04:12.810632 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  Mar 29 20:04:12.811087 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 165, in execute
  Mar 29 20:04:12.811477 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  Mar 29 20:04:12.811776 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 321, in _query
  Mar 29 20:04:12.812094 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  Mar 29 20:04:12.812378 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 860, in query
  Mar 29 20:04:12.812661 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Mar 29 20:04:12.812937 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1061, in _read_query_result
  Mar 29 20:04:12.813324 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  Mar 29 20:04:12.813722 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1349, in read
  Mar 29 20:04:12.814133 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  Mar 29 20:04:12.814666 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1018, in _read_packet
  Mar 29 20:04:12.815045 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
  Mar 29 20:04:12.815417 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 384, in check_error
  Mar 29 20:04:12.815711 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
  Mar 29 20:04:12.815990 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  Mar 29 20:04:12.816272 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
  Mar 29 20:04:12.816598 ubuntu-xenial-rax-dfw-0004550340 neutron-server[11470]: ERROR oslo_db.sqlalchemy.exc_filters oslo_db.exception.DBDeadlock: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'INSERT INTO resourcedeltas (resource, reservation_id, amount) VALUES (%(resource)s, %(reservation_id)s, %(amount)s)'] [parameters: {'reservation_id': '4f198b7d-ac31-42bb-98bd-686c830322ab', 'resource': 'port', 'amount': 1}] (Background on this error at: http://sqlalche.me/e/2j85)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1822382/+subscriptions
References

[Bug 1822382] [NEW] DBDeadlock for INSERT INTO resourcedeltas
From: Michael Johnson, 2019-03-29