yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1394351] Re: deadlock when delete port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Armando Migliaccio <1394351@xxxxxxxxxxxxxxxxxx>
Date: Wed, 17 Aug 2016 23:23:00 -0000
Reply-to: Bug 1394351 <1394351@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (INCUBATOR-JUNO, LIBERTY, MITAKA, NEWTON).
  Valid example: CONFIRMED FOR: INCUBATOR-JUNO


** Changed in: neutron
   Importance: Medium => Undecided

** Changed in: neutron
       Status: Confirmed => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1394351

Title:
  deadlock when delete port

Status in neutron:
  Expired

Bug description:
  netdemoid=$(neutron net-list | awk '{if($4=="'demo-net'"){print $2;}}')
  subnetdemoid=$(neutron subnet-list | awk '{if($4=="'demo-subnet'"){print $2;}}')

  exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}')
  for i in `seq 1 10`; do
      #boot vm, and create floating ip
      nova boot --image cirros --flavor m1.tiny --nic net-id=$netdemoid cirrosdemo${i}
      cirrosdemoid[i]=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $2;}}')
      output=$(neutron floatingip-create $exnetid)
      echo $output
      floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}')
      floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print $4;}}')a
  done

  # Setup router
  neutron router-gateway-set $routerdemoid $exnetid
  neutron router-interface-add demo-router $subnetdemoid
  #wait for VM to be running
  sleep 30

  for i in `seq 1 10`; do
      cirrosfix=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $12;}}')
      cirrosfixip=${cirrosfix#*=}
      output=$(neutron port-list | grep ${cirrosfixip})
      echo $output
      portid=$(echo "$output" | awk '{print $2;}')
      neutron floatingip-associate --fixed-ip-address $cirrosfixip ${floatipid[i]} $portid
      neutron floatingip-delete ${floatipid[i]}
      nova delete ${cirrosdemoid[i]}
  done

  
  With several tries, I have one instance in ERROR state:
  2014-11-19 19:41:02.670 8659 DEBUG neutron.context [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/context.py:83
  2014-11-19 19:41:02.671 8659 DEBUG neutron.plugins.ml2.rpc [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Device 498e7a54-22dd-4e5b-a8db-d6bffb8edd25 details requested by agent ovs-agent-overcloud-controller0-d5wwhbhhtlmp with host overcloud-controller0-d5wwhbhhtlmp get_device_details /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:90
  2014-11-19 19:41:02.707 8659 DEBUG neutron.openstack.common.lockutils [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Got semaphore "db-access" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:168
  2014-11-19 19:41:04.061 8658 ERROR oslo.messaging.rpc.dispatcher [req-4303cd41-c87c-44aa-b78a-549fb914ac9c ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 220, in report_state
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.plugin.create_or_update_agent(context, agent_state)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 180, in create_or_update_agent
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     return self._create_or_update_agent(context, agent)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 174, in _create_or_update_agent
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     greenthread.sleep(0)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.rollback()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     t[1].commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self._do_commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.connection._commit_impl()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self._handle_dbapi_exception(e, None, None, None, None)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     exc_info
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     reraise(type(exception), exception, tb=exc_tb)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.engine.dialect.do_commit(self.connection)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     dbapi_connection.commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
  2014-11-19 19:41:04.071 8659 ERROR oslo.messaging.rpc.dispatcher [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     for device in kwargs.pop('devices', [])
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     host)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     updated = True
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.gen.next()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 121, in nested
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     if exit(*exc):
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.rollback()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     t[1].commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self._do_commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.connection._commit_impl()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self._handle_dbapi_exception(e, None, None, None, None)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     exc_info
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     reraise(type(exception), exception, tb=exc_tb)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.engine.dialect.do_commit(self.connection)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     dbapi_connection.commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
  2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Returning exception (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None to caller
  2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] ['Traceback (most recent call last):\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list\n    for device in kwargs.pop(\'devices\', [])\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details\n    host)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status\n    updated = True\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/contextlib.py", line 121, in nested\n    if exit(*exc):\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__\n    self.rollback()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__\n    self.commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit\n    t[1].commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit\n    self._do_commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit\n    self.connection._commit_impl()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl\n    self._handle_dbapi_exception(e, None, None, None, None)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception\n    exc_info\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl\n    self.engine.dialect.do_commit(self.connection)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit\n    dbapi_connection.commit()\n', "OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None\n"]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1394351/+subscriptions
References

[Bug 1394351] [NEW] deadlock when delete port
From: Zengfa Gao, 2014-11-19