yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1394351] Re: deadlock when delete port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Robert Collins <1394351@xxxxxxxxxxxxxxxxxx>
Date: Wed, 11 Feb 2015 00:30:26 -0000
Reply-to: Bug 1394351 <1394351@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** No longer affects: nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1394351

Title:
  deadlock when delete port

Status in OpenStack Neutron (virtual network service):
  Confirmed

Bug description:
  netdemoid=$(neutron net-list | awk '{if($4=="'demo-net'"){print $2;}}')
  subnetdemoid=$(neutron subnet-list | awk '{if($4=="'demo-subnet'"){print $2;}}')

  exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}')
  for i in `seq 1 10`; do
      #boot vm, and create floating ip
      nova boot --image cirros --flavor m1.tiny --nic net-id=$netdemoid cirrosdemo${i}
      cirrosdemoid[i]=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $2;}}')
      output=$(neutron floatingip-create $exnetid)
      echo $output
      floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}')
      floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print $4;}}')a
  done

  # Setup router
  neutron router-gateway-set $routerdemoid $exnetid
  neutron router-interface-add demo-router $subnetdemoid
  #wait for VM to be running
  sleep 30

  for i in `seq 1 10`; do
      cirrosfix=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $12;}}')
      cirrosfixip=${cirrosfix#*=}
      output=$(neutron port-list | grep ${cirrosfixip})
      echo $output
      portid=$(echo "$output" | awk '{print $2;}')
      neutron floatingip-associate --fixed-ip-address $cirrosfixip ${floatipid[i]} $portid
      neutron floatingip-delete ${floatipid[i]}
      nova delete ${cirrosdemoid[i]}
  done

  
  With several tries, I have one instance in ERROR state:
  2014-11-19 19:41:02.670 8659 DEBUG neutron.context [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/context.py:83
  2014-11-19 19:41:02.671 8659 DEBUG neutron.plugins.ml2.rpc [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Device 498e7a54-22dd-4e5b-a8db-d6bffb8edd25 details requested by agent ovs-agent-overcloud-controller0-d5wwhbhhtlmp with host overcloud-controller0-d5wwhbhhtlmp get_device_details /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:90
  2014-11-19 19:41:02.707 8659 DEBUG neutron.openstack.common.lockutils [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Got semaphore "db-access" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:168
  2014-11-19 19:41:04.061 8658 ERROR oslo.messaging.rpc.dispatcher [req-4303cd41-c87c-44aa-b78a-549fb914ac9c ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 220, in report_state
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.plugin.create_or_update_agent(context, agent_state)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 180, in create_or_update_agent
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     return self._create_or_update_agent(context, agent)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 174, in _create_or_update_agent
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     greenthread.sleep(0)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.rollback()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     t[1].commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self._do_commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.connection._commit_impl()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self._handle_dbapi_exception(e, None, None, None, None)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     exc_info
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     reraise(type(exception), exception, tb=exc_tb)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     self.engine.dialect.do_commit(self.connection)
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher     dbapi_connection.commit()
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
  2014-11-19 19:41:04.071 8659 ERROR oslo.messaging.rpc.dispatcher [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     for device in kwargs.pop('devices', [])
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     host)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     updated = True
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.gen.next()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/contextlib.py", line 121, in nested
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     if exit(*exc):
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.rollback()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     t[1].commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self._do_commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.connection._commit_impl()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self._handle_dbapi_exception(e, None, None, None, None)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     exc_info
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     reraise(type(exception), exception, tb=exc_tb)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     self.engine.dialect.do_commit(self.connection)
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher     dbapi_connection.commit()
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None
  2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
  2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Returning exception (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None to caller
  2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] ['Traceback (most recent call last):\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list\n    for device in kwargs.pop(\'devices\', [])\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details\n    host)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status\n    updated = True\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/contextlib.py", line 121, in nested\n    if exit(*exc):\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__\n    self.rollback()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__\n    self.commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit\n    t[1].commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit\n    self._do_commit()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit\n    self.connection._commit_impl()\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl\n    self._handle_dbapi_exception(e, None, None, None, None)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception\n    exc_info\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl\n    self.engine.dialect.do_commit(self.connection)\n', '  File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit\n    dbapi_connection.commit()\n', "OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None\n"]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1394351/+subscriptions
References

[Bug 1394351] [NEW] deadlock when delete port
From: Zengfa Gao, 2014-11-19