← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #19134

[Bug 1357102] [NEW] Big Switch: Multiple read calls to consistency DB fails

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The Big Switch consistency DB throws an exception if read_for_update() is called multiple times without closing the transaction in between. This was originally because there was a DB lock in place and a single thread could deadlock if it tried twice. However, 
there is no longer a point to this protection because the DB lock is gone and certain response failures result in the DB being read twice (the second time for a retry).

2014-08-14 21:56:41.496 12939 ERROR neutron.plugins.ml2.managers [req-ee311173-b38a-481e-8900-d963c676b05f None] Mechanism driver 'bigswitch' failed in update_port_postcommit
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers Traceback (most recent call last):
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 168, in _call_on_drivers
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_bigswitch/driver.py", line 91, in update_port_postcommit
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     port["network"]["id"], port)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 555, in rest_update_port
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     self.rest_create_port(tenant_id, net_id, port)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 545, in rest_create_port
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     self.rest_action('PUT', resource, data, errstr)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 476, in rest_action
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     timeout)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     return f(*args, **kwargs)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 423, in rest_call
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     hash_handler=hash_handler)
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 139, in rest_call
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     headers[HASH_MATCH_HEADER] = hash_handler.read_for_update()
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/db/consistency_db.py", line 56, in read_for_update
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     raise MultipleReadForUpdateCalls()
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers MultipleReadForUpdateCalls: Only one read_for_update call may be made at a time.
2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers

** Affects: neutron
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1357102

Title:
  Big Switch: Multiple read calls to consistency DB fails

Status in OpenStack Neutron (virtual network service):
  In Progress

Bug description:
  The Big Switch consistency DB throws an exception if read_for_update() is called multiple times without closing the transaction in between. This was originally because there was a DB lock in place and a single thread could deadlock if it tried twice. However, 
  there is no longer a point to this protection because the DB lock is gone and certain response failures result in the DB being read twice (the second time for a retry).

  2014-08-14 21:56:41.496 12939 ERROR neutron.plugins.ml2.managers [req-ee311173-b38a-481e-8900-d963c676b05f None] Mechanism driver 'bigswitch' failed in update_port_postcommit
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers Traceback (most recent call last):
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 168, in _call_on_drivers
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_bigswitch/driver.py", line 91, in update_port_postcommit
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     port["network"]["id"], port)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 555, in rest_update_port
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     self.rest_create_port(tenant_id, net_id, port)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 545, in rest_create_port
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     self.rest_action('PUT', resource, data, errstr)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 476, in rest_action
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     timeout)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/lockutils.py", line 249, in inner
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     return f(*args, **kwargs)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 423, in rest_call
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     hash_handler=hash_handler)
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/servermanager.py", line 139, in rest_call
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     headers[HASH_MATCH_HEADER] = hash_handler.read_for_update()
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers   File "/usr/lib/python2.7/dist-packages/neutron/plugins/bigswitch/db/consistency_db.py", line 56, in read_for_update
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers     raise MultipleReadForUpdateCalls()
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers MultipleReadForUpdateCalls: Only one read_for_update call may be made at a time.
  2014-08-14 21:56:41.496 12939 TRACE neutron.plugins.ml2.managers

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1357102/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next