yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1479265] [NEW] ovs agent crashes with UnboundLocalError in case self.scan_ancillary_ports fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1479265@xxxxxxxxxxxxxxxxxx>
Date: Wed, 29 Jul 2015 08:36:54 -0000
Reply-to: Bug 1479265 <1479265@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It shows in the following functional test crash:

ft1.205: neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_resync_devices_set_up_after_exception(native)_StringException: Empty attachments:
  pythonlogging:'neutron.api.extensions'
  stdout

pythonlogging:'': {{{
2015-07-28 21:38:06,203     INFO [neutron.agent.l2.agent_extensions_manager] Configured agent extensions names: ('qos',)
2015-07-28 21:38:06,204     INFO [neutron.agent.l2.agent_extensions_manager] Loaded agent extensions names: ['qos']
2015-07-28 21:38:06,204     INFO [neutron.agent.l2.agent_extensions_manager] Initializing agent extension 'qos'
2015-07-28 21:38:06,280     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Mapping physical network physnet to bridge br-int359443631
2015-07-28 21:38:06,349  WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int359443631 exceeds the 15 character limitation. It was shortened to int-br-in3cbf05 to fit.
2015-07-28 21:38:06,349  WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int359443631 exceeds the 15 character limitation. It was shortened to phy-br-in3cbf05 to fit.
2015-07-28 21:38:06,970     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Adding test-br69135803 to list of bridges.
2015-07-28 21:38:06,974  WARNING [neutron.agent.securitygroups_rpc] Driver configuration doesn't match with enable_security_group
2015-07-28 21:38:07,061     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Agent out of sync with plugin!
2015-07-28 21:38:07,062     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Agent tunnel out of sync with plugin!
2015-07-28 21:38:07,204    ERROR [neutron.agent.ovsdb.impl_idl] Traceback (most recent call last):
  File "neutron/agent/ovsdb/native/connection.py", line 84, in run
    txn.results.put(txn.do_commit())
  File "neutron/agent/ovsdb/impl_idl.py", line 92, in do_commit
    ctx.reraise = False
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "neutron/agent/ovsdb/impl_idl.py", line 87, in do_commit
    command.run_idl(txn)
  File "neutron/agent/ovsdb/native/commands.py", line 355, in run_idl
    br = idlutils.row_by_value(self.api.idl, 'Bridge', 'name', self.bridge)
  File "neutron/agent/ovsdb/native/idlutils.py", line 59, in row_by_value
    raise RowNotFound(table=table, col=column, match=match)
RowNotFound: Cannot find Bridge with name=test-br69135803

2015-07-28 21:38:07,204    ERROR [neutron.agent.ovsdb.native.commands] Error executing command
Traceback (most recent call last):
  File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
    txn.add(self)
  File "neutron/agent/ovsdb/api.py", line 70, in __exit__
    self.result = self.commit()
  File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
    raise result.ex
RowNotFound: Cannot find Bridge with name=test-br69135803
2015-07-28 21:38:07,205    ERROR [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Error while processing VIF ports
Traceback (most recent call last):
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1569, in rpc_loop
    ancillary_ports)
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1104, in scan_ancillary_ports
    cur_ports |= bridge.get_vif_port_set()
  File "neutron/agent/common/ovs_lib.py", line 376, in get_vif_port_set
    port_names = self.get_port_name_list()
  File "neutron/agent/common/ovs_lib.py", line 313, in get_port_name_list
    return self.ovsdb.list_ports(self.br_name).execute(check_error=True)
  File "neutron/agent/ovsdb/native/commands.py", line 42, in execute
    ctx.reraise = False
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
    txn.add(self)
  File "neutron/agent/ovsdb/api.py", line 70, in __exit__
    self.result = self.commit()
  File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
    raise result.ex
RowNotFound: Cannot find Bridge with name=test-br69135803
}}}

stderr: {{{
Traceback (most recent call last):
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/queue.py", line 117, in switch
    self.greenlet.switch(value)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1612, in rpc_loop
    ancillary_port_info = (ancillary_port_info if self.ancillary_brs
UnboundLocalError: local variable 'ancillary_port_info' referenced before assignment
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 131, in stop_agent
    rpc_loop_thread.wait()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
    return self._exit_event.wait()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1612, in rpc_loop
    ancillary_port_info = (ancillary_port_info if self.ancillary_brs
UnboundLocalError: local variable 'ancillary_port_info' referenced before assignment
}}}

Traceback (most recent call last):
  File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 272, in test_resync_devices_set_up_after_exception
    lambda: self._expected_plugin_rpc_call(
  File "neutron/agent/linux/utils.py", line 323, in wait_until_true
    eventlet.sleep(sleep)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 34, in sleep
    hub.switch()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

Note that functional test failure is tracked in
https://bugs.launchpad.net/neutron/+bug/1479264

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovs

** Tags added: ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1479265

Title:
  ovs agent crashes with UnboundLocalError in case
  self.scan_ancillary_ports fails

Status in neutron:
  New

Bug description:
  It shows in the following functional test crash:

  ft1.205: neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_resync_devices_set_up_after_exception(native)_StringException: Empty attachments:
    pythonlogging:'neutron.api.extensions'
    stdout

  pythonlogging:'': {{{
  2015-07-28 21:38:06,203     INFO [neutron.agent.l2.agent_extensions_manager] Configured agent extensions names: ('qos',)
  2015-07-28 21:38:06,204     INFO [neutron.agent.l2.agent_extensions_manager] Loaded agent extensions names: ['qos']
  2015-07-28 21:38:06,204     INFO [neutron.agent.l2.agent_extensions_manager] Initializing agent extension 'qos'
  2015-07-28 21:38:06,280     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Mapping physical network physnet to bridge br-int359443631
  2015-07-28 21:38:06,349  WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int359443631 exceeds the 15 character limitation. It was shortened to int-br-in3cbf05 to fit.
  2015-07-28 21:38:06,349  WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int359443631 exceeds the 15 character limitation. It was shortened to phy-br-in3cbf05 to fit.
  2015-07-28 21:38:06,970     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Adding test-br69135803 to list of bridges.
  2015-07-28 21:38:06,974  WARNING [neutron.agent.securitygroups_rpc] Driver configuration doesn't match with enable_security_group
  2015-07-28 21:38:07,061     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Agent out of sync with plugin!
  2015-07-28 21:38:07,062     INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Agent tunnel out of sync with plugin!
  2015-07-28 21:38:07,204    ERROR [neutron.agent.ovsdb.impl_idl] Traceback (most recent call last):
    File "neutron/agent/ovsdb/native/connection.py", line 84, in run
      txn.results.put(txn.do_commit())
    File "neutron/agent/ovsdb/impl_idl.py", line 92, in do_commit
      ctx.reraise = False
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "neutron/agent/ovsdb/impl_idl.py", line 87, in do_commit
      command.run_idl(txn)
    File "neutron/agent/ovsdb/native/commands.py", line 355, in run_idl
      br = idlutils.row_by_value(self.api.idl, 'Bridge', 'name', self.bridge)
    File "neutron/agent/ovsdb/native/idlutils.py", line 59, in row_by_value
      raise RowNotFound(table=table, col=column, match=match)
  RowNotFound: Cannot find Bridge with name=test-br69135803

  2015-07-28 21:38:07,204    ERROR [neutron.agent.ovsdb.native.commands] Error executing command
  Traceback (most recent call last):
    File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
      txn.add(self)
    File "neutron/agent/ovsdb/api.py", line 70, in __exit__
      self.result = self.commit()
    File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
      raise result.ex
  RowNotFound: Cannot find Bridge with name=test-br69135803
  2015-07-28 21:38:07,205    ERROR [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Error while processing VIF ports
  Traceback (most recent call last):
    File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1569, in rpc_loop
      ancillary_ports)
    File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1104, in scan_ancillary_ports
      cur_ports |= bridge.get_vif_port_set()
    File "neutron/agent/common/ovs_lib.py", line 376, in get_vif_port_set
      port_names = self.get_port_name_list()
    File "neutron/agent/common/ovs_lib.py", line 313, in get_port_name_list
      return self.ovsdb.list_ports(self.br_name).execute(check_error=True)
    File "neutron/agent/ovsdb/native/commands.py", line 42, in execute
      ctx.reraise = False
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
      txn.add(self)
    File "neutron/agent/ovsdb/api.py", line 70, in __exit__
      self.result = self.commit()
    File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
      raise result.ex
  RowNotFound: Cannot find Bridge with name=test-br69135803
  }}}

  stderr: {{{
  Traceback (most recent call last):
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/queue.py", line 117, in switch
      self.greenlet.switch(value)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1612, in rpc_loop
      ancillary_port_info = (ancillary_port_info if self.ancillary_brs
  UnboundLocalError: local variable 'ancillary_port_info' referenced before assignment
  }}}

  traceback-1: {{{
  Traceback (most recent call last):
    File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 131, in stop_agent
      rpc_loop_thread.wait()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
      return self._exit_event.wait()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
      current.throw(*self._exc)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1612, in rpc_loop
      ancillary_port_info = (ancillary_port_info if self.ancillary_brs
  UnboundLocalError: local variable 'ancillary_port_info' referenced before assignment
  }}}

  Traceback (most recent call last):
    File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 272, in test_resync_devices_set_up_after_exception
      lambda: self._expected_plugin_rpc_call(
    File "neutron/agent/linux/utils.py", line 323, in wait_until_true
      eventlet.sleep(sleep)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 34, in sleep
      hub.switch()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
      return self.greenlet.switch()
  eventlet.timeout.Timeout: 60 seconds

  Note that functional test failure is tracked in
  https://bugs.launchpad.net/neutron/+bug/1479264

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1479265/+subscriptions
Follow ups

[Bug 1479265] Re: ovs agent crashes with UnboundLocalError in case self.scan_ancillary_ports fails
From: Thierry Carrez, 2015-09-03