yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1367881] [NEW] l2pop RPC code throwing an exception in fdb_chg_ip_tun()

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brian Haley <1367881@xxxxxxxxxxxxxxxxxx>
Date: Wed, 10 Sep 2014 19:08:11 -0000
Reply-to: Bug 1367881 <1367881@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I'm seeing an error in the l2pop code where it's failing to add a flow
for the ARP entry responder.

This is sometimes leading to DHCP failures for VMs, although a soft
reboot typically fixes that problem.

Here is the trace:

2014-09-10 15:10:36.954 9351 ERROR neutron.agent.linux.ovs_lib [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 None] OVS flows could not be applied on bridge br-tun
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib Traceback (most recent call last):
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 407, in _fdb_chg_ip 
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     self.local_ip, self.local_vlan_map)
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     return method(*args, **kwargs)
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 250, in fdb_chg_ip_tun
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     for mac, ip in after:
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib TypeError: 'NoneType' object is not iterable
2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib 
2014-09-10 15:10:36.955 9351 ERROR oslo.messaging.rpc.dispatcher [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 ] Exception during message handling: 'NoneType' object is not iterable
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 55, in update_fdb_entries
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     self.fdb_update(context, fdb_entries)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 212, in fdb_update
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     getattr(self, method)(context, values)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 407, in _fdb_chg_ip
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     self.local_ip, self.local_vlan_map)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 250, in fdb_chg_ip_tun
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     for mac, ip in after:
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher 
2014-09-10 15:10:36.957 9351 ERROR oslo.messaging._drivers.common [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 ] Returning exception 'NoneType' object is not iterable to caller

I don't know this code well enough to suggest a fix - whether it's
checking the return from agent_ports.items() better, or that there is a
bug elsewhere, so any help would be appreciated.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1367881

Title:
  l2pop RPC code throwing an exception in fdb_chg_ip_tun()

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  I'm seeing an error in the l2pop code where it's failing to add a flow
  for the ARP entry responder.

  This is sometimes leading to DHCP failures for VMs, although a soft
  reboot typically fixes that problem.

  Here is the trace:

  2014-09-10 15:10:36.954 9351 ERROR neutron.agent.linux.ovs_lib [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 None] OVS flows could not be applied on bridge br-tun
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib Traceback (most recent call last):
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 407, in _fdb_chg_ip 
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     self.local_ip, self.local_vlan_map)
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     return method(*args, **kwargs)
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 250, in fdb_chg_ip_tun
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib     for mac, ip in after:
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib TypeError: 'NoneType' object is not iterable
  2014-09-10 15:10:36.954 9351 TRACE neutron.agent.linux.ovs_lib 
  2014-09-10 15:10:36.955 9351 ERROR oslo.messaging.rpc.dispatcher [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 ] Exception during message handling: 'NoneType' object is not iterable
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 55, in update_fdb_entries
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     self.fdb_update(context, fdb_entries)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 212, in fdb_update
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     getattr(self, method)(context, values)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 407, in _fdb_chg_ip
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     self.local_ip, self.local_vlan_map)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py", line 36, in wrapper
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     return method(*args, **kwargs)
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l2population_rpc.py", line 250, in fdb_chg_ip_tun
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher     for mac, ip in after:
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-09-10 15:10:36.955 9351 TRACE oslo.messaging.rpc.dispatcher 
  2014-09-10 15:10:36.957 9351 ERROR oslo.messaging._drivers.common [req-de0c2985-1fac-46a8-a42b-f0bad5a43805 ] Returning exception 'NoneType' object is not iterable to caller

  I don't know this code well enough to suggest a fix - whether it's
  checking the return from agent_ports.items() better, or that there is
  a bug elsewhere, so any help would be appreciated.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1367881/+subscriptions
Follow ups

[Bug 1367881] Re: l2pop RPC code throwing an exception in fdb_chg_ip_tun()
From: Thierry Carrez, 2014-12-18
[Bug 1367881] [NEW] l2pop RPC code throwing an exception in fdb_chg_ip_tun()
From: Brian Haley, 2014-09-10
References

[Bug 1367881] [NEW] l2pop RPC code throwing an exception in fdb_chg_ip_tun()
From: Brian Haley, 2014-09-10