yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1260791] [NEW] ovs agents flapping

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Robert Pothier <rpothier@xxxxxxxxx>
Date: Fri, 13 Dec 2013 16:22:30 -0000
Reply-to: Bug 1260791 <1260791@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

During deployment of instances using nova boot commands,  I noticed that
the Open vSwitch agents on the compute nodes alive status goes from xxx
to :-) and vice versa for all the compute nodes at random.

Getting the below traceback, the issue seems to be similar to a NEC agent bug.
https://bugs.launchpad.net/neutron/+bug/1235106

2013-11-26 20:07:00.941 16044 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     **args)
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 102, in security_groups_provider_updated
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_provider_updated()
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 151, in security_groups_provider_updated
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall()
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     exc.info, real_topic, msg.get('method'))
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devices" info: "<unknown>"
2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp
2013-11-26 20:07:00.942 16044 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     **args)
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 102, in security_groups_provider_updated
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_provider_updated()
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 151, in security_groups_provider_updated
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall()
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     exc.info, real_topic, msg.get('method'))
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devices" info: "<unknown>"
2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1260791

Title:
  ovs agents flapping

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  During deployment of instances using nova boot commands,  I noticed
  that the Open vSwitch agents on the compute nodes alive status goes
  from xxx to :-) and vice versa for all the compute nodes at random.

  Getting the below traceback, the issue seems to be similar to a NEC agent bug.
  https://bugs.launchpad.net/neutron/+bug/1235106

  2013-11-26 20:07:00.941 16044 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     **args)
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 102, in security_groups_provider_updated
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_provider_updated()
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 151, in security_groups_provider_updated
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall()
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp     exc.info, real_topic, msg.get('method'))
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devices" info: "<unknown>"
  2013-11-26 20:07:00.941 16044 TRACE neutron.openstack.common.rpc.amqp
  2013-11-26 20:07:00.942 16044 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     **args)
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 102, in security_groups_provider_updated
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_provider_updated()
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 151, in security_groups_provider_updated
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall()
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp     exc.info, real_topic, msg.get('method'))
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devices" info: "<unknown>"
  2013-11-26 20:07:00.942 16044 TRACE neutron.openstack.common.rpc.amqp

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1260791/+subscriptions
Follow ups

[Bug 1260791] [NEW] ovs agents flapping
From: Robert Pothier, 2013-12-13
References

[Bug 1260791] [NEW] ovs agents flapping
From: Robert Pothier, 2013-12-13