yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1694524] Re: Neutron OVS agent fails to start when neutron-server is not available

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Emilien Macchi <1694524@xxxxxxxxxxxxxxxxxx>
Date: Fri, 02 Jun 2017 14:48:07 -0000
Reply-to: Bug 1694524 <1694524@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: tripleo
       Status: Triaged => Fix Released

** Tags removed: alert ci promotion-blocker

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1694524

Title:
  Neutron OVS agent fails to start when neutron-server is not available

Status in neutron:
  Confirmed
Status in tripleo:
  Fix Released

Bug description:
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-34115ed3-3043-4fcb-ba3f-ab0e4eb0e83c - - - - -] Agent main thread died of an exception
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     ovs_agent.main(bridge_classes)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2166, in main
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 180, in __init__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.setup_rpc()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return f(*args, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 362, in setup_rpc
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 182, in __init__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.remote_resource_cache = create_cache_for_l2_agent()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 174, in create_cache_for_l2_agent
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     rcache.bulk_flood_cache()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 55, in bulk_flood_cache
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     for resource in puller.bulk_pull(context, rtype):
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 48, in wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return method(*args, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 109, in bulk_pull
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     time.sleep(wait)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.force_reraise()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     six.reraise(self.type_, self.value, self.tb)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return self._original_context.call(ctxt, method, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=self.retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     timeout=timeout, retry=retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 494, in _send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     result = self._waiter.wait(msg_id, timeout)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in wait
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     message = self.waiters.get(msg_id, timeout=timeout)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 275, in get
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     'to message ID %s' % msg_id)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp MessagingTimeout: Timed out waiting for a reply to message ID c79afd6fe556408bbaca568e599feada
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
  2017-05-30 18:58:02.051 27929 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=28388

  https://logs.rdoproject.org/weirdo-generic-puppet-openstack-
  scenario001/1754/weirdo-project/logs/neutron/openvswitch-agent.txt.gz

  This is probably because of push-notifications work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1694524/+subscriptions
References

[Bug 1694524] [NEW] Neutron OVS agent fails to start when neutron-server is not available
From: Ihar Hrachyshka, 2017-05-30