yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1694524] [NEW] Neutron OVS agent fails to start when neutron-server is not available

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1694524@xxxxxxxxxxxxxxxxxx>
Date: Tue, 30 May 2017 19:37:56 -0000
Reply-to: Bug 1694524 <1694524@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-34115ed3-3043-4fcb-ba3f-ab0e4eb0e83c - - - - -] Agent main thread died of an exception
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     ovs_agent.main(bridge_classes)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2166, in main
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 180, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.setup_rpc()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return f(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 362, in setup_rpc
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 182, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.remote_resource_cache = create_cache_for_l2_agent()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 174, in create_cache_for_l2_agent
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     rcache.bulk_flood_cache()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 55, in bulk_flood_cache
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     for resource in puller.bulk_pull(context, rtype):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 48, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return method(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 109, in bulk_pull
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     time.sleep(wait)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.force_reraise()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     six.reraise(self.type_, self.value, self.tb)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return self._original_context.call(ctxt, method, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=self.retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     timeout=timeout, retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 494, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     result = self._waiter.wait(msg_id, timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in wait
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     message = self.waiters.get(msg_id, timeout=timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 275, in get
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     'to message ID %s' % msg_id)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp MessagingTimeout: Timed out waiting for a reply to message ID c79afd6fe556408bbaca568e599feada
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
2017-05-30 18:58:02.051 27929 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=28388

https://logs.rdoproject.org/weirdo-generic-puppet-openstack-
scenario001/1754/weirdo-project/logs/neutron/openvswitch-agent.txt.gz

This is probably because of push-notifications work.

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: ovs

** Changed in: neutron
   Importance: Undecided => High

** Tags added: ovs

** Changed in: neutron
       Status: New => Confirmed

** Changed in: neutron
    Milestone: None => pike-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1694524

Title:
  Neutron OVS agent fails to start when neutron-server is not available

Status in neutron:
  Confirmed

Bug description:
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-34115ed3-3043-4fcb-ba3f-ab0e4eb0e83c - - - - -] Agent main thread died of an exception
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     ovs_agent.main(bridge_classes)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2166, in main
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 180, in __init__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.setup_rpc()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return f(*args, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 362, in setup_rpc
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 182, in __init__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.remote_resource_cache = create_cache_for_l2_agent()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 174, in create_cache_for_l2_agent
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     rcache.bulk_flood_cache()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 55, in bulk_flood_cache
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     for resource in puller.bulk_pull(context, rtype):
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 48, in wrapper
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return method(*args, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 109, in bulk_pull
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     time.sleep(wait)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.force_reraise()
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     six.reraise(self.type_, self.value, self.tb)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return self._original_context.call(ctxt, method, **kwargs)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=self.retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     timeout=timeout, retry=retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     retry=retry)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 494, in _send
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     result = self._waiter.wait(msg_id, timeout)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in wait
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     message = self.waiters.get(msg_id, timeout=timeout)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 275, in get
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     'to message ID %s' % msg_id)
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp MessagingTimeout: Timed out waiting for a reply to message ID c79afd6fe556408bbaca568e599feada
  2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
  2017-05-30 18:58:02.051 27929 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=28388

  https://logs.rdoproject.org/weirdo-generic-puppet-openstack-
  scenario001/1754/weirdo-project/logs/neutron/openvswitch-agent.txt.gz

  This is probably because of push-notifications work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1694524/+subscriptions
Follow ups

[Bug 1694524] Re: Neutron OVS agent fails to start when neutron-server is not available
From: Rodolfo Alonso, 2022-11-30
[Bug 1694524] Re: Neutron OVS agent fails to start when neutron-server is not available
From: Emilien Macchi, 2017-06-02
[Bug 1694524] Re: Neutron OVS agent fails to start when neutron-server is not available
From: Emilien Macchi, 2017-05-31