yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1873265] [NEW] VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yi Yang <1873265@xxxxxxxxxxxxxxxxxx>
Date: Thu, 16 Apr 2020 12:45:43 -0000
Reply-to: Bug 1873265 <1873265@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After rabbitmq cluster is in split brain state because of some network
issues, neutron ovs agents in compute nodes are always reporting error
info, some open flow rules are removed, so VMs can't be accessed
normally.

Here are some important error info:

2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
2020-04-16 08:36:24.575 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded interval (2 vs. 137.142431974)! loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: rocky

** Summary changed:

- VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors and after rabbitmq is in split-brain state
+ VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1873265

Title:
  VM can't be accessed, openflow rules are removed and neutron-ovs-agent
  is always reporting errors after rabbitmq is in split-brain state

Status in neutron:
  New

Bug description:
  After rabbitmq cluster is in split brain state because of some network
  issues, neutron ovs agents in compute nodes are always reporting error
  info, some open flow rules are removed, so VMs can't be accessed
  normally.

  Here are some important error info:

  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
  2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
  2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
  2020-04-16 08:36:24.575 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded interval (2 vs. 137.142431974)! loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1873265/+subscriptions
Follow ups

[Bug 1873265] Re: VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state
From: Slawek Kaplonski, 2020-04-17