yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1873265] Re: VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1873265@xxxxxxxxxxxxxxxxxx>
Date: Fri, 17 Apr 2020 11:15:43 -0000
Reply-to: Bug 1873265 <1873265@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hi,

I'm inclined to agree with Oleg here. If 2 of 3 rabbit servers are
working fine and agent is still connected to the broken one, I think
it's something for oslo.messaging. Neutron is just using oslo.messaging
project to connect with rabbitmq.

** Changed in: neutron
       Status: New => Incomplete

** Also affects: oslo.messaging
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1873265

Title:
  VM can't be accessed, openflow rules are removed and neutron-ovs-agent
  is always reporting errors after rabbitmq is in split-brain state

Status in neutron:
  Incomplete
Status in oslo.messaging:
  New

Bug description:
  After rabbitmq cluster is in split brain state because of some network
  issues, neutron ovs agents in compute nodes are always reporting error
  info, some open flow rules are removed, so VMs can't be accessed
  normally.

  Here are some important error info:

  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID ede4a709807e43dc90df4469ca9d88f3
  2020-04-16 08:35:05.578 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:35:05.586 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
  2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.208 21720 ERROR neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Timeout in RPC method update_device_list. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:35:44.216 21720 WARNING neutron.common.rpc [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Increasing timeout for update_device_list calls to 120 seconds. Restart the agent to restore it to the default value.: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:05.413 21720 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e9e7407c173a438096eb1fad7e22d9fb
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 328, in _report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     True)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc669410f7a345649ca8c653f8ccf939
  2020-04-16 08:36:05.592 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:05.600 21720 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 30.01 sec
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2236, in rpc_loop
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1802, in process_network_ports
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 938, in _bind_devices
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host, agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 170, in update_device_list
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 173, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID c1bb1fe4c2fe42259dd3726e7ff9e48a
  2020-04-16 08:36:24.558 21720 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
  2020-04-16 08:36:24.568 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Agent rpc_loop - iteration:0 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 10, 'removed': 0}}. Elapsed:137.142 loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1944
  2020-04-16 08:36:24.575 21720 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-fe5b94c8-c725-4e77-b47b-1b2c0a1e1583 - - - - -] Loop iteration exceeded interval (2 vs. 137.142431974)! loop_count_and_wait /var/lib/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1951

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1873265/+subscriptions
References

[Bug 1873265] [NEW] VM can't be accessed, openflow rules are removed and neutron-ovs-agent is always reporting errors after rabbitmq is in split-brain state
From: Yi Yang, 2020-04-16