← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #12032

[Bug 1296202] [NEW] openvswitch is sometimes restarting or times out, and this causes neutron-openvswitch-agent to crash

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This was observed recently durin an Ironic CI run:

2014-03-22 02:38:53.436 7916 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-03-22 02:38:53.690 7916 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']. Exception: 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
Exit code: 1
Stdout: ''
Stderr: '2014-03-22T02:38:53Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n'
2014-03-22 02:38:53.693 7916 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1212, in rpc_loop
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info = self.scan_ports(ports, updated_ports_copy)
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 815, in scan_ports
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     cur_ports = self.int_br.get_vif_port_set()
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 311, in get_vif_port_set
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_names = self.get_port_name_list()
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 267, in get_port_name_list
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     res = self.run_vsctl(["list-ports", self.br_name], check_error=True)
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 74, in run_vsctl
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ctxt.reraise = False
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 67, in run_vsctl
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return utils.execute(full_args, root_helper=self.root_helper)
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '2014-03-22T02:38:53Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n'

This should result in a retry/sleep loop, not death of neutron-
openvswitch-agent.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1296202

Title:
  openvswitch is sometimes restarting or times out, and this causes
  neutron-openvswitch-agent to crash

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  This was observed recently durin an Ironic CI run:

  2014-03-22 02:38:53.436 7916 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
  2014-03-22 02:38:53.690 7916 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']. Exception: 
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
  Exit code: 1
  Stdout: ''
  Stderr: '2014-03-22T02:38:53Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n'
  2014-03-22 02:38:53.693 7916 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1212, in rpc_loop
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info = self.scan_ports(ports, updated_ports_copy)
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 815, in scan_ports
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     cur_ports = self.int_br.get_vif_port_set()
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 311, in get_vif_port_set
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_names = self.get_port_name_list()
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 267, in get_port_name_list
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     res = self.run_vsctl(["list-ports", self.br_name], check_error=True)
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 74, in run_vsctl
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ctxt.reraise = False
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 67, in run_vsctl
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return utils.execute(full_args, root_helper=self.root_helper)
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 76, in execute
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
  2014-03-22 02:38:53.693 7916 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '2014-03-22T02:38:53Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n'

  This should result in a retry/sleep loop, not death of neutron-
  openvswitch-agent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1296202/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next