yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1485422] Re: neutron-ovs-cleanup sometimes failed at reboot (Ubuntu 14.04)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miguel Angel Ajo <majopela@xxxxxxxxxx>
Date: Tue, 06 Oct 2015 14:41:48 -0000
Reply-to: Bug 1485422 <1485422@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This bug is incorrect, closing as Invalid.

** Changed in: neutron
       Status: Incomplete => Invalid

** Changed in: neutron
       Status: Invalid => Won't Fix

** Changed in: neutron
       Status: Won't Fix => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1485422

Title:
  neutron-ovs-cleanup sometimes failed at reboot (Ubuntu 14.04)

Status in neutron:
  Invalid

Bug description:
  I am running OpenStack Kilo on Ubuntu 14.04 (Trusty). I have a
  separate network node running Neutron. Sometimes after reboot the
  neutron-plugin-openvswitch-agent is not started, whereas the neutron-
  dhcp-agent and the neutron-l3-agent get started. The problem is that
  sometimes (not always) the neutron-ovs-cleanup fails to start with the
  following exceptions:

  2015-08-17 06:05:07.091 1470 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br'].
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Traceback (most recent call last):
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl     log_fail_as_error=False).rstrip()
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl     raise RuntimeError(m)
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError: 
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdin: 
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdout: 
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl 
  2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl 
  2015-08-17 06:05:07.098 1470 CRITICAL neutron [-] RuntimeError: 
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

  or

  2015-08-16 12:08:57.315 1820 TRACE neutron Traceback (most recent call last):
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/bin/neutron-ovs-cleanup", line 10, in <module>
  2015-08-16 12:08:57.315 1820 TRACE neutron     sys.exit(main())
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 108, in main
  2015-08-16 12:08:57.315 1820 TRACE neutron     delete_neutron_ports(ports)
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 72, in delete_neutron_ports
  2015-08-16 12:08:57.315 1820 TRACE neutron     device.link.delete()
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 293, in delete
  2015-08-16 12:08:57.315 1820 TRACE neutron     self._as_root([], ('delete', self.name))
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-08-16 12:08:57.315 1820 TRACE neutron     use_root_namespace=use_root_namespace)
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-08-16 12:08:57.315 1820 TRACE neutron     log_fail_as_error=self.log_fail_as_error)
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-08-16 12:08:57.315 1820 TRACE neutron     log_fail_as_error=log_fail_as_error)
  2015-08-16 12:08:57.315 1820 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2015-08-16 12:08:57.315 1820 TRACE neutron     raise RuntimeError(m)
  2015-08-16 12:08:57.315 1820 TRACE neutron RuntimeError: 
  2015-08-16 12:08:57.315 1820 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'qg-b731d447-d1']
  2015-08-16 12:08:57.315 1820 TRACE neutron Exit code: 2
  2015-08-16 12:08:57.315 1820 TRACE neutron Stdin: 
  2015-08-16 12:08:57.315 1820 TRACE neutron Stdout: 
  2015-08-16 12:08:57.315 1820 TRACE neutron Stderr: RTNETLINK answers: Operation not supported

  From my point of view there is a race condition at startup. I have
  added a "sleep 5" command to my upstart-script neutron-ovs-
  cleanup.conf and the problem is gone. I don't think this a proper
  solution.

  Has anybody an idea?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1485422/+subscriptions
References

[Bug 1485422] [NEW] neutron-ovs-cleanup sometimes failed at reboot (Ubuntu 14.04)
From: Jens Offenbach, 2015-08-17