yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1524394] [NEW] neutron-openvswitch-agent fails to start when root_helper operates in a different context

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tom Carroll <Thomas.Carroll@xxxxxxxx>
Date: Wed, 09 Dec 2015 15:51:41 -0000
Reply-to: Bug 1524394 <1524394@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Version: Liberty
Compute hypervisor: XenServer 6.5
Compute vm: Ubuntu 14.04.3

This issue appears in liberty--and not before--when running XenServer
hypervisor. In this environment, root-helper is set to /usr/bin/neutron-
rootwrap-xen-dom0, which executes commands in the hypervisor's Dom0
context. This problem keeps the neutron-openvswitch-agent from starting
and thus breaking the networking on the compute nodes.

A backtrace will be appended. The gist of the problem is that
ip_lib.get_devices()  does not use root_helper to obtain a list of the
network interfaces when the network namespace is the global namespace.
Thus, it obtains the interfaces of the compute virtual machine
environment and not the Dom0 environment.

I've appended two patches, one for ip_lib that corrects the listing and
one to netwrap to allow find. There are security implications by
permitting the execution of `find' in netwrap.

Backtrace from openvswitch-agent.log:

 2015-12-09 07:44:10.274 11884 CRITICAL neutron [-] RuntimeError:
Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ip', 'addr', 'show', 'br-int', 'to', '192.168.1.26']
Exit code: 96
Stdin:
Stdout:
Stderr: Traceback (most recent call last):
  File "/usr/bin/neutron-rootwrap-xen-dom0", line 119, in run_command
    {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
  File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 245, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 149, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 219, in _parse_result
    raise Failure(result['ErrorDescription'])
Failure: ['XENAPI_PLUGIN_FAILURE', 'run_command', 'PluginError', 'Device "br-int" does not exist.\n']
2015-12-09 07:44:10.274 11884 ERROR neutron Traceback (most recent call last):
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2015-12-09 07:44:10.274 11884 ERROR neutron     sys.exit(main())
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2015-12-09 07:44:10.274 11884 ERROR neutron     agent_main.main()
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2015-12-09 07:44:10.274 11884 ERROR neutron     mod.main()
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
2015-12-09 07:44:10.274 11884 ERROR neutron     ovs_neutron_agent.main(bridge_classes)
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1899, in main2015-12-09 07:44:10.274 11884 ERROR neutron     validate_local_ip(agent_config['local_ip'])
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1876, in validate_local_ip
2015-12-09 07:44:10.274 11884 ERROR neutron     if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 131, in get_device_by_ip
2015-12-09 07:44:10.274 11884 ERROR neutron     if device.addr.list(to=ip):
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 514, in list
2015-12-09 07:44:10.274 11884 ERROR neutron     for line in self._run(options, tuple(args)).split('\n'):
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 274, in _run
2015-12-09 07:44:10.274 11884 ERROR neutron     return self._parent._run(options, self.COMMAND, args)
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 70, in _run
2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_as_error=self.log_fail_as_error)
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_as_error=log_fail_as_error)
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 159, in execute
2015-12-09 07:44:10.274 11884 ERROR neutron     raise RuntimeError(m)
2015-12-09 07:44:10.274 11884 ERROR neutron RuntimeError:
2015-12-09 07:44:10.274 11884 ERROR neutron Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ip', 'addr', 'show', 'br-int', 'to', '192.168.1.26']
2015-12-09 07:44:10.274 11884 ERROR neutron Exit code: 96
2015-12-09 07:44:10.274 11884 ERROR neutron Stdin:
2015-12-09 07:44:10.274 11884 ERROR neutron Stdout:
2015-12-09 07:44:10.274 11884 ERROR neutron Stderr: Traceback (most recent call last):
2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/bin/neutron-rootwrap-xen-dom0", line 119, in run_command
2015-12-09 07:44:10.274 11884 ERROR neutron     {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})as_error=self.log_fail_as_error)

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: xenserver

** Patch added: "ip_lib.get_devices(): Use root_helper to list network interfaces in global namespace"
   https://bugs.launchpad.net/bugs/1524394/+attachment/4531984/+files/ip_lib.patch

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1524394

Title:
  neutron-openvswitch-agent fails to start when root_helper operates in
  a different context

Status in neutron:
  New

Bug description:
  Version: Liberty
  Compute hypervisor: XenServer 6.5
  Compute vm: Ubuntu 14.04.3

  This issue appears in liberty--and not before--when running XenServer
  hypervisor. In this environment, root-helper is set to /usr/bin
  /neutron-rootwrap-xen-dom0, which executes commands in the
  hypervisor's Dom0 context. This problem keeps the neutron-openvswitch-
  agent from starting and thus breaking the networking on the compute
  nodes.

  A backtrace will be appended. The gist of the problem is that
  ip_lib.get_devices()  does not use root_helper to obtain a list of the
  network interfaces when the network namespace is the global namespace.
  Thus, it obtains the interfaces of the compute virtual machine
  environment and not the Dom0 environment.

  I've appended two patches, one for ip_lib that corrects the listing
  and one to netwrap to allow find. There are security implications by
  permitting the execution of `find' in netwrap.

  Backtrace from openvswitch-agent.log:

   2015-12-09 07:44:10.274 11884 CRITICAL neutron [-] RuntimeError:
  Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ip', 'addr', 'show', 'br-int', 'to', '192.168.1.26']
  Exit code: 96
  Stdin:
  Stdout:
  Stderr: Traceback (most recent call last):
    File "/usr/bin/neutron-rootwrap-xen-dom0", line 119, in run_command
      {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
    File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 245, in __call__
      return self.__send(self.__name, args)
    File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 149, in xenapi_request
      result = _parse_result(getattr(self, methodname)(*full_params))
    File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 219, in _parse_result
      raise Failure(result['ErrorDescription'])
  Failure: ['XENAPI_PLUGIN_FAILURE', 'run_command', 'PluginError', 'Device "br-int" does not exist.\n']
  2015-12-09 07:44:10.274 11884 ERROR neutron Traceback (most recent call last):
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
  2015-12-09 07:44:10.274 11884 ERROR neutron     sys.exit(main())
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
  2015-12-09 07:44:10.274 11884 ERROR neutron     agent_main.main()
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
  2015-12-09 07:44:10.274 11884 ERROR neutron     mod.main()
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
  2015-12-09 07:44:10.274 11884 ERROR neutron     ovs_neutron_agent.main(bridge_classes)
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1899, in main2015-12-09 07:44:10.274 11884 ERROR neutron     validate_local_ip(agent_config['local_ip'])
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1876, in validate_local_ip
  2015-12-09 07:44:10.274 11884 ERROR neutron     if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 131, in get_device_by_ip
  2015-12-09 07:44:10.274 11884 ERROR neutron     if device.addr.list(to=ip):
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 514, in list
  2015-12-09 07:44:10.274 11884 ERROR neutron     for line in self._run(options, tuple(args)).split('\n'):
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 274, in _run
  2015-12-09 07:44:10.274 11884 ERROR neutron     return self._parent._run(options, self.COMMAND, args)
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 70, in _run
  2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_as_error=self.log_fail_as_error)
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
  2015-12-09 07:44:10.274 11884 ERROR neutron     log_fail_as_error=log_fail_as_error)
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 159, in execute
  2015-12-09 07:44:10.274 11884 ERROR neutron     raise RuntimeError(m)
  2015-12-09 07:44:10.274 11884 ERROR neutron RuntimeError:
  2015-12-09 07:44:10.274 11884 ERROR neutron Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ip', 'addr', 'show', 'br-int', 'to', '192.168.1.26']
  2015-12-09 07:44:10.274 11884 ERROR neutron Exit code: 96
  2015-12-09 07:44:10.274 11884 ERROR neutron Stdin:
  2015-12-09 07:44:10.274 11884 ERROR neutron Stdout:
  2015-12-09 07:44:10.274 11884 ERROR neutron Stderr: Traceback (most recent call last):
  2015-12-09 07:44:10.274 11884 ERROR neutron   File "/usr/bin/neutron-rootwrap-xen-dom0", line 119, in run_command
  2015-12-09 07:44:10.274 11884 ERROR neutron     {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})as_error=self.log_fail_as_error)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1524394/+subscriptions
Follow ups

[Bug 1524394] Re: neutron-openvswitch-agent fails to start when root_helper operates in a different context
From: Launchpad Bug Tracker, 2017-02-22