yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1579882] Re: Neutron l3 agent fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Assaf Muller <amuller@xxxxxxxxxx>
Date: Mon, 09 May 2016 21:33:52 -0000
Reply-to: Bug 1579882 <1579882@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
The way I'm reading this, this is a deployment type issue, not a Neutron
bug. I highly recommend ask.openstack.org, it's a very active Q&A type
resource.

I'm setting this to not-a-bug for now. If later you discover a concrete
bug, reply here and I'll re-open the bug.

** Changed in: neutron
     Assignee: Miguel Lavalle (minsel) => (unassigned)

** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1579882

Title:
  Neutron l3 agent fails

Status in neutron:
  Invalid

Bug description:
  Currently I am running LXD on Ubuntu 15.10. I have created two
  containers, controller and compute. Ive gone ahead and installed and
  setup all services. The only issue I am having is with
  neutron-l3-agent and neutron-dhcp-agent. After completing
  configuration I am seeing the following errors:

  neutron-l3-agent.log:
  2016-05-09 19:31:40.558 5045 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tap2d9c4091-9c', 'type', 'veth', 'peer', 'name', 'qr-2d9c4091-9c', 'netns', 'qrouter-9b021858-d363-4e0b-9691-6cedc2af5bcb'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:84
  2016-05-09 19:31:41.390 5045 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists

  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 371, in call
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 960, in process
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     self._process_internal_ports(agent.pd)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 477, in _process_internal_ports
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     self.internal_network_added(p)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 400, in internal_network_added
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     mtu=port.get('mtu'))
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 374, in _internal_network_added
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     prefix=prefix, mtu=mtu)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     bridge, namespace, prefix, mtu)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 447, in plug_new
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     namespace2=namespace)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 174, in add_veth
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     self._as_root([], 'link', tuple(args))
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info
  2016-05-09 19:31:41.394 5045 ERROR neutron.agent.l3.router_info
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9b021858-d363-4e0b-9691-6cedc2af5bcb'
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in _process_router_update
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 436, in _process_router_if_compatible
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 450, in _process_updated_router
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     ri.process(self)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 374, in call
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self.logger(e)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self.force_reraise()
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 371, in call
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 960, in process
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 477, in _process_internal_ports
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self.internal_network_added(p)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 400, in internal_network_added
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     mtu=port.get('mtu'))
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 374, in _internal_network_added
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     prefix=prefix, mtu=mtu)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     bridge, namespace, prefix, mtu)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 447, in plug_new
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     namespace2=namespace)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 174, in add_veth
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     self._as_root([], 'link', tuple(args))
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
  2016-05-09 19:31:41.405 5045 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists

  
  From neutron-dhcp-agent.log:
  2016-05-09 19:35:37.743 14807 INFO neutron.agent.dhcp.agent [-] Finished network 04368988-45c0-4e85-af4c-8348d2533a0b dhcp configuration
  2016-05-09 19:35:38.212 14807 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 1c75e4aaaef14db28ebc733074f2ad06 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 04735962-e569-470a-8247-5a316be5839b.
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 208, in enable
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1240, in setup
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     self.plugin.release_dhcp_port(network.id, port.device_id)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     self.force_reraise()
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     six.reraise(self.type_, self.value, self.tb)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     mtu=network.get('mtu'))
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 248, in plug
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     bridge, namespace, prefix, mtu)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 449, in plug_new
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     ns_veth.link.set_address(mac_address)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 499, in set_address
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     self._as_root([], ('set', self.name, 'address', mac_address))
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     use_root_namespace=use_root_namespace)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     log_fail_as_error=self.log_fail_as_error)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent     raise RuntimeError(msg)
  2016-05-09 19:35:38.217 14807 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: mount of /sys failed: Operation not permitted

  
  output of ip netns:

  root@controller:/sys# ip netns
  qdhcp-04368988-45c0-4e85-af4c-8348d2533a0b (id: 3)
  qdhcp-04735962-e569-470a-8247-5a316be5839b (id: 2)
  qrouter-9b021858-d363-4e0b-9691-6cedc2af5bcb (id: 1)

  
  l3 agent config:

  root@controller:~# cat /etc/neutron/l3_agent.ini |grep -v "#"|grep -v -e '^$'
  [DEFAULT]
  interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
  external_network_bridge = 
  debug = true
  verbose = true
  [AGENT]

  ml2_conf:

  root@controller:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -v "#"|grep -v -e '^$'
  [DEFAULT]
  [ml2]
  type_drivers = flat,vlan,vxlan
  tenant_network_types = vxlan
  mechanism_drivers = linuxbridge,l2population
  extension_drivers = port_security
  [ml2_type_flat]
  flat_networks = provider
  [ml2_type_geneve]
  [ml2_type_gre]
  [ml2_type_vlan]
  [ml2_type_vxlan]
  vni_ranges = 1:1000
  [securitygroup]
  enable_ipset = true

  linux_bridge
  root@controller:~# cat /etc/neutron/plugins/ml2/linuxbridge_agent.ini |grep -v "#"|grep -v -e '^$'
  [DEFAULT]
  [agent]
  [linux_bridge]
  physical_interface_mappings = provider:eth0
  [securitygroup]
  firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
  enable_security_group = true
  [vxlan]
  enable_vxlan = true
  local_ip = 172.16.1.51
  l2_population = true

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1579882/+subscriptions
References

[Bug 1579882] [NEW] Neutron l3 agent fails
From: Daniel, 2016-05-09