yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1497309] Re: l3-agent unable to parse output from ip netns list (iproute2 >= 4.0)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dave Walker <email@xxxxxxxxxx>
Date: Thu, 21 Jan 2016 20:27:42 -0000
Reply-to: Bug 1497309 <1497309@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron/kilo
   Importance: Undecided
       Status: New

** Changed in: neutron/kilo
       Status: New => Fix Committed

** Changed in: neutron/kilo
    Milestone: None => 2015.1.3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1497309

Title:
  l3-agent unable to parse output from ip netns list (iproute2 >= 4.0)

Status in neutron:
  Fix Released
Status in neutron kilo series:
  Fix Committed
Status in neutron package in Ubuntu:
  Fix Released

Bug description:
  When run through sudo, ip netns has some extra output on Ubuntu wily:

  $ sudo ip netns
  qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1)
  qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)

  and from l3-agent:

  2015-09-18 14:15:47.889 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 59f744cafcab474baee68232c4cf70e9 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
  2015-09-18 14:15:47.895 26554 DEBUG neutron.agent.l3.agent [-] Starting _process_routers_loop _process_routers_loop /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:509
  2015-09-18 14:15:47.896 26554 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron.agent.l3.agent.L3NATAgentWithStateReport._report_state' sleeping for 29.93 seconds _run_loop /usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
  2015-09-18 14:15:47.916 26554 DEBUG neutron.agent.l3.agent [-] Starting periodic_sync_routers_task - fullsync:True periodic_sync_routers_task /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:521
  2015-09-18 14:15:47.953 26554 DEBUG neutron.agent.linux.utils [-]
  Command: ['ip', 'netns', 'list']
  Exit code: 0
   execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151
  2015-09-18 14:15:47.954 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is d04be6386ef7495ebeb3cb656fb330a8 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
  2015-09-18 14:15:48.268 26554 DEBUG neutron.agent.l3.agent [-] Processing :[{u'status': u'ACTIVE', u'_interfaces': [{u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'192.168.21.0/24', u'gateway_ip': u'192.168.21.1', u'id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'35fc068a-750d-4add-b1d2-af392dbd8790', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:3f:4a:90', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_interface', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'prefixlen': 24, u'ip_address': u'192.168.21.1'}], u'id': u'bfc7b6e2-29a7-428b-b7fd-0675e9bf5df8', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}], u'enable_snat': True, u'ha_vr_id': 0, u'gw_port_host': None, u'gw_port_id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'admin_state_up': True, u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'gw_port': {u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'10.5.0.0/16', u'gateway_ip': u'10.5.0.1', u'id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:6f:24:28', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_gateway', u'tenant_id': u'', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'prefixlen': 16, u'ip_address': u'10.5.150.0'}], u'id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}, u'distributed': False, u'_snat_router_interfaces': [], u'_floatingip_agent_interfaces': [], u'_floatingips': [{u'router_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'status': u'DOWN', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'floating_network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'fixed_ip_address': u'192.168.21.3', u'floating_ip_address': u'10.5.150.1', u'port_id': u'a5d45770-98ea-4a2b-b839-5274e783abca', u'id': u'7067f627-91ea-4917-ab31-48570d3c397b'}], u'routes': [], u'external_gateway_info': {u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'enable_snat': True, u'external_fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'ip_address': u'10.5.150.0'}]}, u'ha': False, u'id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'name': u'provider-router'}] fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:555
  2015-09-18 14:15:48.272 26554 DEBUG neutron.agent.l3.agent [-] periodic_sync_routers_task successfully completed fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:570
  2015-09-18 14:15:48.273 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240
  2015-09-18 14:15:48.274 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240
  2015-09-18 14:15:48.276 26554 DEBUG neutron.agent.linux.external_process [-] No process started for 49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0) disable /usr/lib/python2.7/dist-packages/neutron/agent/linux/external_process.py:118
  2015-09-18 14:15:48.287 26554 DEBUG neutron.agent.l3.agent [-] Starting router update for 49c6d7b1-8399-4944-81ad-093b6e786db0, action None, priority 1 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:456
  2015-09-18 14:15:48.326 26554 DEBUG neutron.agent.linux.utils [-]
  Command: ['ip', '-o', 'link', 'show', 'br-ex']
  Exit code: 0
   execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151
  2015-09-18 14:15:48.329 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 1502ca45d5e844e3937f095d9af27875 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
  2015-09-18 14:15:48.359 26554 ERROR neutron.agent.linux.utils [-]
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  Exit code: 1
  Stdin:
  Stdout:
  Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory

  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager [-] Failed to destroy stale namespace <neutron.agent.l3.namespaces.RouterNamespace object at 0x7f6d668b7110>
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Traceback (most recent call last):
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespace_manager.py", line 143, in _cleanup
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager     ns.delete()
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespaces.py", line 101, in delete
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager     for d in ns_ip.get_devices(exclude_loopback=True):
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 112, in get_devices
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager     log_fail_as_error=self.log_fail_as_error
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 154, in execute
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager     raise RuntimeError(m)
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager RuntimeError:
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Exit code: 1
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdin:
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdout:
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager
  2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager

  ProblemType: Bug
  DistroRelease: Ubuntu 15.10
  Package: neutron-l3-agent 2:7.0.0~b3-0ubuntu2
  ProcVersionSignature: Ubuntu 4.2.0-7.7-generic 4.2.0
  Uname: Linux 4.2.0-7-generic x86_64
  ApportVersion: 2.18.1-0ubuntu1
  Architecture: amd64
  Date: Fri Sep 18 14:16:35 2015
  JournalErrors:
   -- Logs begin at Fri 2015-09-18 14:01:29 UTC, end at Fri 2015-09-18 14:16:43 UTC. --
   Sep 18 14:06:25 hostname kernel: conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module.
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1497309/+subscriptions