yahoo-eng-team team mailing list archive

[iain MacDonnell <1830295@xxxxxxxxxxxxxxxxxx>]

Reinstalling oslo.privsep seems to have "fixed" it on the 16.04 box too.
Still don't understand why - must have been some bad cache or something.
Will put this down to "gremlins" unless it resurfaces :/

** Changed in: neutron
       Status: New => Invalid

** Changed in: oslo.privsep
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1830295

Title:
  devstack py3 get_link_devices() KeyError: 'index'

Status in neutron:
  Invalid
Status in oslo.privsep:
  Invalid

Bug description:
  devstack master with py3. openvswitch agent has suddenly stopped
  working, with no change in config or environment (other than
  rebuilding devstack). Stack trace below. For some reason (yet
  undetermined), privileged.get_link_devices() now seems to be returning
  byte arrays instead of strings as the dict keys:

  >>> from neutron.privileged.agent.linux import ip_lib as privileged
  >>> privileged.get_link_devices(None)[0].keys() 
  dict_keys([b'index', b'family', b'__align', b'header', b'flags', b'ifi_type', b'event', b'change', b'attrs'])
  >>> 

  
  From agent startup:

  neutron-openvswitch-agent[42936]: ERROR neutron Traceback (most recent call last):
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/bin/neutron-openvswitch-agent", line 10, in <module>
  neutron-openvswitch-agent[42936]: ERROR neutron     sys.exit(main())
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     agent_main.main()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 47, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     mod.main()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/base/app_manager.py", line 375, in run_apps
  neutron-openvswitch-agent[42936]: ERROR neutron     hub.joinall(services)  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 102, in joinall
  neutron-openvswitch-agent[42936]: ERROR neutron     t.wait()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 180, in wait
  neutron-openvswitch-agent[42936]: ERROR neutron     return self._exit_event.wait()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
  neutron-openvswitch-agent[42936]: ERROR neutron     current.throw(*self._exc)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 219, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     result = function(*args, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 64, in _launch
  neutron-openvswitch-agent[42936]: ERROR neutron     raise e
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 59, in _launch
  neutron-openvswitch-agent[42936]: ERROR neutron     return func(*args, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
  neutron-openvswitch-agent[42936]: ERROR neutron     LOG.exception("Agent main thread died of an exception")
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  neutron-openvswitch-agent[42936]: ERROR neutron     self.force_reraise()     
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  neutron-openvswitch-agent[42936]: ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
  neutron-openvswitch-agent[42936]: ERROR neutron     raise value
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
  neutron-openvswitch-agent[42936]: ERROR neutron     ovs_agent.main(bridge_classes)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2393, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     validate_tunnel_config(cfg.CONF.AGENT.tunnel_types, cfg.CONF.OVS.local_ip)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2362, in validate_tunnel_config
  neutron-openvswitch-agent[42936]: ERROR neutron     validate_local_ip(local_ip)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2350, in validate_local_ip
  neutron-openvswitch-agent[42936]: ERROR neutron     if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 187, in get_device_by_ip 
  neutron-openvswitch-agent[42936]: ERROR neutron     devices = get_devices_with_ip(self.namespace, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in get_devices_with_ip
  neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in <genexpr>
  neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1388, in _parse_link_device
  neutron-openvswitch-agent[42936]: ERROR neutron     index=device['index'],   
  neutron-openvswitch-agent[42936]: ERROR neutron KeyError: 'index'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1830295/+subscriptions