yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1830295] [NEW] devstack py3 get_link_devices() KeyError: 'index'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: iain MacDonnell <1830295@xxxxxxxxxxxxxxxxxx>
Date: Fri, 24 May 2019 00:17:59 -0000
Reply-to: Bug 1830295 <1830295@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

devstack master with py3. openvswitch agent has suddenly stopped
working, with no change in config or environment (other than rebuilding
devstack). Stack trace below. For some reason (yet undetermined),
privileged.get_link_devices() now seems to be returning byte arrays
instead of strings as the dict keys:

>>> from neutron.privileged.agent.linux import ip_lib as privileged
>>> privileged.get_link_devices(None)[0].keys() 
dict_keys([b'index', b'family', b'__align', b'header', b'flags', b'ifi_type', b'event', b'change', b'attrs'])
>>> 


>From agent startup:

neutron-openvswitch-agent[42936]: ERROR neutron Traceback (most recent call last):
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/bin/neutron-openvswitch-agent", line 10, in <module>
neutron-openvswitch-agent[42936]: ERROR neutron     sys.exit(main())
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
neutron-openvswitch-agent[42936]: ERROR neutron     agent_main.main()
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 47, in main
neutron-openvswitch-agent[42936]: ERROR neutron     mod.main()
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
neutron-openvswitch-agent[42936]: ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/base/app_manager.py", line 375, in run_apps
neutron-openvswitch-agent[42936]: ERROR neutron     hub.joinall(services)  
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 102, in joinall
neutron-openvswitch-agent[42936]: ERROR neutron     t.wait()
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 180, in wait
neutron-openvswitch-agent[42936]: ERROR neutron     return self._exit_event.wait()
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
neutron-openvswitch-agent[42936]: ERROR neutron     current.throw(*self._exc)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 219, in main
neutron-openvswitch-agent[42936]: ERROR neutron     result = function(*args, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 64, in _launch
neutron-openvswitch-agent[42936]: ERROR neutron     raise e
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 59, in _launch
neutron-openvswitch-agent[42936]: ERROR neutron     return func(*args, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
neutron-openvswitch-agent[42936]: ERROR neutron     LOG.exception("Agent main thread died of an exception")
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
neutron-openvswitch-agent[42936]: ERROR neutron     self.force_reraise()     
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
neutron-openvswitch-agent[42936]: ERROR neutron     six.reraise(self.type_, self.value, self.tb)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
neutron-openvswitch-agent[42936]: ERROR neutron     raise value
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
neutron-openvswitch-agent[42936]: ERROR neutron     ovs_agent.main(bridge_classes)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2393, in main
neutron-openvswitch-agent[42936]: ERROR neutron     validate_tunnel_config(cfg.CONF.AGENT.tunnel_types, cfg.CONF.OVS.local_ip)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2362, in validate_tunnel_config
neutron-openvswitch-agent[42936]: ERROR neutron     validate_local_ip(local_ip)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2350, in validate_local_ip
neutron-openvswitch-agent[42936]: ERROR neutron     if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 187, in get_device_by_ip 
neutron-openvswitch-agent[42936]: ERROR neutron     devices = get_devices_with_ip(self.namespace, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in get_devices_with_ip
neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in <genexpr>
neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1388, in _parse_link_device
neutron-openvswitch-agent[42936]: ERROR neutron     index=device['index'],   
neutron-openvswitch-agent[42936]: ERROR neutron KeyError: 'index'

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1830295

Title:
  devstack py3 get_link_devices() KeyError: 'index'

Status in neutron:
  New

Bug description:
  devstack master with py3. openvswitch agent has suddenly stopped
  working, with no change in config or environment (other than
  rebuilding devstack). Stack trace below. For some reason (yet
  undetermined), privileged.get_link_devices() now seems to be returning
  byte arrays instead of strings as the dict keys:

  >>> from neutron.privileged.agent.linux import ip_lib as privileged
  >>> privileged.get_link_devices(None)[0].keys() 
  dict_keys([b'index', b'family', b'__align', b'header', b'flags', b'ifi_type', b'event', b'change', b'attrs'])
  >>> 

  
  From agent startup:

  neutron-openvswitch-agent[42936]: ERROR neutron Traceback (most recent call last):
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/bin/neutron-openvswitch-agent", line 10, in <module>
  neutron-openvswitch-agent[42936]: ERROR neutron     sys.exit(main())
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     agent_main.main()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 47, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     mod.main()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/base/app_manager.py", line 375, in run_apps
  neutron-openvswitch-agent[42936]: ERROR neutron     hub.joinall(services)  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 102, in joinall
  neutron-openvswitch-agent[42936]: ERROR neutron     t.wait()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 180, in wait
  neutron-openvswitch-agent[42936]: ERROR neutron     return self._exit_event.wait()
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
  neutron-openvswitch-agent[42936]: ERROR neutron     current.throw(*self._exc)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 219, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     result = function(*args, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 64, in _launch
  neutron-openvswitch-agent[42936]: ERROR neutron     raise e
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 59, in _launch
  neutron-openvswitch-agent[42936]: ERROR neutron     return func(*args, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
  neutron-openvswitch-agent[42936]: ERROR neutron     LOG.exception("Agent main thread died of an exception")
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  neutron-openvswitch-agent[42936]: ERROR neutron     self.force_reraise()     
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  neutron-openvswitch-agent[42936]: ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
  neutron-openvswitch-agent[42936]: ERROR neutron     raise value
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
  neutron-openvswitch-agent[42936]: ERROR neutron     ovs_agent.main(bridge_classes)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2393, in main
  neutron-openvswitch-agent[42936]: ERROR neutron     validate_tunnel_config(cfg.CONF.AGENT.tunnel_types, cfg.CONF.OVS.local_ip)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2362, in validate_tunnel_config
  neutron-openvswitch-agent[42936]: ERROR neutron     validate_local_ip(local_ip)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2350, in validate_local_ip
  neutron-openvswitch-agent[42936]: ERROR neutron     if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 187, in get_device_by_ip 
  neutron-openvswitch-agent[42936]: ERROR neutron     devices = get_devices_with_ip(self.namespace, **kwargs)
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in get_devices_with_ip
  neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in <genexpr>
  neutron-openvswitch-agent[42936]: ERROR neutron     for device in devices):  
  neutron-openvswitch-agent[42936]: ERROR neutron   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1388, in _parse_link_device
  neutron-openvswitch-agent[42936]: ERROR neutron     index=device['index'],   
  neutron-openvswitch-agent[42936]: ERROR neutron KeyError: 'index'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1830295/+subscriptions
Follow ups

[Bug 1830295] Re: devstack py3 get_link_devices() KeyError: 'index'
From: iain MacDonnell, 2019-05-28
[Bug 1830295] Re: devstack py3 get_link_devices() KeyError: 'index'
From: iain MacDonnell, 2019-05-24