← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #74974

[Bug 1795064] [NEW] SR-IOV error IndexError: pop from empty list

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I am building SR-IOV support in compute node on Queens i have following
NIC card and VF enabled

[root@ostack-compute-63 ~]# lspci -nn | grep -i eth
03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
03:01.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.2 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.3 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.4 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.5 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
03:01.6 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]


I have setup everything according official documents and so far everything looks good. 

I have created neutron-port and when i trying to launch instance i got
following error on compute node.


2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager [req-095c3f53-a558-4178-84ee-cf79bf7f3c7c eebe97b4bc714b8f814af8a44d08c2a4 2927a06cf30f4f7e938fdda2cc05aed2 - default default] Instance failed network setup after 1 attempt(s): IndexError: pop from empty list
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager Traceback (most recent call last):
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/compute/manager.py", line 1398, in _allocate_network_async
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id, available_macs, requested_ports_dict)
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     vif.destroy()
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     self.force_reraise()
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1042, in _update_ports_for_instance
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1192, in _populate_neutron_extension_values
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     port_req_body)
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1138, in _populate_neutron_binding_profile
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     instance, pci_request_id).pop()
2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager IndexError: pop from empty list


------------- also every 60 second i am getting following error
---------------

2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = execute(f, *args, **kwargs)
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     six.reraise(c, e, tb)
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = meth(*args, **kwargs)
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/libvirt.py", line 4232, in nodeDeviceLookupByName
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self)
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp3s1f4_00_00_00_00_00_00'
2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager


also i was reading this but it didn't help: http://lists.openstack.org/pipermail/openstack/2018-January/045982.html

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1795064

Title:
  SR-IOV error IndexError: pop from empty list

Status in OpenStack Compute (nova):
  New

Bug description:
  I am building SR-IOV support in compute node on Queens i have
  following NIC card and VF enabled

  [root@ostack-compute-63 ~]# lspci -nn | grep -i eth
  03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
  03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
  03:01.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.2 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.3 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.4 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.5 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.6 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]

  
  I have setup everything according official documents and so far everything looks good. 

  I have created neutron-port and when i trying to launch instance i got
  following error on compute node.

  
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager [req-095c3f53-a558-4178-84ee-cf79bf7f3c7c eebe97b4bc714b8f814af8a44d08c2a4 2927a06cf30f4f7e938fdda2cc05aed2 - default default] Instance failed network setup after 1 attempt(s): IndexError: pop from empty list
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager Traceback (most recent call last):
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/compute/manager.py", line 1398, in _allocate_network_async
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id, available_macs, requested_ports_dict)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     vif.destroy()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     self.force_reraise()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1042, in _update_ports_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1192, in _populate_neutron_extension_values
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     port_req_body)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1138, in _populate_neutron_binding_profile
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     instance, pci_request_id).pop()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager IndexError: pop from empty list


  ------------- also every 60 second i am getting following error
  ---------------

  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = execute(f, *args, **kwargs)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     six.reraise(c, e, tb)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = meth(*args, **kwargs)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/libvirt.py", line 4232, in nodeDeviceLookupByName
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp3s1f4_00_00_00_00_00_00'
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager

  
  also i was reading this but it didn't help: http://lists.openstack.org/pipermail/openstack/2018-January/045982.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1795064/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next