yahoo-eng-team team mailing list archive

[OpenStack Infra <1795064@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/607650
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=035708c37d587e4c5ede7fe80270bdbff98016ac
Submitter: Zuul
Branch:    master

commit 035708c37d587e4c5ede7fe80270bdbff98016ac
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Wed Oct 3 12:54:53 2018 -0400

    Handle IndexError in _populate_neutron_binding_profile
    
    This fixes the code that was blindly pop'ing an entry
    from an empty list of PCI devices claimed by the instance.
    It's not exactly clear how we can get into this situation,
    presumably there was a failure in the actual PCI device
    claim logic in the ResourceTracker - maybe related to the
    configured PCI passthrough whitelist. Regardless, we should
    handle the empty PCI device list in this method and raise
    an appropriate exception to fail the build on this host.
    
    Change-Id: I401bb74cf6e17c2b72cc62bf8ec03ec58238c44a
    Closes-Bug: #1795064


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1795064

Title:
  SR-IOV error IndexError: pop from empty list

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed
Status in OpenStack Compute (nova) queens series:
  Confirmed
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  I am building SR-IOV support in compute node on Queens i have
  following NIC card and VF enabled

  [root@ostack-compute-63 ~]# lspci -nn | grep -i eth
  03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
  03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
  03:01.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.2 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.3 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.4 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.5 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]
  03:01.6 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af]

  
  I have setup everything according official documents and so far everything looks good. 

  I have created neutron-port and when i trying to launch instance i got
  following error on compute node.

  
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager [req-095c3f53-a558-4178-84ee-cf79bf7f3c7c eebe97b4bc714b8f814af8a44d08c2a4 2927a06cf30f4f7e938fdda2cc05aed2 - default default] Instance failed network setup after 1 attempt(s): IndexError: pop from empty list
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager Traceback (most recent call last):
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/compute/manager.py", line 1398, in _allocate_network_async
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id, available_macs, requested_ports_dict)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     vif.destroy()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     self.force_reraise()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1042, in _update_ports_for_instance
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     bind_host_id=bind_host_id)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1192, in _populate_neutron_extension_values
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     port_req_body)
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1138, in _populate_neutron_binding_profile
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager     instance, pci_request_id).pop()
  2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager IndexError: pop from empty list


  ------------- also every 60 second i am getting following error
  ---------------

  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = execute(f, *args, **kwargs)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     six.reraise(c, e, tb)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     rv = meth(*args, **kwargs)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager   File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/libvirt.py", line 4232, in nodeDeviceLookupByName
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager     if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self)
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp3s1f4_00_00_00_00_00_00'
  2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager

  
  also i was reading this but it didn't help: http://lists.openstack.org/pipermail/openstack/2018-January/045982.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1795064/+subscriptions