yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1808738] [NEW] No net device was found for VF

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Satish Patel <satish.txt@xxxxxxxxx>
Date: Mon, 17 Dec 2018 03:23:25 -0000
Reply-to: Bug 1808738 <1808738@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am running queens openstack with 150 SR-IOV compute nodes and
everything working great so far, but i am seeing following WARNING mesg
very frequently and not sure its a bug or configuration issue, can
someone provide clarity on this logs

compute node "ostack-compute-sriov-01" running 2 sr-iov instances and
each instance has two SR-IOV nic associated, so total 4 vf in use on
compute node.

[root@ostack-compute-sriov-01 ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000540              running
 2     instance-000005c4              running


[root@ostack-compute-sriov-01 ~]# lspci -v | grep -i eth | grep "Virtual Function"
03:09.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.2 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.3 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.4 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.5 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.6 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
03:09.7 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function


[root@ostack-compute-sriov-01 ~]# ip link show dev eno2
4: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether c4:34:6b:cb:a0:f4 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC fa:16:3e:2d:58:69, vlan 11, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 1 MAC fa:16:3e:a6:67:60, vlan 200, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 2 MAC fa:16:3e:9a:98:e0, vlan 200, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 3 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 4 MAC fa:16:3e:7d:ef:0c, vlan 11, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 5 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 6 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
    vf 7 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto


I am seeing following WARNING messages from all my compute nodes, interesting thing is all 4 lines timestamp is same so pretty much pop up in log file same time, I do have 


2018-12-16 22:11:05.070 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.4: PciDeviceNotFoundById: PCI device 0000:03:09.4 not found
2018-12-16 22:11:05.237 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.1: PciDeviceNotFoundById: PCI device 0000:03:09.1 not found
2018-12-16 22:11:05.242 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.2: PciDeviceNotFoundById: PCI device 0000:03:09.2 not found
2018-12-16 22:11:05.269 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.0: PciDeviceNotFoundById: PCI device 0000:03:09.0 not found


currently this warning not causing issue but worried if its related to
something big issue and i am not aware... if this is informative mesg
then how do i reduce it because otherside it creating noise in my logs
spelunking..

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1808738

Title:
  No net device was found for VF

Status in OpenStack Compute (nova):
  New

Bug description:
  I am running queens openstack with 150 SR-IOV compute nodes and
  everything working great so far, but i am seeing following WARNING
  mesg very frequently and not sure its a bug or configuration issue,
  can someone provide clarity on this logs

  compute node "ostack-compute-sriov-01" running 2 sr-iov instances and
  each instance has two SR-IOV nic associated, so total 4 vf in use on
  compute node.

  [root@ostack-compute-sriov-01 ~]# virsh list
   Id    Name                           State
  ----------------------------------------------------
   1     instance-00000540              running
   2     instance-000005c4              running

  
  [root@ostack-compute-sriov-01 ~]# lspci -v | grep -i eth | grep "Virtual Function"
  03:09.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.2 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.3 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.4 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.5 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.6 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function
  03:09.7 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function


  [root@ostack-compute-sriov-01 ~]# ip link show dev eno2
  4: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether c4:34:6b:cb:a0:f4 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC fa:16:3e:2d:58:69, vlan 11, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 1 MAC fa:16:3e:a6:67:60, vlan 200, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 2 MAC fa:16:3e:9a:98:e0, vlan 200, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 3 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 4 MAC fa:16:3e:7d:ef:0c, vlan 11, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 5 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 6 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto
      vf 7 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto

  
  I am seeing following WARNING messages from all my compute nodes, interesting thing is all 4 lines timestamp is same so pretty much pop up in log file same time, I do have 

  
  2018-12-16 22:11:05.070 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.4: PciDeviceNotFoundById: PCI device 0000:03:09.4 not found
  2018-12-16 22:11:05.237 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.1: PciDeviceNotFoundById: PCI device 0000:03:09.1 not found
  2018-12-16 22:11:05.242 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.2: PciDeviceNotFoundById: PCI device 0000:03:09.2 not found
  2018-12-16 22:11:05.269 40288 WARNING nova.pci.utils [req-0d87b5e4-6ece-4beb-880c-51c7c5835a66 - - - - -] No net device was found for VF 0000:03:09.0: PciDeviceNotFoundById: PCI device 0000:03:09.0 not found


  currently this warning not causing issue but worried if its related to
  something big issue and i am not aware... if this is informative mesg
  then how do i reduce it because otherside it creating noise in my logs
  spelunking..

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1808738/+subscriptions