yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1893292] [NEW] sr-iov port binding failed lead to nova-compute restart

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yawei yang <1893292@xxxxxxxxxxxxxxxxxx>
Date: Fri, 28 Aug 2020 02:22:40 -0000
Reply-to: Bug 1893292 <1893292@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
I did restart nova-compute,but nova-compute service wasn't running. query log found '_noca_to_osvif_vif_binding_failed' lead to nova-compute unable to run

log file
"""
2020-08-24 20:21:08.453 6 INFO nova.virt.libvirt.driver [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] VIF driver: <nova.virt.libvirt.vif.LibvirtGenericVIFDriver object at 0x7f4d8da35e10>
2020-08-24 20:21:08.457 6 INFO oslo.privsep.daemon [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpxvlDxS/privsep.sock']
2020-08-24 20:21:10.124 6 INFO oslo.privsep.daemon [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Spawned new privsep daemon via rootwrap
2020-08-24 20:21:10.045 61 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-08-24 20:21:10.052 61 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-08-24 20:21:10.057 61 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2020-08-24 20:21:10.057 61 INFO oslo.privsep.daemon [-] privsep daemon running as pid 61
2020-08-24 20:21:10.776 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:60:e1:8f,bridge_name='qbr87ed9c11-92',has_traffic_filtering=True,id=87ed9c11-9210-4ff8-a61f-0d0603a05234,network=Network(dd12c83c-ca1a-443c-b57d-683fc984fb37),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap87ed9c11-92')
2020-08-24 20:21:11.337 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:30:d2:ff,bridge_name='qbr76b0f80e-91',has_traffic_filtering=True,id=76b0f80e-91ba-44ea-8906-4b1967639f15,network=Network(b5adfd9b-897a-471d-9209-936934bd799d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap76b0f80e-91')
2020-08-24 20:21:11.865 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:be:af:45,bridge_name='qbr0183de38-8d',has_traffic_filtering=True,id=0183de38-8dde-4d7d-80ab-b4160d5282fd,network=Network(1f87f85a-e29c-4c70-9b7a-e3a80b5140ca),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap0183de38-8d')
2020-08-24 20:21:12.419 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:47:fa:b1,bridge_name='qbr6e1d5f5f-a0',has_traffic_filtering=True,id=6e1d5f5f-a04c-4b21-bfc4-f65ee3c3df46,network=Network(dd12c83c-ca1a-443c-b57d-683fc984fb37),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap6e1d5f5f-a0')
2020-08-24 20:21:12.993 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:66:42:d5,bridge_name='qbr66ffd35a-46',has_traffic_filtering=True,id=66ffd35a-461f-42a8-8d7a-05d141fe83d2,network=Network(1f87f85a-e29c-4c70-9b7a-e3a80b5140ca),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap66ffd35a-46')
2020-08-24 20:21:13.034 6 ERROR oslo_service.service [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Error starting thread.: NovaException: Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
2020-08-24 20:21:13.034 6 ERROR oslo_service.service Traceback (most recent call last):
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_service/service.py", line 794, in run_service
2020-08-24 20:21:13.034 6 ERROR oslo_service.service     service.start()
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/service.py", line 162, in start
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/compute/manager.py", line 972, in _init_instance
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/virt/libvirt/driver.py", line 815, in plug_vifs
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/virt/libvirt/vif.py", line 801, in plug
2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/network/os_vif_util.py", line 505, in nova_to_osvif_vif
2020-08-24 20:21:13.034 6 ERROR oslo_service.service NovaException: Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
2020-08-24 20:21:13.034 6 ERROR oslo_service.service 
2020-08-24 20:21:17.528 7 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge, noop
2020-08-24 20:21:18.008 7 INFO nova.compute.rpcapi [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35
2020-08-24 20:21:18.010 7 INFO nova.virt.driver [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Loading compute driver 'libvirt.LibvirtDriver'
2020-08-24 20:21:18.011 7 WARNING oslo_config.cfg [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Deprecated: Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
nova-network is deprecated, as are any related configuration options.
).  Its value may be silently ignored in the future.
2020-08-24 20:21:18.017 7 WARNING os_brick.initiator.connectors.remotefs [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Connection details not present. RemoteFsClient may not initialize properly.
2020-08-24 20:21:18.019 7 INFO nova.inspur.compute.manager [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Loading inspur compute driver inspur.InspurDriver .
2020-08-24 20:21:18.021 7 WARNING os_brick.initiator.connectors.remotefs [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Connection details not present. RemoteFsClient may not initialize properly.
2020-08-24 20:21:18.024 7 WARNING nova.compute.monitors [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).
2020-08-24 20:21:18.048 7 WARNING oslo_config.cfg [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.
2020-08-24 20:21:18.060 7 INFO nova.service [-] Starting compute node (version 18.0.0)
2020-08-24 20:21:18.073 7 INFO nova.virt.libvirt.driver [-] Connection event '1' reason 'None'
2020-08-24 20:21:18.114 7 INFO nova.virt.libvirt.host [-] Libvirt host capabilities <capabilities>

""""

Steps to reproduce
==================
1、create a vm with SR-IOV
2、find this SR-IOV port id，enter database set the port id 'vif_type'='binding_failed'
3、restart nova-compute service 

Expected result
===============
nova-compute running

Actual result
=============
nova-compute unable to run

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1893292

Title:
  sr-iov port binding failed lead to nova-compute restart

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I did restart nova-compute,but nova-compute service wasn't running. query log found '_noca_to_osvif_vif_binding_failed' lead to nova-compute unable to run

  log file
  """
  2020-08-24 20:21:08.453 6 INFO nova.virt.libvirt.driver [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] VIF driver: <nova.virt.libvirt.vif.LibvirtGenericVIFDriver object at 0x7f4d8da35e10>
  2020-08-24 20:21:08.457 6 INFO oslo.privsep.daemon [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpxvlDxS/privsep.sock']
  2020-08-24 20:21:10.124 6 INFO oslo.privsep.daemon [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Spawned new privsep daemon via rootwrap
  2020-08-24 20:21:10.045 61 INFO oslo.privsep.daemon [-] privsep daemon starting
  2020-08-24 20:21:10.052 61 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2020-08-24 20:21:10.057 61 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
  2020-08-24 20:21:10.057 61 INFO oslo.privsep.daemon [-] privsep daemon running as pid 61
  2020-08-24 20:21:10.776 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:60:e1:8f,bridge_name='qbr87ed9c11-92',has_traffic_filtering=True,id=87ed9c11-9210-4ff8-a61f-0d0603a05234,network=Network(dd12c83c-ca1a-443c-b57d-683fc984fb37),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap87ed9c11-92')
  2020-08-24 20:21:11.337 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:30:d2:ff,bridge_name='qbr76b0f80e-91',has_traffic_filtering=True,id=76b0f80e-91ba-44ea-8906-4b1967639f15,network=Network(b5adfd9b-897a-471d-9209-936934bd799d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap76b0f80e-91')
  2020-08-24 20:21:11.865 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:be:af:45,bridge_name='qbr0183de38-8d',has_traffic_filtering=True,id=0183de38-8dde-4d7d-80ab-b4160d5282fd,network=Network(1f87f85a-e29c-4c70-9b7a-e3a80b5140ca),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap0183de38-8d')
  2020-08-24 20:21:12.419 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:47:fa:b1,bridge_name='qbr6e1d5f5f-a0',has_traffic_filtering=True,id=6e1d5f5f-a04c-4b21-bfc4-f65ee3c3df46,network=Network(dd12c83c-ca1a-443c-b57d-683fc984fb37),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap6e1d5f5f-a0')
  2020-08-24 20:21:12.993 6 INFO os_vif [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:66:42:d5,bridge_name='qbr66ffd35a-46',has_traffic_filtering=True,id=66ffd35a-461f-42a8-8d7a-05d141fe83d2,network=Network(1f87f85a-e29c-4c70-9b7a-e3a80b5140ca),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap66ffd35a-46')
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service [None req-fad58237-9d9e-43f8-87bb-78a75c06d9bb - - - - -] Error starting thread.: NovaException: Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service Traceback (most recent call last):
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_service/service.py", line 794, in run_service
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service     service.start()
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/service.py", line 162, in start
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/compute/manager.py", line 972, in _init_instance
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/virt/libvirt/driver.py", line 815, in plug_vifs
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/virt/libvirt/vif.py", line 801, in plug
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service   File "build/bdist.linux-x86_64/egg/nova/network/os_vif_util.py", line 505, in nova_to_osvif_vif
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service NovaException: Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
  2020-08-24 20:21:13.034 6 ERROR oslo_service.service 
  2020-08-24 20:21:17.528 7 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge, noop
  2020-08-24 20:21:18.008 7 INFO nova.compute.rpcapi [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35
  2020-08-24 20:21:18.010 7 INFO nova.virt.driver [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Loading compute driver 'libvirt.LibvirtDriver'
  2020-08-24 20:21:18.011 7 WARNING oslo_config.cfg [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Deprecated: Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
  nova-network is deprecated, as are any related configuration options.
  ).  Its value may be silently ignored in the future.
  2020-08-24 20:21:18.017 7 WARNING os_brick.initiator.connectors.remotefs [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Connection details not present. RemoteFsClient may not initialize properly.
  2020-08-24 20:21:18.019 7 INFO nova.inspur.compute.manager [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Loading inspur compute driver inspur.InspurDriver .
  2020-08-24 20:21:18.021 7 WARNING os_brick.initiator.connectors.remotefs [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Connection details not present. RemoteFsClient may not initialize properly.
  2020-08-24 20:21:18.024 7 WARNING nova.compute.monitors [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).
  2020-08-24 20:21:18.048 7 WARNING oslo_config.cfg [None req-1669e8e5-c6da-46a5-b59d-59c820a03360 - - - - -] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.
  2020-08-24 20:21:18.060 7 INFO nova.service [-] Starting compute node (version 18.0.0)
  2020-08-24 20:21:18.073 7 INFO nova.virt.libvirt.driver [-] Connection event '1' reason 'None'
  2020-08-24 20:21:18.114 7 INFO nova.virt.libvirt.host [-] Libvirt host capabilities <capabilities>

  """"

  Steps to reproduce
  ==================
  1、create a vm with SR-IOV
  2、find this SR-IOV port id，enter database set the port id 'vif_type'='binding_failed'
  3、restart nova-compute service 

  Expected result
  ===============
  nova-compute running

  Actual result
  =============
  nova-compute unable to run

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1893292/+subscriptions
Follow ups

[Bug 1893292] Re: sr-iov port binding failed lead to nova-compute restart
From: Launchpad Bug Tracker, 2020-11-03