yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #76406
[Bug 1809136] Re: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound' on compute restart
Reviewed: https://review.openstack.org/626228
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1def76a1c49032d93ab6c7ee61dbbfe8e29cafca
Submitter: Zuul
Branch: master
commit 1def76a1c49032d93ab6c7ee61dbbfe8e29cafca
Author: Stephen Finucane <sfinucan@xxxxxxxxxx>
Date: Wed Dec 19 16:03:22 2018 +0000
Handle unbound vif plug errors on compute restart
As with change Ia963a093a1b26d90b4de2e8fc623031cf175aece, we can
sometimes cache failed port binding information which we'll see on
startup. Long term, the fix for both issues is to figure out how this is
being cached and stop that happening but for now we simply need to allow
the service to start up.
To this end, we copy the approach in the aforementioned change and
implement a translation function in os_vif_util for unbound which
will make the plug_vifs code raise VirtualInterfacePlugException which
is what the _init_instance code in ComputeManager is already handling.
This has the same caveats as that change, namely that there may be
smarter ways to do this that we should explore. However, that change
also included a note which goes someway to explaining this.
Change-Id: Iaec1f6fd12dba8b11991b7a7595593d5c8b1db50
Signed-off-by: Stephen Finucane <sfinucan@xxxxxxxxxx>
Related-bug: #1784579
Closes-bug: #1809136
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809136
Title:
Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound' on
compute restart
Status in OpenStack Compute (nova):
Fix Released
Bug description:
This is a variant of an existing bug:
- https://bugs.launchpad.net/nova/+bug/1738373 tracks a similar
exception ('_nova_to_osvif_vif_binding_failed') on compute startup.
There are also two other closely related bugs:
- https://bugs.launchpad.net/nova/+bug/1783917 tracks this same exception ('_nova_to_osvif_vif_unbound') but for live migrations
- https://bugs.launchpad.net/nova/+bug/1784579 tracks a similar exception ('_nova_to_osvif_vif_binding_failed') but for live migration
In addition, there are a few bugs which are likely the root cause of
all of the above issues (and this one) in the first place:
- https://bugs.launchpad.net/nova/+bug/1751923
In this instance, as with bug 1738373, we are unable to start nova-
compute service on compute node due to an os-vif invoked error.
nova-compute.log on compute shows:
2018-05-12 16:42:47.323 305978 INFO os_vif [req-0a72cdea-843a-4932-b8a0-bc24c2f21d9f - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:41:a9:2c,bridge_name='qbr8d027ff4-23',has_traffic_filtering=True,id=8d027ff4-2328-47df-9f9a-2c1a9914a83b,network=Network(9a98b244-b1d2-46b3-ab0e-be8456e3a984),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8d027ff4-23')
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service [req-0a72cdea-843a-4932-b8a0-bc24c2f21d9f - - - - -] Error starting thread.
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service Traceback (most recent call last):
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service service.start()
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 117, in start
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.manager.init_host()
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1154, in init_host
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self._init_instance(context, instance)
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 957, in _init_instance
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 703, in plug_vifs
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.vif_driver.plug(instance, vif)
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 771, in plug
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service vif_obj = os_vif_util.nova_to_osvif_vif(vif)
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 408, in nova_to_osvif_vif
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service {'type': vif['type'], 'func': funcname})
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'
2018-05-12 16:42:47.369 305978 ERROR oslo_service.service
Inspecting the available ports shows the port does exist, so this
looks like a caching issue.
[stack@director:~]$ neutron port-list | grep fa:16:3e:41:a9:2c
| 8d027ff4-2328-47df-9f9a-2c1a9914a83b | | fa:16:3e:41:a9:2c | {"subnet_id": "1f5ed9bc-aa7d-49bd-ac48-23b430fc0eb4", "ip_address": "172.19.9.17"} |
[stack@director:~]$ neutron port-show 8d027ff4-2328-47df-9f9a-2c1a9914a83b
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | overcloud-compute-7.localdomain |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-10-31T12:31:45Z |
| description | |
| device_id | b4ef4d0b-9e39-4741-a2dd-7fd7c066d13b |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "1f5ed9bc-aa7d-49bd-ac48-23b430fc0eb4", "ip_address": "172.19.9.17"} |
| id | 8d027ff4-2328-47df-9f9a-2c1a9914a83b |
| mac_address | fa:16:3e:41:a9:2c |
| name | |
| network_id | 9a98b244-b1d2-46b3-ab0e-be8456e3a984 |
| port_security_enabled | True |
| project_id | 3b2049626c954cdc9147beee2d34b441 |
| qos_policy_id | |
| revision_number | 184 |
| security_groups | 97aa0764-c0b5-47d1-88b2-285673d46a31 |
| | c7addc13-5a77-4322-953a-9d89d42468e6 |
| | cecdad42-7c78-45e7-9ec2-fef1086dbb7e |
| | de0a6da8-c44e-475f-90fd-1fb625840c52 |
| status | ACTIVE |
| tenant_id | 3b2049626c954cdc9147beee2d34b441 |
| updated_at | 2018-05-12T15:37:46Z |
+-----------------------+------------------------------------------------------------------------------------+
We should figure out why the invalid cache is getting saved, but we're
going to track that effort separately. For now, we should just focus
on letting the service start, putting instances with errors like this
into error state.
This was originally reported here
https://bugzilla.redhat.com/show_bug.cgi?id=1578028
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809136/+subscriptions
References
