yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1983623] [NEW] Fail to attach interface after update to Victoria

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: B Sperry <1983623@xxxxxxxxxxxxxxxxxx>
Date: Thu, 04 Aug 2022 22:23:06 -0000
Reply-to: Bug 1983623 <1983623@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx

Public bug reported:

Updated our cluster from Ussuri to Victoria and we are seeing issues
with instances that have an interface on one of our networks. Network is
named ext-net which happens to be the only shared + external net we have
setup.

Nodes are all running Centos 8.5. Cluster was setup with Kolla Ansible

When adding an interface attached to the ext-net to an existing instance I get this error from dashboard:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception_Remote.PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-3774e75d-ce1a-4897-96f7-85b4e4884b7c)

Nova logs that mention this request ID are:
nova/nova-api.log:47665:2022-08-04 16:51:08.887 24 ERROR nova.api.openstack.wsgi [req-3774e75d-ce1a-4897-96f7-85b4e4884b7c a28f518f46a34f45bc765ff756567715 5701f45c383b42fba7e95a3d23852ee5 - default 09d5536177654b04a4ba99a2b071ed0c] Unexpected exception in API method: nova.exception_Remote.PortBindingFailed_Remote: Binding failed for port e92486de-128d-4b61-a640-48f0bf80869b, please check neutron logs for more information.
nova/nova-api.log:47844:2022-08-04 16:51:08.890 24 INFO nova.api.openstack.wsgi [req-3774e75d-ce1a-4897-96f7-85b4e4884b7c a28f518f46a34f45bc765ff756567715 5701f45c383b42fba7e95a3d23852ee5 - default 09d5536177654b04a4ba99a2b071ed0c] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
nova/nova-api.log:47846:2022-08-04 16:51:08.892 24 INFO nova.api.openstack.requestlog [req-3774e75d-ce1a-4897-96f7-85b4e4884b7c a28f518f46a34f45bc765ff756567715 5701f45c383b42fba7e95a3d23852ee5 - default 09d5536177654b04a4ba99a2b071ed0c] 172.31.87.254 "POST /v2.1/servers/e9dfe6de-2c2b-4c9c-9e22-c3433b2cd697/os-interface" status: 500 len: 220 microversion: 2.1 time: 8.287640

I can attach more logs before/after those lines if more info is needed.

The controller and a handful of hypervisors have updated to Victoria. We
only see this error on instances that are running on hypervisors running
Victoria. I can attach an interface to the ext-net just fine on
instances running on hosts still using Ussuri.

Will also note I get errors when migrating an instance with an interface
on ext-net from a Ussuri to a Victoria hypervisor. The error messages in
that case is similar (PortBindingError), I can paste in those errors
also but expect they are the same issue just from different sides so
thinking fixing one will fix both.

** Affects: nova
Importance: Undecided
Status: New

--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1983623

Title:
Fail to attach interface after update to Victoria

Status in OpenStack Compute (nova):
New

Bug description:
Updated our cluster from Ussuri to Victoria and we are seeing issues
with instances that have an interface on one of our networks. Network
is named ext-net which happens to be the only shared + external net we
have setup.

Nodes are all running Centos 8.5. Cluster was setup with Kolla
Ansible

I can attach more logs before/after those lines if more info is
needed.

The controller and a handful of hypervisors have updated to Victoria.
We only see this error on instances that are running on hypervisors
running Victoria. I can attach an interface to the ext-net just fine
on instances running on hosts still using Ussuri.

Will also note I get errors when migrating an instance with an
interface on ext-net from a Ussuri to a Victoria hypervisor. The error
messages in that case is similar (PortBindingError), I can paste in
those errors also but expect they are the same issue just from
different sides so thinking fixing one will fix both.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1983623/+subscriptions

Follow ups

[Bug 1983623] Re: Fail to attach interface after update to Victoria
From: Launchpad Bug Tracker, 2022-10-22