← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95441

[Bug 2100888] [NEW] Resizing instance with multiple SR-IOV ports plus a normal port fails

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description:
============
We have instances with multiple SR-IOV ports for dataplane plus a normal type port for management.

When attempting to resize these instances they fail with Exception
during message handling: nova.exception.PortUpdateFailed: Port update
failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to
correlate PCI slot 0000:02:02.0.

Steps to reproduce:
===================
Create an instance with one normal vnic type port.
Create 2 or more direct vnic type ports. 
Attach the direct vnic type ports to the instance.
Attempt to resize the instance.

The resize fails with the below error.
 nova-compute[4015884]: 2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server [None req-a76660db-8434-4cfb-9c08-a97a70b38052 a104d813c07cc2da2de506ef3a996c3ef63793a043213f4d1c9a968a0603f142 74928e73d73f448f8a4ec1c74195f585 - - 6bb0ca268c3946539b152f02910baf8c default] Exception during message handling: nova.exception.PortUpdateFailed: Port update failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to correlate PCI slot 0000:02:02.0
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6348, in finish_resize
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6345, in finish_resize
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self._finish_resize_helper(context, disk_info, image, instance,
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6378, in _finish_resize_helper
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server network_info = self._finish_resize(context, instance, migration,
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6263, in _finish_resize
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.network_api.migrate_instance_finish(
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/network/neutron.py", line 3219, in migrate_instance_finish
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self._update_port_binding_for_instance(
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/network/neutron.py", line 3793, in _update_port_binding_for_instance
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise exception.PortUpdateFailed(port_id=p['id'],
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server nova.exception.PortUpdateFailed: Port update failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to correlate PCI slot 0000:02:02.0
                                                            2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server


Environment:
============

Running 2024.1 Nova 29.1.0.
Deployed with openstack-ansible 2024.1 
Running neutron networking with linuxbridge-agent 
KVM hypervisor
Ceph RBD storage

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2100888

Title:
  Resizing instance with multiple SR-IOV ports plus a normal port fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:
  ============
  We have instances with multiple SR-IOV ports for dataplane plus a normal type port for management.

  When attempting to resize these instances they fail with Exception
  during message handling: nova.exception.PortUpdateFailed: Port update
  failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to
  correlate PCI slot 0000:02:02.0.

  Steps to reproduce:
  ===================
  Create an instance with one normal vnic type port.
  Create 2 or more direct vnic type ports. 
  Attach the direct vnic type ports to the instance.
  Attempt to resize the instance.

  The resize fails with the below error.
   nova-compute[4015884]: 2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server [None req-a76660db-8434-4cfb-9c08-a97a70b38052 a104d813c07cc2da2de506ef3a996c3ef63793a043213f4d1c9a968a0603f142 74928e73d73f448f8a4ec1c74195f585 - - 6bb0ca268c3946539b152f02910baf8c default] Exception during message handling: nova.exception.PortUpdateFailed: Port update failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to correlate PCI slot 0000:02:02.0
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6348, in finish_resize
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.force_reraise()
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise self.value
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6345, in finish_resize
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self._finish_resize_helper(context, disk_info, image, instance,
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6378, in _finish_resize_helper
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server network_info = self._finish_resize(context, instance, migration,
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 6263, in _finish_resize
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self.network_api.migrate_instance_finish(
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/network/neutron.py", line 3219, in migrate_instance_finish
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server self._update_port_binding_for_instance(
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/network/neutron.py", line 3793, in _update_port_binding_for_instance
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server raise exception.PortUpdateFailed(port_id=p['id'],
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server nova.exception.PortUpdateFailed: Port update failed for port 04686da2-b2a3-43a0-9cc0-4afbd4261c03: Unable to correlate PCI slot 0000:02:02.0
                                                              2025-02-12 23:05:20.783 4015884 ERROR oslo_messaging.rpc.server

  
  Environment:
  ============

  Running 2024.1 Nova 29.1.0.
  Deployed with openstack-ansible 2024.1 
  Running neutron networking with linuxbridge-agent 
  KVM hypervisor
  Ceph RBD storage

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2100888/+subscriptions
  Thread Previous • Date Previous • • Thread Next