yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #94644
[Bug 2082541] [NEW] Live migrating instance SR-IOV ports fails KeyError pci_slot
Public bug reported:
Intermittently seeing instance with SR-IOV ports attached fail live migration with KeyError pci_slot
After running the live migration command:
openstack server migrate --live-migration --os-compute-api 2.50 --host $TARGET_HOST $SERVER
The server went into error state with the below error, and SR-IOV ports
were no longer attached.
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/utils.py", line 1453, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 9524, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 9520, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.driver.post_live_migration_at_destination(
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 11606, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self._reattach_instance_vifs(context, instance, network_info)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 11158, in _reattach_instance_vifs
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.attach_interface(context, instance,
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2976, in attach_interface
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server cfg = self.vif_driver.get_config(instance, vif, image_meta,
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py", line 597, in get_config
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return self.get_config_hw_veb(*args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py", line 364, in get_config_hw_veb
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server conf, net_type, profile['pci_slot'],
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
We are running 2024.1 deployed via OpenStack-Ansible
On KVM with linuxbridge.
/openstack/venvs/nova-29.0.1/bin/pip show nova
Name: nova
Version: 29.1.0
Summary: Cloud computing fabric controller
Home-page: https://docs.openstack.org/nova/latest/
Author: OpenStack
Author-email: openstack-discuss@xxxxxxxxxxxxxxxxxxx
License: UNKNOWN
Location: /openstack/venvs/nova-29.0.1/lib/python3.10/site-packages
Requires: alembic, castellan, cryptography, cursive, decorator, eventlet, futurist, greenlet, iso8601, Jinja2, jsonschema, keystoneauth1, keystonemiddleware, lxml, microversion-parse, netaddr, netifaces, openstacksdk, os-brick, os-resource-classes, os-service-types, os-traits, os-vif, oslo.cache, oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.limit, oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils, oslo.versionedobjects, paramiko, Paste, PasteDeploy, pbr, PrettyTable, psutil, python-cinderclient, python-dateutil, python-glanceclient, python-neutronclient, PyYAML, requests, retrying, rfc3986, Routes, SQLAlchemy, stevedore, tooz, WebOb, websockify
Required-by:
/openstack/venvs/neutron-29.0.1/bin/pip show neutron
Name: neutron
Version: 24.0.1.dev22
Summary: OpenStack Networking
Home-page: https://docs.openstack.org/neutron/latest/
Author: OpenStack
Author-email: openstack-discuss@xxxxxxxxxxxxxxxxxxx
License:
Location: /openstack/venvs/neutron-29.0.1/lib/python3.10/site-packages
Requires: alembic, debtcollector, decorator, eventlet, futurist, httplib2, Jinja2, keystoneauth1, keystonemiddleware, netaddr, netifaces, neutron-lib, openstacksdk, os-ken, os-resource-classes, os-vif, oslo.cache, oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils, oslo.versionedobjects, osprofiler, ovs, ovsdbapp, Paste, PasteDeploy, pbr, pecan, psutil, pyOpenSSL, pyroute2, python-designateclient, python-neutronclient, python-novaclient, requests, Routes, SQLAlchemy, stevedore, tenacity, tooz, WebOb
Required-by:
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2082541
Title:
Live migrating instance SR-IOV ports fails KeyError pci_slot
Status in OpenStack Compute (nova):
New
Bug description:
Intermittently seeing instance with SR-IOV ports attached fail live migration with KeyError pci_slot
After running the live migration command:
openstack server migrate --live-migration --os-compute-api 2.50 --host $TARGET_HOST $SERVER
The server went into error state with the below error, and SR-IOV
ports were no longer attached.
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/utils.py", line 1453, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 9524, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py", line 9520, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.driver.post_live_migration_at_destination(
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 11606, in post_live_migration_at_destination
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self._reattach_instance_vifs(context, instance, network_info)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 11158, in _reattach_instance_vifs
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.attach_interface(context, instance,
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2976, in attach_interface
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server cfg = self.vif_driver.get_config(instance, vif, image_meta,
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py", line 597, in get_config
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return self.get_config_hw_veb(*args)
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py", line 364, in get_config_hw_veb
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server conf, net_type, profile['pci_slot'],
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
2024-09-26 16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
We are running 2024.1 deployed via OpenStack-Ansible
On KVM with linuxbridge.
/openstack/venvs/nova-29.0.1/bin/pip show nova
Name: nova
Version: 29.1.0
Summary: Cloud computing fabric controller
Home-page: https://docs.openstack.org/nova/latest/
Author: OpenStack
Author-email: openstack-discuss@xxxxxxxxxxxxxxxxxxx
License: UNKNOWN
Location: /openstack/venvs/nova-29.0.1/lib/python3.10/site-packages
Requires: alembic, castellan, cryptography, cursive, decorator, eventlet, futurist, greenlet, iso8601, Jinja2, jsonschema, keystoneauth1, keystonemiddleware, lxml, microversion-parse, netaddr, netifaces, openstacksdk, os-brick, os-resource-classes, os-service-types, os-traits, os-vif, oslo.cache, oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.limit, oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils, oslo.versionedobjects, paramiko, Paste, PasteDeploy, pbr, PrettyTable, psutil, python-cinderclient, python-dateutil, python-glanceclient, python-neutronclient, PyYAML, requests, retrying, rfc3986, Routes, SQLAlchemy, stevedore, tooz, WebOb, websockify
Required-by:
/openstack/venvs/neutron-29.0.1/bin/pip show neutron
Name: neutron
Version: 24.0.1.dev22
Summary: OpenStack Networking
Home-page: https://docs.openstack.org/neutron/latest/
Author: OpenStack
Author-email: openstack-discuss@xxxxxxxxxxxxxxxxxxx
License:
Location: /openstack/venvs/neutron-29.0.1/lib/python3.10/site-packages
Requires: alembic, debtcollector, decorator, eventlet, futurist, httplib2, Jinja2, keystoneauth1, keystonemiddleware, netaddr, netifaces, neutron-lib, openstacksdk, os-ken, os-resource-classes, os-vif, oslo.cache, oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils, oslo.versionedobjects, osprofiler, ovs, ovsdbapp, Paste, PasteDeploy, pbr, pecan, psutil, pyOpenSSL, pyroute2, python-designateclient, python-neutronclient, python-novaclient, requests, Routes, SQLAlchemy, stevedore, tenacity, tooz, WebOb
Required-by:
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2082541/+subscriptions
