yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2060931] [NEW] Intermittent migration failures after upgrade to 2023.2

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Bonney <2060931@xxxxxxxxxxxxxxxxxx>
Date: Thu, 11 Apr 2024 09:05:50 -0000
Reply-to: Bug 2060931 <2060931@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Following an upgrade from 2023.1 to 2023.2 we are seeing intermittent issues with requests to perform live migrations (and potentially cold migrations). The failed requests result in the impacted instances going into an error state which appears similar to https://bugs.launchpad.net/nova/+bug/2044235, but the fact that a messaging timeout is occurring at all seems to be a new behaviour post-upgrade so I thought it would be worth recording it independently.

After a failure to migrate to a given destination host I believe
subsequent requests to migrate to that destination will then succeed, at
least until a reasonable time has then expired.

Whether this is a Nova issue directly or an issue in an underlying
library I can't be sure.

Steps to reproduce
==================
* Deploy Nova
* Wait for a reasonable period of time. I can't quantify this, but it doesn't seem to happen immediately after Nova services have been restarted.
* Attempt live migration of a number of instances

Expected result
===============
Live migration should proceed as expected

Actual result
=============
Live migration fails for some cases with a 500 error returned to the user via the API (POST /v2.1/servers/<uuid>/action). The instance which migration was attempted for enters the ERROR state.

The post-error message from nova-compute may be a clue, but I'm not
familiar enough to know:

2024-04-11 08:22:52.434 3774901 INFO oslo_messaging._drivers.amqpdriver
[-] No calling threads waiting for msg_id :
1fb21ba089eb47959ac55a29a155a675

Environment
===========
Nova 2023.2 45a926156c863b468318cce462a21027685d07a6 with oslo.messaging 14.4.1

This didn't happen with Nova 2023.1
252e6602f792f24141a9dd59211470601f3b0267 and oslo.messaging 14.2.1

Libvirt + KVM
Ceph
Neutron + LXB

Logs & Configs
==============

Nova API:

2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._queues[msg_id].get(block=True, timeout=timeout)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.10/queue.py", line 179, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise Empty
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi _queue.Empty
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   [Previous line repeated 1 more time]
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/compute/migrate_server.py", line 126, in _migrate_live
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.compute_api.live_migrate(context, instance, block_migration,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 389, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 374, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 310, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 357, in wrapper
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 279, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 260, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 242, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 168, in inner
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5578, in live_migrate
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     with excutils.save_and_reraise_exception():
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.force_reraise()
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise self.value
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5573, in live_migrate
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.compute_task_api.live_migrate_instance(context, instance,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/api.py", line 110, in live_migrate_instance
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.conductor_compute_rpcapi.migrate_server(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 352, in migrate_server
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return cctxt.call(context, 'migrate_server', **kw)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     result = self.transport._send(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._driver.send(target, ctxt, message,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout,
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise oslo_messaging.MessagingTimeout(
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6d20c3260d104854a3c676026ea5abf9
2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 


Nova Compute (believed to be destination host for migration):

2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._queues[msg_id].get(block=True, timeout=timeout)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return waiter.wait()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return get_hub().switch()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server _queue.Empty
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise self.value
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     if not instance.vcpu_model or not instance.vcpu_model.model:
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.obj_load_attr(name)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return fn(self, attrname)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self._obj_load_attr(attrname)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self._load_vcpu_model()
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = cls.indirection_api.object_class_action_versions(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return cctxt.call(context, 'object_class_action_versions',
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = self.transport._send(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._driver.send(target, ctxt, message,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = self._waiter.wait(msg_id, timeout,
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     message = self.waiters.get(msg_id, timeout=timeout)
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise oslo_messaging.MessagingTimeout(
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
2024-04-11 08:22:52.434 3774901 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 1fb21ba089eb47959ac55a29a155a675


Nova Conductor:

2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager Traceback (most recent call last):
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/manager.py", line 505, in _live_migrate
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     task.execute()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 25, in wrap
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     with excutils.save_and_reraise_exception():
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self.force_reraise()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     raise self.value
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return original(self)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 40, in execute
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._execute()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 100, in _execute
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self.destination, dest_node, self.limits = self._find_destination()
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self._call_livem_checks_on_host(host, provider_mapping)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 360, in _call_livem_checks_on_host
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     check_can_live_migrate_destination(self.context, self.instance,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/rpcapi.py", line 604, in check_can_live_migrate_destination
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'check_can_live_migrate_destination', **kwargs)
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     result = self.transport._send(
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._driver.send(target, ctxt, message,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     raise result
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager ['Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n    return self._queues[msg_id].get(block=True, timeout=timeout)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n    return waiter.wait()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n    return get_hub().switch()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n    return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped\n    with excutils.save_and_reraise_exception():\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function\n    with excutils.save_and_reraise_exception():\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination\n    dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination\n    if not instance.vcpu_model or not instance.vcpu_model.model:\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter\n    self.obj_load_attr(name)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper\n    return fn(self, attrname)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr\n    self._obj_load_attr(attrname)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr\n    self._load_vcpu_model()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model\n    self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper\n    result = cls.indirection_api.object_class_action_versions(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions\n    return cctxt.call(context, \'object_class_action_versions\',\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call\n    result = self.transport._send(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n    return self._driver.send(target, ctxt, message,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n    result = self._waiter.wait(msg_id, timeout,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n    message = self.waiters.get(msg_id, timeout=timeout)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n    raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675\n'].
2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2060931

Title:
  Intermittent migration failures after upgrade to 2023.2

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Following an upgrade from 2023.1 to 2023.2 we are seeing intermittent issues with requests to perform live migrations (and potentially cold migrations). The failed requests result in the impacted instances going into an error state which appears similar to https://bugs.launchpad.net/nova/+bug/2044235, but the fact that a messaging timeout is occurring at all seems to be a new behaviour post-upgrade so I thought it would be worth recording it independently.

  After a failure to migrate to a given destination host I believe
  subsequent requests to migrate to that destination will then succeed,
  at least until a reasonable time has then expired.

  Whether this is a Nova issue directly or an issue in an underlying
  library I can't be sure.

  Steps to reproduce
  ==================
  * Deploy Nova
  * Wait for a reasonable period of time. I can't quantify this, but it doesn't seem to happen immediately after Nova services have been restarted.
  * Attempt live migration of a number of instances

  Expected result
  ===============
  Live migration should proceed as expected

  Actual result
  =============
  Live migration fails for some cases with a 500 error returned to the user via the API (POST /v2.1/servers/<uuid>/action). The instance which migration was attempted for enters the ERROR state.

  The post-error message from nova-compute may be a clue, but I'm not
  familiar enough to know:

  2024-04-11 08:22:52.434 3774901 INFO
  oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for
  msg_id : 1fb21ba089eb47959ac55a29a155a675

  Environment
  ===========
  Nova 2023.2 45a926156c863b468318cce462a21027685d07a6 with oslo.messaging 14.4.1

  This didn't happen with Nova 2023.1
  252e6602f792f24141a9dd59211470601f3b0267 and oslo.messaging 14.2.1

  Libvirt + KVM
  Ceph
  Neutron + LXB

  Logs & Configs
  ==============

  Nova API:

  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._queues[msg_id].get(block=True, timeout=timeout)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.10/queue.py", line 179, in get
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise Empty
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi _queue.Empty
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   [Previous line repeated 1 more time]
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/api/openstack/compute/migrate_server.py", line 126, in _migrate_live
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.compute_api.live_migrate(context, instance, block_migration,
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 389, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 374, in wrapper
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(self, context, instance, *args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 310, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 357, in wrapper
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return func(self, context, instance, *args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 279, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 260, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 242, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 168, in inner
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5578, in live_migrate
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     with excutils.save_and_reraise_exception():
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.force_reraise()
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise self.value
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/api.py", line 5573, in live_migrate
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.compute_task_api.live_migrate_instance(context, instance,
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/api.py", line 110, in live_migrate_instance
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     self.conductor_compute_rpcapi.migrate_server(
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 352, in migrate_server
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return cctxt.call(context, 'migrate_server', **kw)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     result = self.transport._send(
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._driver.send(target, ctxt, message,
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout,
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi     raise oslo_messaging.MessagingTimeout(
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6d20c3260d104854a3c676026ea5abf9
  2024-04-11 08:22:46.061 122354 ERROR nova.api.openstack.wsgi 

  
  Nova Compute (believed to be destination host for migration):

  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._queues[msg_id].get(block=True, timeout=timeout)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return waiter.wait()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return get_hub().switch()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server _queue.Empty
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise self.value
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     if not instance.vcpu_model or not instance.vcpu_model.model:
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.obj_load_attr(name)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return fn(self, attrname)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self._obj_load_attr(attrname)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self._load_vcpu_model()
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = cls.indirection_api.object_class_action_versions(
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return cctxt.call(context, 'object_class_action_versions',
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = self.transport._send(
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._driver.send(target, ctxt, message,
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     result = self._waiter.wait(msg_id, timeout,
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     message = self.waiters.get(msg_id, timeout=timeout)
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server     raise oslo_messaging.MessagingTimeout(
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
  2024-04-11 08:22:52.298 3774901 ERROR oslo_messaging.rpc.server 
  2024-04-11 08:22:52.434 3774901 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 1fb21ba089eb47959ac55a29a155a675

  
  Nova Conductor:

  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager Traceback (most recent call last):
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/manager.py", line 505, in _live_migrate
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     task.execute()
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 25, in wrap
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     with excutils.save_and_reraise_exception():
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self.force_reraise()
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     raise self.value
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return original(self)
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/base.py", line 40, in execute
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._execute()
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 100, in _execute
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self.destination, dest_node, self.limits = self._find_destination()
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     self._call_livem_checks_on_host(host, provider_mapping)
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/tasks/live_migrate.py", line 360, in _call_livem_checks_on_host
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     check_can_live_migrate_destination(self.context, self.instance,
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/rpcapi.py", line 604, in check_can_live_migrate_destination
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'check_can_live_migrate_destination', **kwargs)
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     result = self.transport._send(
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._driver.send(target, ctxt, message,
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager   File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager     raise result
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager ['Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n    return self._queues[msg_id].get(block=True, timeout=timeout)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n    return waiter.wait()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n    return get_hub().switch()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n    return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped\n    with excutils.save_and_reraise_exception():\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 213, in decorated_function\n    with excutils.save_and_reraise_exception():\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/compute/manager.py", line 8520, in check_can_live_migrate_destination\n    dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9664, in check_can_live_migrate_destination\n    if not instance.vcpu_model or not instance.vcpu_model.model:\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 67, in getter\n    self.obj_load_attr(name)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/base.py", line 120, in wrapper\n    return fn(self, attrname)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1159, in obj_load_attr\n    self._obj_load_attr(attrname)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1187, in _obj_load_attr\n    self._load_vcpu_model()\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/objects/instance.py", line 1000, in _load_vcpu_model\n    self.vcpu_model = objects.VirtCPUModel.get_by_instance_uuid(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper\n    result = cls.indirection_api.object_class_action_versions(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions\n    return cctxt.call(context, \'object_class_action_versions\',\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call\n    result = self.transport._send(\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n    return self._driver.send(target, ctxt, message,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n    result = self._waiter.wait(msg_id, timeout,\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n    message = self.waiters.get(msg_id, timeout=timeout)\n', '  File "/openstack/venvs/nova-28.1.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n    raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 1fb21ba089eb47959ac55a29a155a675\n'].
  2024-04-11 08:22:52.768 122305 ERROR nova.conductor.manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2060931/+subscriptions