yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88308
[Bug 1960465] Re: nova-compute Timed out waiting for a reply to message ID
Based on your solution. I think the RabbitMQ was the root cause of the
problem. From the included nova logs I only see that nova cannot
communicate via Rabbit.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1960465
Title:
nova-compute Timed out waiting for a reply to message ID
Status in OpenStack Compute (nova):
Invalid
Bug description:
I am running wallaby release in production and today when my rabbitMQ
went down so i have rebuild whole rabbitMQ cluster but after that all
my compute nodes nova-compute service started throwing errors like
following. Not sure what is wrong here even i re-build rabbitMQ. any
idea how to fix?
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager [req-092f8eb6-1bcc-4545-979a-a8e0eb28da65 - - - - -] Error updating resources for node ostack-phx-comp-sriov-1-1.v1v0x.net.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return waiter.wait()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return get_hub().switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return self.greenlet.switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager _queue.Empty
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py", line 9934, in _update_available_resource_for_node
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py", line 895, in update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return f(*args, **kwargs)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py", line 975, in _update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py", line 1230, in _update
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager self.pci_tracker.save(context)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/pci/manager.py", line 95, in save
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager dev.save()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager updates, result = self.indirection_api.object_action(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 247, in object_action
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return cctxt.call(context, 'object_action', objinst=objinst,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 175, in call
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return self._driver.send(target, ctxt, message,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 669, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager result = self._waiter.wait(msg_id, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager message = self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager raise oslo_messaging.MessagingTimeout(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
Feb 10 02:44:47 ostack-phx-comp-sriov-1-1.v1v0x.net nova-compute[827533]: 2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task [req-b111abf5-19d8-4424-9934-f368ad55216b - - - - -] Error during ComputeManager._run_image_cache_manager_pass: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return waiter.wait()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return get_hub().switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return self.greenlet.switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task _queue.Empty
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred:
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task task(self, context)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py", line 10481, in _run_image_cache_manager_pass
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task self.driver.manage_image_cache(context, filtered_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10649, in manage_image_cache
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task self.image_cache_manager.update(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/imagecache.py", line 381, in update
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task running = self._list_running_instances(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/imagecache.py", line 56, in _list_running_instances
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task instance_bdms = objects.BlockDeviceMappingList.bdms_by_instance_uuid(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/objects/block_device.py", line 370, in bdms_by_instance_uuid
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task bdms = cls.get_by_instance_uuids(context, instance_uuids)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result = cls.indirection_api.object_class_action_versions(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return cctxt.call(context, 'object_class_action_versions',
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 175, in call
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return self._driver.send(target, ctxt, message,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 669, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task raise oslo_messaging.MessagingTimeout(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1960465/+subscriptions
References
