yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1904181] Re: nova-compute fails to start if cell conductor is not running.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Elod Illes <1904181@xxxxxxxxxxxxxxxxxx>
Date: Fri, 05 Feb 2021 12:38:47 -0000
Reply-to: Bug 1904181 <1904181@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/victoria
   Importance: Undecided
       Status: New

** Changed in: nova/victoria
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1904181

Title:
  nova-compute fails to start if cell conductor is not running.

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) victoria series:
  Fix Released

Bug description:
  Since https://review.opendev.org/#/c/738482/ if the nova-compute is
  started earlier than the cell conductor the compute belongs to then
  the nova-compute service will fail with a MessagingTimeout and exists.
  Before https://review.opendev.org/#/c/738482/ in this scenario the
  nova compute would retry until the conductor is ready.

  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: CRITICAL nova [None req-a8777aaa-2c27-4770-83d5-63d869124590 None None] Unhandled error: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 5f9597ea9baf4b6fb37ba85d81040b62
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova Traceback (most recent call last):
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 405, in get
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return self._queues[msg_id].get(block=True, timeout=timeout)
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/eventlet/queue.py", line 322, in get
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return waiter.wait()
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/eventlet/queue.py", line 141, in wait
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return get_hub().switch()
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 313, in switch
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return self.greenlet.switch()
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova _queue.Empty
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova 
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova During handling of the above exception, another exception occurred:
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova 
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova Traceback (most recent call last):
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/bin/nova-compute", line 10, in <module>
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     sys.exit(main())
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/opt/stack/nova/nova/cmd/compute.py", line 58, in main
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     server = service.Service.create(binary='nova-compute',
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/opt/stack/nova/nova/service.py", line 252, in create
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     utils.raise_if_old_compute()
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/opt/stack/nova/nova/utils.py", line 1088, in raise_if_old_compute
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     current_service_version = service.Service.get_minimum_version(
  Nov 12 12:59:12.962343 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 175, in wrapper
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     result = cls.indirection_api.object_class_action_versions(
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return cctxt.call(context, 'object_class_action_versions',
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 175, in call
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     self.transport._send(self.target, msg_ctxt, msg,
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return self._driver.send(target, ctxt, message,
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 652, in send
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     return self._send(target, ctxt, message, wait_for_reply, timeout,
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 641, in _send
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     result = self._waiter.wait(msg_id, timeout,
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 531, in wait
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     message = self.waiters.get(msg_id, timeout=timeout)
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 407, in get
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova     raise oslo_messaging.MessagingTimeout(
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 5f9597ea9baf4b6fb37ba85d81040b62
  Nov 12 12:59:12.965655 ubuntu-focal-ovh-bhs1-0021738724 nova-compute[81122]: ERROR nova 
  Nov 12 12:59:13.407277 ubuntu-focal-ovh-bhs1-0021738724 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE
  Nov 12 12:59:13.407306 ubuntu-focal-ovh-bhs1-0021738724 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'.

  Example run:
  https://1d1ac8c4d7a38514d020-4dcc482f43713c13ecd75f64a0eb3df3.ssl.cf1.rackcdn.com/762319/1/check
  /tempest-integrated-compute/0b7fa72/controller/logs/screen-n-cond-
  cell1.txt

  Signature:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22current_service_version%20%3D%20service.Service.get_minimum_version(%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1904181/+subscriptions
References

[Bug 1904181] [NEW] nova-compute fails to start if cell conductor is not running.
From: Balazs Gibizer, 2020-11-13