yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1868203] [NEW] nova-compute error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: YG Kumar <ygk.kmr@xxxxxxxxx>
Date: Fri, 20 Mar 2020 05:42:53 -0000
Reply-to: Bug 1868203 <1868203@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi All,

I have a rocky setup. I am unable to launch instances on one of the
compute nodes.

I see the following error in the conductor logs and the libvirtd service
on the compute. The host is ubuntu 18.04:


------------------

  File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
    raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 1237, in schedule_and_build_instances
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 750, in _schedule_instances
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return_alternates=return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 50, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     retry=self.retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     retry=retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     raise result
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return func(*args, **kwargs)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     raise exception.NoValidHost(reason="")
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.390 35363 WARNING nova.scheduler.utils [req-cd25c8b1-44a2-40d7-9f0f-0d81bf5c3c08 2db30fa813074deb80e54e76529e2e71 aa73777f89414369924a0794b3eaf99e - default default] Failed to compute_task_build_instances: No valid host was found.
-------------------------------- 


This is the error on the nova-compute service on that compute node:
------------- 
2020-03-20 01:44:24.158 6193 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID ea3af0ff-fb69-449a-b484-5ac722010a1f
------------------


This is the status of the libvirtd service on that compute node:


------------------------
Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
Mar 19 04:32:14 b2a-1 libvirtd[5732]: Failed to open file '/sys/class/net/tap529b886c-d1/operstate': No such file or directory
Mar 19 04:32:14 b2a-1 libvirtd[5732]: unable to read: /sys/class/net/tap529b886c-d1/operstate: No such file or directory
-------------------------------------

Please help me out with this issue.


Thanks
Kumar

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1868203

Title:
  nova-compute error

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi All,

  I have a rocky setup. I am unable to launch instances on one of the
  compute nodes.

  I see the following error in the conductor logs and the libvirtd
  service on the compute. The host is ubuntu 18.04:

  
  ------------------

    File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
      raise exception.NoValidHost(reason="")

  NoValidHost: No valid host was found.
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 1237, in schedule_and_build_instances
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 750, in _schedule_instances
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return_alternates=return_alternates)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 50, in select_destinations
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     retry=self.retry)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     retry=retry)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     raise result
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     return func(*args, **kwargs)
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager     raise exception.NoValidHost(reason="")
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost: No valid host was found.
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
  2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
  2020-03-20 05:31:09.390 35363 WARNING nova.scheduler.utils [req-cd25c8b1-44a2-40d7-9f0f-0d81bf5c3c08 2db30fa813074deb80e54e76529e2e71 aa73777f89414369924a0794b3eaf99e - default default] Failed to compute_task_build_instances: No valid host was found.
  -------------------------------- 


  
  This is the error on the nova-compute service on that compute node:
  ------------- 
  2020-03-20 01:44:24.158 6193 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID ea3af0ff-fb69-449a-b484-5ac722010a1f
  ------------------


  This is the status of the libvirtd service on that compute node:

  
  ------------------------
  Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
  Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
  Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
  Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
  Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
  Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
  Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
  Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
  Mar 19 04:32:14 b2a-1 libvirtd[5732]: Failed to open file '/sys/class/net/tap529b886c-d1/operstate': No such file or directory
  Mar 19 04:32:14 b2a-1 libvirtd[5732]: unable to read: /sys/class/net/tap529b886c-d1/operstate: No such file or directory
  -------------------------------------

  Please help me out with this issue.

  
  Thanks
  Kumar

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1868203/+subscriptions
Follow ups

[Bug 1868203] Re: nova-compute error
From: Kashyap Chamarthy, 2020-03-20