yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1809693] [NEW] service-delete failed, ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.ComputeHostNotFound'> (HTTP 500)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Charlotte Han <han.rong3@xxxxxxxxxx>
Date: Tue, 25 Dec 2018 03:35:38 -0000
Reply-to: Bug 1809693 <1809693@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Batch create compute nodes with docker, then delete some dockers because of resource limits. 
So some nova-computer services are down because docker is not exist.
Then delete these servers which is down, but some services can't be deleted.

nova service-delete 44dceff3-0924-4cac-b07f-2f28f23d8b8b

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.ComputeHostNotFound'> (HTTP 500) (Request-ID: req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82)

2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions [req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82 c8d6730b1b3c436f83f0ca96dcd141cb bda69a34aa6e49a1832e7d19d0c1073b - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host NailShelve2Slot10 could not be found.
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 242, in delete
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     context, service.compute_node, cascade=True)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 267, in obj_load_attr
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     self._context, self.host)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 468, in get_all_by_host
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     use_slave=use_slave)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 463, in _db_compute_node_get_all_by_host
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return db.compute_node_get_all_by_host(context, host)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 297, in compute_node_get_all_by_host
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return IMPL.compute_node_get_all_by_host(context, host)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 713, in compute_node_get_all_by_host
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     raise exception.ComputeHostNotFound(host=host)
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions ComputeHostNotFound: Compute host NailShelve2Slot10 could not be found.
2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions
2018-12-24 16:22:19.702 6864 INFO nova.api.openstack.wsgi [req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82 c8d6730b1b3c436f83f0ca96dcd141cb bda69a34aa6e49a1832e7d19d0c1073b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.ComputeHostNotFound'>


Steps to reproduce
==================
I have a simple reproducing method instead of batch creating and delete compute resource to simulate a nova-compute service which has no compute_nodes. 

* I did modify compute_driver from libvirt.LibvirtDriver to ironic.IronicDriver in nova.conf
* then I did restart nova-compute
* then I did command of 'nova service-delete'.

** Affects: nova
     Importance: Low
         Status: New

** Changed in: nova
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809693

Title:
  service-delete failed, ERROR (ClientException): Unexpected API Error.
  Please report this at http://bugs.launchpad.net/nova/ and attach the
  Nova API log if possible. <class 'nova.exception.ComputeHostNotFound'>
  (HTTP 500)

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Batch create compute nodes with docker, then delete some dockers because of resource limits. 
  So some nova-computer services are down because docker is not exist.
  Then delete these servers which is down, but some services can't be deleted.

  nova service-delete 44dceff3-0924-4cac-b07f-2f28f23d8b8b

  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.ComputeHostNotFound'> (HTTP 500) (Request-ID: req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82)

  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions [req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82 c8d6730b1b3c436f83f0ca96dcd141cb bda69a34aa6e49a1832e7d19d0c1073b - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host NailShelve2Slot10 could not be found.
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 242, in delete
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     context, service.compute_node, cascade=True)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 267, in obj_load_attr
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     self._context, self.host)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 468, in get_all_by_host
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 463, in _db_compute_node_get_all_by_host
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return db.compute_node_get_all_by_host(context, host)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 297, in compute_node_get_all_by_host
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return IMPL.compute_node_get_all_by_host(context, host)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 713, in compute_node_get_all_by_host
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions     raise exception.ComputeHostNotFound(host=host)
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions ComputeHostNotFound: Compute host NailShelve2Slot10 could not be found.
  2018-12-24 16:22:19.694 6864 ERROR nova.api.openstack.extensions
  2018-12-24 16:22:19.702 6864 INFO nova.api.openstack.wsgi [req-0506ae55-fa5f-4c7d-ac1d-1e14fb0dbb82 c8d6730b1b3c436f83f0ca96dcd141cb bda69a34aa6e49a1832e7d19d0c1073b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.ComputeHostNotFound'>


  Steps to reproduce
  ==================
  I have a simple reproducing method instead of batch creating and delete compute resource to simulate a nova-compute service which has no compute_nodes. 

  * I did modify compute_driver from libvirt.LibvirtDriver to ironic.IronicDriver in nova.conf
  * then I did restart nova-compute
  * then I did command of 'nova service-delete'.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809693/+subscriptions