yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1860312] Re: compute service failed to delete

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: sean mooney <1860312@xxxxxxxxxxxxxxxxxx>
Date: Wed, 28 Jul 2021 12:02:10 -0000
Reply-to: Bug 1860312 <1860312@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
actully the operator woudl be deleting the comptue service after removing the compute nodes.
you shoudl remove the compute service first but we shoudl fix this regardless.

you should be able to recreate this bug by just creating a compute servce
and then deleteing it.

** Changed in: nova
       Status: Expired => Triaged

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860312

Title:
  compute service failed to delete

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Description
  ===========
  I deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic replica of the deployment is 1) breakdown.It may be scheduled to another node by kubernetes.When I try to delete the old compute service(status down), it failed.

  Steps to reproduce
  ==================
  Firstly, openstack was deployed in kubernetes cluster, and the replica of the nova-compute-ironic is 1.
  * I deleted the pod nova-compute-ironic-xxxxx
  * then wait for the new pod to start
  * then exec openstack compute service list, there will be two compute service for ironic, the status of the old one would be down.
  * then I try to delete the old compute service

  Expected result
  ===============
  the old compute service could be deleted successfully

  Actual result
  =============
  failed to delete, and returned an http 500

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     18.2.2, rocky

  2. Which hypervisor did you use?
     Libvirt + KVM

  2. Which storage type did you use?
     ceph

  3. Which networking type did you use?
     Neutron with OpenVSwitch

  Logs & Configs
  ==============
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi 
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860312/+subscriptions
References

[Bug 1860312] [NEW] compute service failed to delete
From: tanghang, 2020-01-20