yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1860312] [NEW] compute service failed to delete

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: tanghang <1860312@xxxxxxxxxxxxxxxxxx>
Date: Mon, 20 Jan 2020 08:02:24 -0000
Reply-to: Bug 1860312 <1860312@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
I deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic replica of the deployment is 1) breakdown.It may be scheduled to another node by kubernetes.When I try to delete the old compute service(status down), it failed.

Steps to reproduce
==================
Firstly, openstack was deployed in kubernetes cluster, and the replica of the nova-compute-ironic is 1.
* I deleted the pod nova-compute-ironic-xxxxx
* then wait for the new pod to start
* then exec openstack compute service list, there will be two compute service for ironic, the status of the old one would be down.
* then I try to delete the old compute service

Expected result
===============
the old compute service could be deleted successfully

Actual result
=============
failed to delete, and returned an http 500

Environment
===========
1. Exact version of OpenStack you are running. See the following
   18.2.2, rocky

2. Which hypervisor did you use?
   Libvirt + KVM

2. Which storage type did you use?
   ceph

3. Which networking type did you use?
   Neutron with OpenVSwitch

Logs & Configs
==============
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found.
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi 
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860312

Title:
  compute service failed to delete

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic replica of the deployment is 1) breakdown.It may be scheduled to another node by kubernetes.When I try to delete the old compute service(status down), it failed.

  Steps to reproduce
  ==================
  Firstly, openstack was deployed in kubernetes cluster, and the replica of the nova-compute-ironic is 1.
  * I deleted the pod nova-compute-ironic-xxxxx
  * then wait for the new pod to start
  * then exec openstack compute service list, there will be two compute service for ironic, the status of the old one would be down.
  * then I try to delete the old compute service

  Expected result
  ===============
  the old compute service could be deleted successfully

  Actual result
  =============
  failed to delete, and returned an http 500

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     18.2.2, rocky

  2. Which hypervisor did you use?
     Libvirt + KVM

  2. Which storage type did you use?
     ceph

  3. Which networking type did you use?
     Neutron with OpenVSwitch

  Logs & Configs
  ==============
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi 
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860312/+subscriptions
Follow ups

[Bug 1860312] Re: compute service failed to delete
From: OpenStack Infra, 2021-08-07
[Bug 1860312] Re: compute service failed to delete
From: sean mooney, 2021-07-28
[Bug 1860312] Re: compute service failed to delete
From: Launchpad Bug Tracker, 2020-06-21