yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1860312] Re: compute service failed to delete

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1860312@xxxxxxxxxxxxxxxxxx>
Date: Sat, 07 Aug 2021 19:39:07 -0000
Reply-to: Bug 1860312 <1860312@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/nova/+/801285
Committed: https://opendev.org/openstack/nova/commit/880611df0b6b967adabd3f08886e385d0a100c5c
Submitter: "Zuul (22348)"
Branch:    master

commit 880611df0b6b967adabd3f08886e385d0a100c5c
Author: Artom Lifshitz <alifshit@xxxxxxxxxx>
Date:   Mon Jul 19 12:58:29 2021 +0200

    Allow deletion of compute service with no compute nodes
    
    Consider the following situation:
    
    - Using the Ironic virt driver
    - Replacing (so removing and re-adding) all baremetal nodes
      associated with a single nova-compute service
    
    The update resources periodic will have destroyed the compute node
    records because they're no longer being reported by the virt driver.
    If we then attempt to manually delete the compute service record, the
    datbase layer will raise an exception, as there are no longer any
    compute node records for the host. Previously, this exception would
    get bubbled up as an error 500 in the API. This patch catches it and
    allows service deletion to complete succefully.
    
    Closes bug: 1860312
    Change-Id: I2f9ad3df25306e070c8c3538bfed1212d6d8682f


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860312

Title:
  compute service failed to delete

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  I deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic replica of the deployment is 1) breakdown.It may be scheduled to another node by kubernetes.When I try to delete the old compute service(status down), it failed.

  Steps to reproduce
  ==================
  Firstly, openstack was deployed in kubernetes cluster, and the replica of the nova-compute-ironic is 1.
  * I deleted the pod nova-compute-ironic-xxxxx
  * then wait for the new pod to start
  * then exec openstack compute service list, there will be two compute service for ironic, the status of the old one would be down.
  * then I try to delete the old compute service

  Expected result
  ===============
  the old compute service could be deleted successfully

  Actual result
  =============
  failed to delete, and returned an http 500

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     18.2.2, rocky

  2. Which hypervisor did you use?
     Libvirt + KVM

  2. Which storage type did you use?
     ceph

  3. Which networking type did you use?
     Neutron with OpenVSwitch

  Logs & Configs
  ==============
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi 
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi [req-922cc601-9aa1-4c3d-ad9c-71f73a341c28 40e7b8c3d59943e08a52acd24fe30652 d13f1690c08d41ac854d720ea510a710 - default default] Unexpected exception in API method: ComputeHostNotFound: Compute host mgt-slave03 could not be found.
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 252, in delete
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     context, service.host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 443, in get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 213, in wrapper
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/objects/compute_node.py", line 438, in _db_compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return db.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/api.py", line 291, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return IMPL.compute_node_get_all_by_host(context, host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 258, in wrapped
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 659, in compute_node_get_all_by_host
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi     raise exception.ComputeHostNotFound(host=host)
  2020-01-20 06:44:53.480 1 ERROR nova.api.openstack.wsgi ComputeHostNotFound: Compute host mgt-slave03 could not be found

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860312/+subscriptions
References

[Bug 1860312] [NEW] compute service failed to delete
From: tanghang, 2020-01-20