yahoo-eng-team team mailing list archive

[OpenStack Infra <1852993@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/694756
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cff9ecb20870daa56b1cfd6fbb9f5817d1306fda
Submitter: Zuul
Branch:    master

commit cff9ecb20870daa56b1cfd6fbb9f5817d1306fda
Author: Pavel Glushchak <pglushchak@xxxxxxxxxxxxx>
Date:   Mon Nov 18 14:53:42 2019 +0300

    Don't delete compute node, when deleting service other than nova-compute
    
    We should not try to delete compute node from compute_nodes table,
    when destroying service other than nova-compute.
    
    Change-Id: If5b5945e699ec2e2da51d5fa90616431274849b0
    Closes-Bug: #1852993
    Signed-off-by: Pavel Glushchak <pglushchak@xxxxxxxxxxxxx>


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852993

Title:
  Don't delete compute node when deleting service other than nova-
  compute

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed
Status in OpenStack Compute (nova) train series:
  In Progress

Bug description:
  When upgrading to Stein, nova-consoleauth service is deprecated and
  should be removed. However if nova-consoleauth service is located on
  the same host with nova-compute, matching row in compute_nodes table
  is soft-deleted as well, making nova-compute service report in log,
  that stale resource provider exists in placement:

  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update(context, cn)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     inv_data,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(self, *a, **k)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain

  Steps to reproduce
  ==================

  # nova service-list
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | Id                                   | Binary           | Host                   | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:36.904174 | -               | False       |
  | 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:40.111038 | -               | False       |
  | 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up    | -                          | -               | False       |
  | 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute     | vzstor1.vstoragedomain | nova     | enabled | up    | 2019-11-18T12:33:43.500922 | -               | False       |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |

  # nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is
  nova-consoleauth service

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted

  Expected result
  ===============
  compute_node is not deleted from db if nova-consoleauth service is removed

  Actual result
  =============
  compute_node is deleted from db if nova-consoleauth service is removed

  Environment
  ===========
  Queens, Stein

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852993/+subscriptions