yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1852993] [NEW] Don't delete compute node when deleting service other than nova-compute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pavel Gluschak <trustytwelve@xxxxxxxxx>
Date: Mon, 18 Nov 2019 12:13:35 -0000
Reply-to: Bug 1852993 <1852993@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When upgrading to Stein, nova-consoleauth service is deprecated and
should be removed. However if nova-consoleauth service is located on the
same host with nova-compute, corresponding row in compute_nodes table is
soft-deleted, making nova-compute service report in log:

2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._init_compute_node(context, resources)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update(context, cn)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     inv_data,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(self, *a, **k)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain

Steps to reproduce
==================

# nova service-list
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id                                   | Binary           | Host                   | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:36.904174 | -               | False       |
| 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:40.111038 | -               | False       |
| 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up    | -                          | -               | False       |
| 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute     | vzstor1.vstoragedomain | nova     | enabled | up    | 2019-11-18T12:33:43.500922 | -               | False       |
+--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+

nova=# select uuid,deleted_at from compute_nodes;
                 uuid                 |         deleted_at         
--------------------------------------+----------------------------
 13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |

# nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is
nova-consoleauth service

nova=# select uuid,deleted_at from compute_nodes;
                 uuid                 |         deleted_at         
--------------------------------------+----------------------------
 13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted

Expected result
===============
compute_node is not deleted from db if nova-consoleauth service is removed

Actual result
=============
compute_node is deleted from db if nova-consoleauth service is removed

Environment
===========
Queens, Stein

** Affects: nova
     Importance: Undecided
     Assignee: Pavel Gluschak (scsnow)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Pavel Gluschak (scsnow)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852993

Title:
  Don't delete compute node when deleting service other than nova-
  compute

Status in OpenStack Compute (nova):
  New

Bug description:
  When upgrading to Stein, nova-consoleauth service is deprecated and
  should be removed. However if nova-consoleauth service is located on
  the same host with nova-compute, corresponding row in compute_nodes
  table is soft-deleted, making nova-compute service report in log:

  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update(context, cn)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     inv_data,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(self, *a, **k)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain

  Steps to reproduce
  ==================

  # nova service-list
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | Id                                   | Binary           | Host                   | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:36.904174 | -               | False       |
  | 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:40.111038 | -               | False       |
  | 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up    | -                          | -               | False       |
  | 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute     | vzstor1.vstoragedomain | nova     | enabled | up    | 2019-11-18T12:33:43.500922 | -               | False       |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at         
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |

  # nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is
  nova-consoleauth service

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at         
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted

  Expected result
  ===============
  compute_node is not deleted from db if nova-consoleauth service is removed

  Actual result
  =============
  compute_node is deleted from db if nova-consoleauth service is removed

  Environment
  ===========
  Queens, Stein

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852993/+subscriptions
Follow ups

[Bug 1852993] Re: Don't delete compute node when deleting service other than nova-compute
From: OpenStack Infra, 2019-11-20
[Bug 1852993] Re: Don't delete compute node when deleting service other than nova-compute
From: Matt Riedemann, 2019-11-19