yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Deleting a nova-consoleauth service shouldn't have anything to do with
deleting compute_nodes records, only deleting nova-compute services, but
I guess this doesn't filter on the service binary being nova-compute:

https://github.com/openstack/nova/blob/a054d03adef692db22e2466084e50cbf50112bb0/nova/db/sqlalchemy/api.py#L415

However, a nova-consoleauth service id shouldn't be mapped to a compute
node record, but that's probably where the OR is breaking things - the
service_id is likely NULL and the host is the same.

** Changed in: nova
   Importance: Undecided => Medium

** Tags added: db

** Also affects: nova/train
   Importance: Undecided
       Status: New

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Changed in: nova/rocky
       Status: New => Confirmed

** Changed in: nova/stein
       Status: New => Confirmed

** Changed in: nova/stein
   Importance: Undecided => Medium

** Changed in: nova/train
   Importance: Undecided => Medium

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova/train
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852993

Title:
  Don't delete compute node when deleting service other than nova-
  compute

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed
Status in OpenStack Compute (nova) train series:
  Confirmed

Bug description:
  When upgrading to Stein, nova-consoleauth service is deprecated and
  should be removed. However if nova-consoleauth service is located on
  the same host with nova-compute, matching row in compute_nodes table
  is soft-deleted as well, making nova-compute service report in log,
  that stale resource provider exists in placement:

  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last):
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     self._update(context, cn)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     inv_data,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 66, in wrapper
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     return f(self, *a, **k)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 614, in _create_resource_provider
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
  2019-11-18 16:03:20.069 7 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain

  Steps to reproduce
  ==================

  # nova service-list
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | Id                                   | Binary           | Host                   | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
  | 57885a23-6bc2-4400-a199-a7e0defe2e00 | nova-conductor   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:36.904174 | -               | False       |
  | 0b08d3ea-1b4a-4076-a51e-fbd892241b9b | nova-scheduler   | vzstor1.vstoragedomain | internal | enabled | up    | 2019-11-18T12:33:40.111038 | -               | False       |
  | 7d7f3dc6-da81-41b4-be30-c2cc451b560a | nova-consoleauth | vzstor1.vstoragedomain | internal | enabled | up    | -                          | -               | False       |
  | 367a4591-cce5-4b7b-ad7c-69135aa803aa | nova-compute     | vzstor1.vstoragedomain | nova     | enabled | up    | 2019-11-18T12:33:43.500922 | -               | False       |
  +--------------------------------------+------------------+------------------------+----------+---------+-------+----------------------------+-----------------+-------------+

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f |

  # nova service-delete 7d7f3dc6-da81-41b4-be30-c2cc451b560a <-- this is
  nova-consoleauth service

  nova=# select uuid,deleted_at from compute_nodes;
                   uuid                 |         deleted_at
  --------------------------------------+----------------------------
   13c1fbd5-fbc1-4301-8a6e-9d50bde6826f | 2019-11-18 12:19:30.080625 <-- compute_node with the same host is also deleted

  Expected result
  ===============
  compute_node is not deleted from db if nova-consoleauth service is removed

  Actual result
  =============
  compute_node is deleted from db if nova-consoleauth service is removed

  Environment
  ===========
  Queens, Stein

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852993/+subscriptions