yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1890219] [NEW] nova-compute can not boot cause of old resource provider

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eric Xie <1890219@xxxxxxxxxxxxxxxxxx>
Date: Tue, 04 Aug 2020 00:50:10 -0000
Reply-to: Bug 1890219 <1890219@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
The service 'nova-compute' will register resource provider in placement when it's starting.
But if old one exist with same name, nova-compute serivce can not boot successfully.

Steps to reproduce
==================
* Boot nova-compute with hostname 'host1'
* Create one instance placed on the compute node
* Change hostname to 'host2' and boot nova-compute service
* Rollback hostname to 'host1' and boot nova-compute serivce

Expected result
===============
Service 'nova-compute' booted successfully

Actual result
=============
Got error 'Failed to create resource provider'

Environment
===========
1. nova: stable/rocky
$ git log
commit e3093d42f46af810f316421a9b59eafe94039807 (HEAD -> stable/rocky, origin/stable/rocky)
Author: Luigi Toscano <ltoscano@xxxxxxxxxx>
Date:   Fri Jul 10 13:26:48 2020 +0200

    zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full

    The job was part of the neutron experimental queue but then removed
    during the ussuri lifecycle.
    See https://review.opendev.org/#/c/693630/

    Conflicts:
            .zuul.yaml
    The content of .zuul.yaml changed slightly.

    Change-Id: I04717b95dd44ae89f24bd74525d1c9607e3bc0fc
    (cherry picked from commit bce4a3ab97320bdc2a6a43e2a961a0aa0b8ffb63)
    (cherry picked from commit cf399a363ca530151895c4b7cf49ad7b2a79e01b)
    (cherry picked from commit b1ead1fb2adf25493e5cab472d529fde31f985f0)
    (cherry picked from commit 7b005f37853a56e3ec6da455008fa5ef0d03c21b)

2. Which hypervisor did you use?
libvirt+KVM

Logs & Configs
==============
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager [req-52534aeb-4dd3-4f83-83f1-e6e47e1aa13e - - - - -] Error updating resources for node compute01.: ResourceProviderCreationFailed: Failed to create resource provider compute01
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager Traceback (most recent call last):
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8157, in _update_available_resource_for_node
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 724, in update_available_resource
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 801, in _update_available_resource
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update(context, cn)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 206, in call
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 247, in get
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 200, in call
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 963, in _update
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update_to_placement(context, compute_node)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 915, in _update_to_placement
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 672, in _ensure_resource_provider
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 74, in wrapper
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return f(self, *a, **k)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 606, in _create_resource_provider
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider compute01

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1890219

Title:
  nova-compute can not boot cause of old resource provider

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  The service 'nova-compute' will register resource provider in placement when it's starting.
  But if old one exist with same name, nova-compute serivce can not boot successfully.

  Steps to reproduce
  ==================
  * Boot nova-compute with hostname 'host1'
  * Create one instance placed on the compute node
  * Change hostname to 'host2' and boot nova-compute service
  * Rollback hostname to 'host1' and boot nova-compute serivce

  Expected result
  ===============
  Service 'nova-compute' booted successfully

  Actual result
  =============
  Got error 'Failed to create resource provider'

  Environment
  ===========
  1. nova: stable/rocky
  $ git log
  commit e3093d42f46af810f316421a9b59eafe94039807 (HEAD -> stable/rocky, origin/stable/rocky)
  Author: Luigi Toscano <ltoscano@xxxxxxxxxx>
  Date:   Fri Jul 10 13:26:48 2020 +0200

      zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full

      The job was part of the neutron experimental queue but then removed
      during the ussuri lifecycle.
      See https://review.opendev.org/#/c/693630/

      Conflicts:
              .zuul.yaml
      The content of .zuul.yaml changed slightly.

      Change-Id: I04717b95dd44ae89f24bd74525d1c9607e3bc0fc
      (cherry picked from commit bce4a3ab97320bdc2a6a43e2a961a0aa0b8ffb63)
      (cherry picked from commit cf399a363ca530151895c4b7cf49ad7b2a79e01b)
      (cherry picked from commit b1ead1fb2adf25493e5cab472d529fde31f985f0)
      (cherry picked from commit 7b005f37853a56e3ec6da455008fa5ef0d03c21b)

  2. Which hypervisor did you use?
  libvirt+KVM

  Logs & Configs
  ==============
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager [req-52534aeb-4dd3-4f83-83f1-e6e47e1aa13e - - - - -] Error updating resources for node compute01.: ResourceProviderCreationFailed: Failed to create resource provider compute01
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager Traceback (most recent call last):
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8157, in _update_available_resource_for_node
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 724, in update_available_resource
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return f(*args, **kwargs)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 801, in _update_available_resource
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update(context, cn)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 206, in call
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 247, in get
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/retrying.py", line 200, in call
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 963, in _update
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     self._update_to_placement(context, compute_node)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 915, in _update_to_placement
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 672, in _ensure_resource_provider
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 74, in wrapper
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     return f(self, *a, **k)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 606, in _create_resource_provider
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)
  2020-08-03 08:38:01.296 21734 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider compute01

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1890219/+subscriptions
Follow ups

[Bug 1890219] Re: nova-compute can not boot cause of old resource provider
From: Balazs Gibizer, 2020-08-06