yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1826382] Re: Updates to placement api fail if placement endpoint changes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Liam Young <1826382@xxxxxxxxxxxxxxxxxx>
Date: Mon, 19 Oct 2020 10:47:39 -0000
Reply-to: Bug 1826382 <1826382@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Work around in the charm committed here:

https://review.opendev.org/#/c/755089
https://review.opendev.org/#/c/755076/

** Also affects: charm-keystone
   Importance: Undecided
       Status: New

** Also affects: charm-nova-cloud-controller
   Importance: Undecided
       Status: New

** Changed in: nova
     Assignee: Liam Young (gnuoy) => (unassigned)

** Changed in: charm-keystone
     Assignee: (unassigned) => Liam Young (gnuoy)

** Changed in: charm-nova-cloud-controller
     Assignee: (unassigned) => Liam Young (gnuoy)

** Changed in: charm-nova-compute
       Status: Triaged => Invalid

** Changed in: charm-keystone
       Status: New => Fix Committed

** Changed in: charm-nova-cloud-controller
       Status: New => Fix Committed

** Changed in: charm-keystone
   Importance: Undecided => High

** Changed in: charm-nova-cloud-controller
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1826382

Title:
  Updates to placement api fail if placement endpoint changes

Status in OpenStack keystone charm:
  Fix Committed
Status in OpenStack nova-cloud-controller charm:
  Fix Committed
Status in OpenStack nova-compute charm:
  Invalid
Status in OpenStack Compute (nova):
  Triaged

Bug description:
  If the url of the placement api changes after nova-compute has been
  started then placement updates fail as nova-compute appears to cache
  the old endpoint url.

  To reproduce, update the placement endpoint to something incorrect in
  keystone and restart nova-compute. Errors contacting the placement api
  will be reported every minute or so. Now, correct the entry in
  keystone. The errors will continue despite the catalogue now being
  correct. Restarting nova-compute fixes the issue.

  In my deployment this occurred when the placement end point switched
  from http to https after the nova-compute node had started. This
  resulted in the following in the nova-compute log:

  2019-04-25 09:58:12.175 31793 ERROR nova.scheduler.client.report [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>400 Bad Request</title>
  </head><body>
  <h1>Bad Request</h1>
  <p>Your browser sent a request that this server could not understand.<br />
  Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
   Instead use the HTTPS scheme to access this URL, please.<br />
  </p>
  <hr>
  <address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address>
  </body></html>
  .
  2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrency.lockutils [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 0.099s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Error updating resources for node juju-7a9f5c-zaza-19a393f3689b-16.project.serverstack.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager Traceback (most recent call last):
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 721, in update_available_resource
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     return f(*args, **kwargs)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 798, in _update_available_resource
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     self._update(context, cn)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     raise value
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 960, in _update
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     self._update_to_placement(context, compute_node)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 912, in _update_to_placement
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     rps_to_refresh = self._get_providers_in_tree(context, uuid)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 74, in wrapper
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     return f(self, *a, **k)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager   File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
  2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-keystone/+bug/1826382/+subscriptions