yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1656075] [NEW] DiscoveryFailure when trying to get resource providers from the scheduler report client

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 12 Jan 2017 19:15:49 -0000
Reply-to: Bug 1656075 <1656075@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I noticed this in a TripleO job:

http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack-
integration-4-scenario004-tempest-centos-7/5d95a8c/logs/nova/nova-
compute.txt.gz#_2017-01-12_18_53_43_459

2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager [req-59098025-7c99-41b2-aaa9-0e5714770b3a - - - - -] Error updating resources for node centos-7-osic-cloud1-s3500-6631948.
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Traceback (most recent call last):
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6537, in update_available_resource_for_node
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 540, in update_available_resource
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _update_available_resource
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._init_compute_node(context, resources)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 451, in _init_compute_node
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.scheduler_client.update_resource_stats(self.compute_node)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.reportclient.update_resource_stats(compute_node)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 476, in update_resource_stats
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     compute_node.hypervisor_hostname)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 296, in _ensure_resource_provider
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rp = self._get_resource_provider(uuid)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 53, in wrapper
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(self, *a, **k)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 209, in _get_resource_provider
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     resp = self.get("/resource_providers/%s" % uuid)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 174, in get
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     endpoint_filter=self.ks_filter, raise_exc=False)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 710, in get
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return wrapped(*args, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     auth_headers = self.get_auth_headers(auth)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_auth_headers
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return auth.get_headers(self, **kwargs)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     token = self.get_token(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.get_access(session).auth_token
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.auth_ref = self.get_auth_ref(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._plugin = self._do_create_plugin(session)
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager 

Looks like another type of keystoneauth1 exception that we're not
handling in the safe_connect() decorator. At some point we'll probably
just want to handle all KSA failures globally...

** Affects: nova
     Importance: Low
         Status: Confirmed


** Tags: low-hanging-fruit placement

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1656075

Title:
  DiscoveryFailure when trying to get resource providers from the
  scheduler report client

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I noticed this in a TripleO job:

  http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack-
  integration-4-scenario004-tempest-centos-7/5d95a8c/logs/nova/nova-
  compute.txt.gz#_2017-01-12_18_53_43_459

  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager [req-59098025-7c99-41b2-aaa9-0e5714770b3a - - - - -] Error updating resources for node centos-7-osic-cloud1-s3500-6631948.
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Traceback (most recent call last):
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6537, in update_available_resource_for_node
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 540, in update_available_resource
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _update_available_resource
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 451, in _init_compute_node
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.scheduler_client.update_resource_stats(self.compute_node)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.reportclient.update_resource_stats(compute_node)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 476, in update_resource_stats
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     compute_node.hypervisor_hostname)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 296, in _ensure_resource_provider
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rp = self._get_resource_provider(uuid)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 53, in wrapper
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(self, *a, **k)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 209, in _get_resource_provider
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     resp = self.get("/resource_providers/%s" % uuid)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 174, in get
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     endpoint_filter=self.ks_filter, raise_exc=False)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 710, in get
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return wrapped(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     auth_headers = self.get_auth_headers(auth)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_auth_headers
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return auth.get_headers(self, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     token = self.get_token(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.get_access(session).auth_token
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.auth_ref = self.get_auth_ref(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._plugin = self._do_create_plugin(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager 

  Looks like another type of keystoneauth1 exception that we're not
  handling in the safe_connect() decorator. At some point we'll probably
  just want to handle all KSA failures globally...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1656075/+subscriptions
Follow ups

[Bug 1656075] Re: DiscoveryFailure when trying to get resource providers from the scheduler report client
From: OpenStack Infra, 2017-01-28
[Bug 1656075] Re: DiscoveryFailure when trying to get resource providers from the scheduler report client
From: Matt Riedemann, 2017-01-14