yahoo-eng-team team mailing list archive

[OpenStack Infra <1656075@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/422780
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4dbc0ddc740881a409085feeca2a1771a9ea95f3
Submitter: Jenkins
Branch:    master

commit 4dbc0ddc740881a409085feeca2a1771a9ea95f3
Author: Roman Dobosz <roman.dobosz@xxxxxxxxx>
Date:   Thu Jan 19 18:26:45 2017 +0100

    Prevent compute crash on discovery failure
    
    Keystoneauth1 may crash during discovery of URL for placement API. This is
    often happen on situation, when all services are started at once, compute will
    first be up while keystone still not. This patch fixes this by catching
    DiscoveyFailure as additional expected failures.
    
    Closes-Bug: #1656075
    
    Change-Id: I48aa6802286b408260b9d10f9d13860b44bd7d34


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1656075

Title:
  DiscoveryFailure when trying to get resource providers from the
  scheduler report client

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  New

Bug description:
  I noticed this in a TripleO job:

  http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack-
  integration-4-scenario004-tempest-centos-7/5d95a8c/logs/nova/nova-
  compute.txt.gz#_2017-01-12_18_53_43_459

  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager [req-59098025-7c99-41b2-aaa9-0e5714770b3a - - - - -] Error updating resources for node centos-7-osic-cloud1-s3500-6631948.
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Traceback (most recent call last):
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6537, in update_available_resource_for_node
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 540, in update_available_resource
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _update_available_resource
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 451, in _init_compute_node
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.scheduler_client.update_resource_stats(self.compute_node)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.reportclient.update_resource_stats(compute_node)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 476, in update_resource_stats
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     compute_node.hypervisor_hostname)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 296, in _ensure_resource_provider
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     rp = self._get_resource_provider(uuid)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 53, in wrapper
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return f(self, *a, **k)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 209, in _get_resource_provider
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     resp = self.get("/resource_providers/%s" % uuid)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 174, in get
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     endpoint_filter=self.ks_filter, raise_exc=False)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 710, in get
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.request(url, 'GET', **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return wrapped(*args, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     auth_headers = self.get_auth_headers(auth)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_auth_headers
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return auth.get_headers(self, **kwargs)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     token = self.get_token(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     return self.get_access(session).auth_token
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self.auth_ref = self.get_auth_ref(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     self._plugin = self._do_create_plugin(session)
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
  2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager 

  Looks like another type of keystoneauth1 exception that we're not
  handling in the safe_connect() decorator. At some point we'll probably
  just want to handle all KSA failures globally...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1656075/+subscriptions