yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1461653] Re: Attaching volume fails if keystone has multiple endpoints of Cinder (juno)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Tue, 17 Nov 2015 13:43:38 -0000
Reply-to: Bug 1461653 <1461653@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Which is a server side configuration, not something that comes from the
client.

** Changed in: nova
       Status: New => Invalid

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1461653

Title:
  Attaching volume fails if keystone has multiple endpoints of Cinder
  (juno)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  I recently deployed my 2nd OpenStack Juno region and as soon as the
  cinder node came up in the 2nd region attaching of volumes stopped
  working in both regions.

  I have tested through horizon & directly on the command line using
  nova & cinder.  I have OS_REGION_NAME specified as an environment
  variable & have the region name properly selected when using horizon.

  OS_AUTH_URL=https://cloudapi-Region1.mydomain.net:5000/v2.0
  OS_PASSWORD=mypassword
  OS_REGION_NAME=Region1
  OS_TENANT_NAME='Test Tenant'
  OS_USERNAME=myuser

  When I try to attach I get the following error:
  nova volume-attach c38ed460-4547-4a7d-b917-7d6c7aafa38e c57544a1-443c-4ee0-a009-19b5e1c3598a
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-caa01caa-cbae-4164-88ce-fa1c2c9834a9)

  And when I check the nova api log I see the following error:
  2015-06-03 18:20:56.634 11151 ERROR nova.api.openstack [req-caa01caa-cbae-4164-88ce-fa1c2c9834a9 None] Caught error: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack Traceback (most recent call last):
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self._app(env, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     content_type, body, accept)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return method(req=request, **action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/volumes.py", line 418, in create
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_id, device)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 221, in wrapped
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 211, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 192, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2947, in attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     disk_bus, device_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2928, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_bdm.destroy(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2921, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 185, in wrapper
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 283, in get
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     version = get_cinder_client_version(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 254, in get_cinder_client_version
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     endpoint_type=endpoint_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 86, in url_for
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     raise cinderclient.exceptions.AmbiguousEndpoints(endpoints=eplist)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]

  I don't know why the REGION_NAME environment or command line (--os-
  region-name) aren't getting passed through to nova or cinder.

  I am running on ubuntu 14.04.2 LTS with that latest updates for OS
  Juno.

  This is similar to bug 1087735 from 2013 but is occurring on Juno.

  ii  nova-api                             1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
  ii  nova-cert                            1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
  ii  nova-common                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-conductor                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
  ii  nova-consoleauth                     1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                      1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
  ii  cinder-api                           1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - API server
  ii  cinder-common                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - common files
  ii  cinder-scheduler                     1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
  ii  python-cinder                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder Python libraries
  ii  python-cinderclient                  1:1.1.0-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1461653/+subscriptions
References

[Bug 1461653] [NEW] Attaching volume fails if keystone has multiple endpoints of Cinder (juno)
From: David M. Zendzian, 2015-06-03