yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1461653] [NEW] Attaching volume fails if keystone has multiple endpoints of Cinder (juno)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "David M. Zendzian" <dmz@xxxxxxxxxxxxx>
Date: Wed, 03 Jun 2015 18:29:32 -0000
Reply-to: Bug 1461653 <1461653@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I recently deployed my 2nd OpenStack Juno region and as soon as the
cinder node came up in the 2nd region attaching of volumes stopped
working in both regions.

I have tested through horizon & directly on the command line using nova
& cinder.  I have OS_REGION_NAME specified as an environment variable &
have the region name properly selected when using horizon.

OS_AUTH_URL=https://cloudapi-Region1.mydomain.net:5000/v2.0
OS_PASSWORD=mypassword
OS_REGION_NAME=Region1
OS_TENANT_NAME='Test Tenant'
OS_USERNAME=myuser

When I try to attach I get the following error:
nova volume-attach c38ed460-4547-4a7d-b917-7d6c7aafa38e c57544a1-443c-4ee0-a009-19b5e1c3598a
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-caa01caa-cbae-4164-88ce-fa1c2c9834a9)

And when I check the nova api log I see the following error:
2015-06-03 18:20:56.634 11151 ERROR nova.api.openstack [req-caa01caa-cbae-4164-88ce-fa1c2c9834a9 None] Caught error: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack Traceback (most recent call last):
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return req.get_response(self.application)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self._app(env, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     content_type, body, accept)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return method(req=request, **action_args)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/volumes.py", line 418, in create
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_id, device)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 221, in wrapped
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 211, in inner
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 192, in inner
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2947, in attach_volume
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     disk_bus, device_type)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2928, in _attach_volume
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_bdm.destroy(context)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2921, in _attach_volume
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 185, in wrapper
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 283, in get
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     version = get_cinder_client_version(context)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 254, in get_cinder_client_version
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     endpoint_type=endpoint_type)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 86, in url_for
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     raise cinderclient.exceptions.AmbiguousEndpoints(endpoints=eplist)
2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]

I don't know why the REGION_NAME environment or command line (--os-
region-name) aren't getting passed through to nova or cinder.

I am running on ubuntu 14.04.2 LTS with that latest updates for OS Juno.

This is similar to bug 1087735 from 2013 but is occurring on Juno.

ii  nova-api                             1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
ii  nova-cert                            1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
ii  nova-common                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - common files
ii  nova-conductor                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
ii  nova-consoleauth                     1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                      1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
ii  python-nova                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
ii  python-novaclient                    1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
ii  cinder-api                           1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - API server
ii  cinder-common                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - common files
ii  cinder-scheduler                     1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
ii  python-cinder                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder Python libraries
ii  python-cinderclient                  1:1.1.0-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  I recently deployed my 2nd OpenStack Juno region and as soon as the
  cinder node came up in the 2nd region attaching of volumes stopped
  working in both regions.
  
  I have tested through horizon & directly on the command line using nova
  & cinder.  I have OS_REGION_NAME specified as an environment variable &
  have the region name properly selected when using horizon.
  
- OS_AUTH_URL=https://cloudapi-wdc001.zzservers.net:5000/v2.0
+ OS_AUTH_URL=https://cloudapi-Region1.mydomain.net:5000/v2.0
  OS_PASSWORD=mypassword
  OS_REGION_NAME=Region1
  OS_TENANT_NAME='Test Tenant'
  OS_USERNAME=myuser
- 
  
  When I try to attach I get the following error:
  nova volume-attach c38ed460-4547-4a7d-b917-7d6c7aafa38e c57544a1-443c-4ee0-a009-19b5e1c3598a
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-caa01caa-cbae-4164-88ce-fa1c2c9834a9)
  
  And when I check the nova api log I see the following error:
  2015-06-03 18:20:56.634 11151 ERROR nova.api.openstack [req-caa01caa-cbae-4164-88ce-fa1c2c9834a9 None] Caught error: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack Traceback (most recent call last):
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self._app(env, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     content_type, body, accept)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return method(req=request, **action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/volumes.py", line 418, in create
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_id, device)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 221, in wrapped
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 211, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 192, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2947, in attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     disk_bus, device_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2928, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_bdm.destroy(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2921, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 185, in wrapper
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 283, in get
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     version = get_cinder_client_version(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 254, in get_cinder_client_version
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     endpoint_type=endpoint_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 86, in url_for
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     raise cinderclient.exceptions.AmbiguousEndpoints(endpoints=eplist)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]
  
- 
- I don't know why the REGION_NAME environment or command line (--os-region-name) aren't getting passed through to nova or cinder.  
+ I don't know why the REGION_NAME environment or command line (--os-
+ region-name) aren't getting passed through to nova or cinder.
  
  I am running on ubuntu 14.04.2 LTS with that latest updates for OS Juno.
  
  This is similar to bug 1087735 from 2013 but is occurring on Juno.
  
  ii  nova-api                             1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
  ii  nova-cert                            1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
  ii  nova-common                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-conductor                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
  ii  nova-consoleauth                     1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                      1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
  ii  cinder-api                           1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - API server
  ii  cinder-common                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - common files
  ii  cinder-scheduler                     1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
  ii  python-cinder                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder Python libraries
  ii  python-cinderclient                  1:1.1.0-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1461653

Title:
  Attaching volume fails if keystone has multiple endpoints of Cinder
  (juno)

Status in OpenStack Compute (Nova):
  New

Bug description:
  I recently deployed my 2nd OpenStack Juno region and as soon as the
  cinder node came up in the 2nd region attaching of volumes stopped
  working in both regions.

  I have tested through horizon & directly on the command line using
  nova & cinder.  I have OS_REGION_NAME specified as an environment
  variable & have the region name properly selected when using horizon.

  OS_AUTH_URL=https://cloudapi-Region1.mydomain.net:5000/v2.0
  OS_PASSWORD=mypassword
  OS_REGION_NAME=Region1
  OS_TENANT_NAME='Test Tenant'
  OS_USERNAME=myuser

  When I try to attach I get the following error:
  nova volume-attach c38ed460-4547-4a7d-b917-7d6c7aafa38e c57544a1-443c-4ee0-a009-19b5e1c3598a
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-caa01caa-cbae-4164-88ce-fa1c2c9834a9)

  And when I check the nova api log I see the following error:
  2015-06-03 18:20:56.634 11151 ERROR nova.api.openstack [req-caa01caa-cbae-4164-88ce-fa1c2c9834a9 None] Caught error: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack Traceback (most recent call last):
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self._app(env, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     content_type, body, accept)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return method(req=request, **action_args)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/volumes.py", line 418, in create
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_id, device)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 221, in wrapped
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 211, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 192, in inner
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2947, in attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     disk_bus, device_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2928, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume_bdm.destroy(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2921, in _attach_volume
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 185, in wrapper
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 283, in get
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     version = get_cinder_client_version(context)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 254, in get_cinder_client_version
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     endpoint_type=endpoint_type)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 86, in url_for
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack     raise cinderclient.exceptions.AmbiguousEndpoints(endpoints=eplist)
  2015-06-03 18:20:56.634 11151 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'adminURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region2', u'id': u'5447a4d47f1e439899b99d01ada74426', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region2.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}, {u'adminURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'region': u'Region1', u'id': u'5544fc1b1b7449af83161c56d4d3dfe9', 'serviceName': u'cinder', u'internalURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63', u'publicURL': u'https://cloudapi-Region1.mydomain.net:8776/v1/f6121817631f4c35bf40c9db1d973e63'}]

  I don't know why the REGION_NAME environment or command line (--os-
  region-name) aren't getting passed through to nova or cinder.

  I am running on ubuntu 14.04.2 LTS with that latest updates for OS
  Juno.

  This is similar to bug 1087735 from 2013 but is occurring on Juno.

  ii  nova-api                             1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
  ii  nova-cert                            1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
  ii  nova-common                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-conductor                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
  ii  nova-consoleauth                     1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                      1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                       1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                          1:2014.2.3-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
  ii  cinder-api                           1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - API server
  ii  cinder-common                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - common files
  ii  cinder-scheduler                     1:2014.2.3-0ubuntu1~cloud0            all          Cinder storage service - Scheduler server
  ii  python-cinder                        1:2014.2.3-0ubuntu1~cloud0            all          Cinder Python libraries
  ii  python-cinderclient                  1:1.1.0-0ubuntu1~cloud0               all          python bindings to the OpenStack Volume API

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1461653/+subscriptions
Follow ups

[Bug 1461653] Re: Attaching volume fails if keystone has multiple endpoints of Cinder (juno)
From: Sean Dague, 2015-11-17
[Bug 1461653] [NEW] Attaching volume fails if keystone has multiple endpoints of Cinder (juno)
From: David M. Zendzian, 2015-06-03
References

[Bug 1461653] [NEW] Attaching volume fails if keystone has multiple endpoints of Cinder (juno)
From: David M. Zendzian, 2015-06-03