← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #11239

[Bug 1289135] [NEW] cinderclient AmbiguousEndpoints in Nova API when deleting nested stack

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

While chasing down some errors I found the first one was the following,
found in the log from the Nova API process.

2014-03-06 22:17:41.713 ERROR nova.api.openstack [req-0a2e7b6b-8ea8-48f1-b6c9-4c6a20ba27b4 admin admin] Caught error: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return req.get_response(self.application)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 596, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return self.app(env, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 925, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     content_type, body, accept)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 987, in _process_stack
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1074, in dispatch
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/volumes.py", line 502, in delete
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 174, in wrapper
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 207, in get
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 92, in cinderclient
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     endpoint_type=endpoint_type)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/python-cinderclient/cinderclient/service_catalog.py", line 83, in url_for
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     endpoints=matching_endpoints)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]

This is reminiscent of bug 1154809 --- which was fixed a year ago.

My scenario is as follows.  I started with a fresh DevStack install
around 04:00 on March 5, 2013 UTC.  Heat Orchestration Template
VM_with_EBS.yaml has three resources: a VM instance, a volume, and an
attachment between the two.  HOT ASRG2.yaml has several resources, one
of which is an OS::Heat::AutoScalingGroup whose element type is a stack
made from VM_with_EBS.yaml.  FYI, an OS::Heat::AutoScalingGroup is
implemented by a nested stack.  Thus, there are two levels of nesting:
VM_with_EBS within the Heat ASG, and the Heat ASG within ASRG2.
Creating all this works fine.  When I POST to the URL for scaling down
the OS::Heat::AutoScalingGroup, that is when the AmbiguousEndpoints
exception happens.

I am attaching VM_with_EBS.yaml with the initial bug report; I will
attach ASRG2.yaml when I figure out how to do so.

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "The Heat Orchestration Template for the innermost stacks"
   https://bugs.launchpad.net/bugs/1289135/+attachment/4010810/+files/VM_with_EBS.yaml

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1289135

Title:
  cinderclient AmbiguousEndpoints in Nova API when deleting nested stack

Status in OpenStack Compute (Nova):
  New

Bug description:
  While chasing down some errors I found the first one was the
  following, found in the log from the Nova API process.

  2014-03-06 22:17:41.713 ERROR nova.api.openstack [req-0a2e7b6b-8ea8-48f1-b6c9-4c6a20ba27b4 admin admin] Caught error: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack Traceback (most recent call last):
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 596, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 925, in __call__
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     content_type, body, accept)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 987, in _process_stack
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1074, in dispatch
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/volumes.py", line 502, in delete
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     volume = self.volume_api.get(context, volume_id)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 174, in wrapper
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     res = method(self, ctx, volume_id, *args, **kwargs)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 207, in get
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     item = cinderclient(context).volumes.get(volume_id)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/nova/nova/volume/cinder.py", line 92, in cinderclient
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     endpoint_type=endpoint_type)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack   File "/opt/stack/python-cinderclient/cinderclient/service_catalog.py", line 83, in url_for
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack     endpoints=matching_endpoints)
  2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]

  This is reminiscent of bug 1154809 --- which was fixed a year ago.

  My scenario is as follows.  I started with a fresh DevStack install
  around 04:00 on March 5, 2013 UTC.  Heat Orchestration Template
  VM_with_EBS.yaml has three resources: a VM instance, a volume, and an
  attachment between the two.  HOT ASRG2.yaml has several resources, one
  of which is an OS::Heat::AutoScalingGroup whose element type is a
  stack made from VM_with_EBS.yaml.  FYI, an OS::Heat::AutoScalingGroup
  is implemented by a nested stack.  Thus, there are two levels of
  nesting: VM_with_EBS within the Heat ASG, and the Heat ASG within
  ASRG2.  Creating all this works fine.  When I POST to the URL for
  scaling down the OS::Heat::AutoScalingGroup, that is when the
  AmbiguousEndpoints exception happens.

  I am attaching VM_with_EBS.yaml with the initial bug report; I will
  attach ASRG2.yaml when I figure out how to do so.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1289135/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next