yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1586557] [NEW] Exception raised when trying to boot VM from Horizon UI

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tong Liu <lexuns@xxxxxxxxx>
Date: Fri, 27 May 2016 23:29:56 -0000
Reply-to: Bug 1586557 <1586557@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When tried to boot a VM on Horizon UI, the VM failed to start with
exceptions from both horizon log and n-cpu log.

Horizon log:
2016-05-27 23:27:55.589939 DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://10.116.252.142/identity/users/29bade25d86140bab88869d8337c130e/projects -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f357cbec23bb75ded25e846cd2efdb811a199fad"
2016-05-27 23:27:55.595678 DEBUG:keystoneauth.session:RESP: [404] Date: Fri, 27 May 2016 23:27:55 GMT Server: Apache/2.4.7 (Ubuntu) Vary: X-Auth-Token Content-Length: 93 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
2016-05-27 23:27:55.595715 RESP BODY: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-05-27 23:27:55.595723
2016-05-27 23:27:55.595804 DEBUG:keystoneauth.session:Request returned failure status: 404
2016-05-27 23:27:55.596264 Unable to retrieve project list.
2016-05-27 23:27:55.596287 Traceback (most recent call last):
2016-05-27 23:27:55.596293   File "/usr/local/lib/python2.7/dist-packages/openstack_auth/user.py", line 314, in authorized_tenants
2016-05-27 23:27:55.596298     is_federated=self.is_federated)2016-05-27 23:27:55.596302   File "/usr/local/lib/python2.7/dist-packages/openstack_auth/utils.py", line 325, in get_project_list2016-05-27 23:27:55.596306     projects = client.projects.list(user=kwargs.get('user_id'))2016-05-27 23:27:55.596311   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner2016-05-27 23:27:55.596316     return wrapped(*args, **kwargs)2016-05-27 23:27:55.596320   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/projects.py", line 108, in list2016-05-27 23:27:55.596325     **kwargs)2016-05-27 23:27:55.596328   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 74, in func2016-05-27 23:27:55.596332     return f(*args, **new_kwargs)2016-05-27 23:27:55.596336   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 386, in list2016-05-27 23:27:55.596340     self.collection_key)2016-05-27 23:27:55.596344   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 124, in _list2016-05-27 23:27:55.596349     resp, body = self.client.get(url, **kwargs)2016-05-27 23:27:55.596353   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 178, in get2016-05-27 23:27:55.596357     return self.request(url, 'GET', **kwargs)2016-05-27 23:27:55.596360   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 335, in request2016-05-27 23:27:55.596365     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)2016-05-27 23:27:55.596369   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 103, in request2016-05-27 23:27:55.596373     return self.session.request(url, method, **kwargs)2016-05-27 23:27:55.596379   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner2016-05-27 23:27:55.596383     return wrapped(*args, **kwargs)2016-05-27 23:27:55.596387   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request2016-05-27 23:27:55.596391     raise exceptions.from_response(resp, method, url)2016-05-27 23:27:55.596395 NotFound: The resource could not be found. (HTTP 404)2016-05-27 23:27:55.628082 DEBUG:oslo_policy.policy:Rule [telemetry:compute_statistics] does not exist
2016-05-27 23:27:55.628398 DEBUG:oslo_policy.policy:Rule [default] does not exist
2016-05-27 23:27:55.628769 DEBUG:oslo_policy.policy:Rule [telemetry:get_meter] does not exist


n-cpu log:
n2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:298
2016-05-27 23:27:54.595 INFO nova.virt.block_device [req-bbe0deb8-0f43-4f26-9a5e-8dc4687d3659 demo demo] [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Booting with blank volume at /dev/vda
2016-05-27 23:27:54.596 ERROR nova.compute.manager [req-bbe0deb8-0f43-4f26-9a5e-8dc4687d3659 demo demo] [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Instance failed block device setup
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Traceback (most recent call last):
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/compute/manager.py", line 1577, in _prep_block_device
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     wait_func=self._await_block_device_map_created)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 518, in attach_block_devices
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     map(_log_and_attach, block_device_mapping)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 516, in _log_and_attach
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     bdm.attach(*attach_args, **attach_kwargs)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 406, in attach
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     availability_zone=av_zone)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 187, in wrapper
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     res = method(self, ctx, *args, **kwargs)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 210, in wrapper
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     res = method(self, ctx, volume_id, *args, **kwargs)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 417, in create
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     client = cinderclient(context)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 82, in cinderclient
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     url = _SESSION.get_endpoint(auth, **service_parameters)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 682, in get_endpoint
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     return auth.get_endpoint(self, **kwargs)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/context.py", line 61, in get_endpoint
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     region_name=region_name)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     return wrapped(*args, **kwargs)
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     raise exceptions.EmptyCatalog('The service catalog is empty.')
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] EmptyCatalog: The service catalog is empty.
2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]

** Affects: horizon
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1586557

Title:
  Exception raised when trying to boot VM from Horizon UI

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  When tried to boot a VM on Horizon UI, the VM failed to start with
  exceptions from both horizon log and n-cpu log.

  Horizon log:
  2016-05-27 23:27:55.589939 DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://10.116.252.142/identity/users/29bade25d86140bab88869d8337c130e/projects -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f357cbec23bb75ded25e846cd2efdb811a199fad"
  2016-05-27 23:27:55.595678 DEBUG:keystoneauth.session:RESP: [404] Date: Fri, 27 May 2016 23:27:55 GMT Server: Apache/2.4.7 (Ubuntu) Vary: X-Auth-Token Content-Length: 93 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
  2016-05-27 23:27:55.595715 RESP BODY: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
  2016-05-27 23:27:55.595723
  2016-05-27 23:27:55.595804 DEBUG:keystoneauth.session:Request returned failure status: 404
  2016-05-27 23:27:55.596264 Unable to retrieve project list.
  2016-05-27 23:27:55.596287 Traceback (most recent call last):
  2016-05-27 23:27:55.596293   File "/usr/local/lib/python2.7/dist-packages/openstack_auth/user.py", line 314, in authorized_tenants
  2016-05-27 23:27:55.596298     is_federated=self.is_federated)2016-05-27 23:27:55.596302   File "/usr/local/lib/python2.7/dist-packages/openstack_auth/utils.py", line 325, in get_project_list2016-05-27 23:27:55.596306     projects = client.projects.list(user=kwargs.get('user_id'))2016-05-27 23:27:55.596311   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner2016-05-27 23:27:55.596316     return wrapped(*args, **kwargs)2016-05-27 23:27:55.596320   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/projects.py", line 108, in list2016-05-27 23:27:55.596325     **kwargs)2016-05-27 23:27:55.596328   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 74, in func2016-05-27 23:27:55.596332     return f(*args, **new_kwargs)2016-05-27 23:27:55.596336   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 386, in list2016-05-27 23:27:55.596340     self.collection_key)2016-05-27 23:27:55.596344   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py", line 124, in _list2016-05-27 23:27:55.596349     resp, body = self.client.get(url, **kwargs)2016-05-27 23:27:55.596353   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 178, in get2016-05-27 23:27:55.596357     return self.request(url, 'GET', **kwargs)2016-05-27 23:27:55.596360   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 335, in request2016-05-27 23:27:55.596365     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)2016-05-27 23:27:55.596369   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 103, in request2016-05-27 23:27:55.596373     return self.session.request(url, method, **kwargs)2016-05-27 23:27:55.596379   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner2016-05-27 23:27:55.596383     return wrapped(*args, **kwargs)2016-05-27 23:27:55.596387   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 487, in request2016-05-27 23:27:55.596391     raise exceptions.from_response(resp, method, url)2016-05-27 23:27:55.596395 NotFound: The resource could not be found. (HTTP 404)2016-05-27 23:27:55.628082 DEBUG:oslo_policy.policy:Rule [telemetry:compute_statistics] does not exist
  2016-05-27 23:27:55.628398 DEBUG:oslo_policy.policy:Rule [default] does not exist
  2016-05-27 23:27:55.628769 DEBUG:oslo_policy.policy:Rule [telemetry:get_meter] does not exist

  
  n-cpu log:
  n2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:298
  2016-05-27 23:27:54.595 INFO nova.virt.block_device [req-bbe0deb8-0f43-4f26-9a5e-8dc4687d3659 demo demo] [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Booting with blank volume at /dev/vda
  2016-05-27 23:27:54.596 ERROR nova.compute.manager [req-bbe0deb8-0f43-4f26-9a5e-8dc4687d3659 demo demo] [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Instance failed block device setup
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] Traceback (most recent call last):
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/compute/manager.py", line 1577, in _prep_block_device
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     wait_func=self._await_block_device_map_created)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 518, in attach_block_devices
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     map(_log_and_attach, block_device_mapping)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 516, in _log_and_attach
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     bdm.attach(*attach_args, **attach_kwargs)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/virt/block_device.py", line 406, in attach
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     availability_zone=av_zone)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 187, in wrapper
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     res = method(self, ctx, *args, **kwargs)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 210, in wrapper
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     res = method(self, ctx, volume_id, *args, **kwargs)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 417, in create
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     client = cinderclient(context)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/volume/cinder.py", line 82, in cinderclient
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     url = _SESSION.get_endpoint(auth, **service_parameters)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 682, in get_endpoint
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     return auth.get_endpoint(self, **kwargs)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/opt/stack/nova/nova/context.py", line 61, in get_endpoint
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     region_name=region_name)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     return wrapped(*args, **kwargs)
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]     raise exceptions.EmptyCatalog('The service catalog is empty.')
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e] EmptyCatalog: The service catalog is empty.
  2016-05-27 23:27:54.596 TRACE nova.compute.manager [instance: 19a8a4ee-8bb2-47a0-8232-5a7c77a17a7e]

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1586557/+subscriptions