yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1558808] [NEW] openstack liberty: glance crash on v1 api called from nova

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrea <andrelisci@xxxxxxxxx>
Date: Thu, 17 Mar 2016 21:49:55 -0000
Reply-to: Bug 1558808 <1558808@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

running 'nova image-list' gives the following crash

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-d834d6f8-e2fa-4f00-ac53-97efff2f1da7)

same command called by glance with api v2 works well

glance image-list
+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| 7fbaa9c3-c09b-4ddd-aad5-7b6d51111695 | cirros |
+--------------------------------------+--------+

the problem seems involved in the glance v1 api

I'm running centos 7 with the following nova,glance packages:

python-glanceclient-1.1.0-1.el7.noarch
python-glance-store-0.9.1-1.el7.noarch
openstack-glance-11.0.1-2.el7.noarch
python-glance-11.0.1-2.el7.noarch

openstack-nova-common-12.0.1-1.el7.noarch
openstack-nova-api-12.0.1-1.el7.noarch
python-novaclient-2.30.1-1.el7.noarch
python-nova-12.0.1-1.el7.noarch
openstack-nova-cert-12.0.1-1.el7.noarch
openstack-nova-novncproxy-12.0.1-1.el7.noarch
openstack-nova-scheduler-12.0.1-1.el7.noarch
openstack-nova-conductor-12.0.1-1.el7.noarch
openstack-nova-console-12.0.1-1.el7.noarch

here follows nova logs, glance logs, and the http flow of the error

nova
-----
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions [req-c84a66d7-8801-417e-bbca-5bd46c11182b a96dc90182dd499ebaa3814573336608 1af03692813f48f49dd0130588c53909 - - -] Unexpected exception in API method
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 145, in detail
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     **page_params)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return session.detail(context, **kwargs)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 284, in detail
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     for image in images:
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 254, in list
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     for image in paginate(params, return_request_id):
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 238, in paginate
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     images, resp = self._list(url, "images")
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     resp, body = self.client.get(url)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in get
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return self._request('GET', url, **kwargs)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in _request
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     resp, body_iter = self._handle_response(resp)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 93, in _handle_response
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     raise exc.from_response(resp, resp.content)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions 

glance
--------
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi [req-f63ca99a-8663-4874-8118-cc032b913af1 a96dc90182dd499ebaa3814573336608 1af03692813f48f49dd0130588c53909 - - -] Caught error: [Errno 111] ECONNREFUSED
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi Traceback (most recent call last):
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 879, in __call__
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     request, **action_args)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 907, in dispatch
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return method(*args, **kwargs)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v1/images.py", line 367, in detail
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     images = registry.get_images_detail(req.context, **params)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/api.py", line 155, in get_images_detail
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return c.get_images_detailed(**kwargs)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 150, in get_images_detailed
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     res = self.do_request("GET", "/images/detail", params=params)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 136, in do_request
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     'exc_name': exc_name})
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 121, in do_request
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     **kwargs)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return func(self, *args, **kwargs)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 375, in do_request
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     headers=copy.deepcopy(headers))
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 88, in wrapped
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return func(self, method, url, body, headers)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 540, in _do_request
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     raise exception.ClientConnectionError(e)
2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi ClientConnectionError: [Errno 111] ECONNREFUSED

glance http v1 stream
-------------------------
GET /v1/images/detail?is_public=none&limit=20 HTTP/1.1
Host: 172.16.90.10:9292
Accept-Encoding: gzip, deflate
X-Tenant-Id: 1af03692813f48f49dd0130588c53909
Accept: */*
User-Agent: python-glanceclient
Connection: keep-alive
X-User-Id: a96dc90182dd499ebaa3814573336608
X-Auth-Token: 2e4ae379eead44afad88af328102af8f
X-Identity-Status: Confirmed
Content-Type: application/octet-stream
X-Roles: admin

HTTP/1.1 500 Internal Server Error
Content-Length: 228
Content-Type: text/html; charset=UTF-8
X-Openstack-Request-Id: req-e9fa43b9-b4a5-4462-b3a4-956fcb56a02e
Date: Thu, 17 Mar 2016 16:00:48 GMT
Connection: keep-alive

<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  The server has either erred or is incapable of performing the requested operation.<br /><br />


 </body>
</html>

glance http v2 stream
--------------------------
GET /v2/images?limit=20&sort_key=name&sort_dir=asc HTTP/1.1
Host: 172.16.90.10:9292
Accept-Encoding: gzip, deflate
Accept: */*
X-Auth-Token: 48dc48e173d246ae89757890dfc56657
Connection: keep-alive
User-Agent: python-glanceclient
Content-Type: application/octet-stream

HTTP/1.1 200 OK
Content-Length: 689
Content-Type: application/json; charset=UTF-8
X-Openstack-Request-Id: req-4159a122-4fb0-4495-807e-83f598050409
Date: Thu, 17 Mar 2016 16:00:45 GMT
Connection: keep-alive

{"images": [{"status": "active", "name": "cirros", "tags": [], "container_format": "bare", "created_at": "2016-03-16T17:43:27Z", "size": 13287936, "disk_format": "qcow2", "updated_at": "2016-03-16T17:43:27Z", "visibility": "public", "self": "/v2/images/7fbaa9c3-c09b-4ddd-aad5-7b6d51111695", "min_disk": 0, "protected": false, "id": "7fbaa9c3-c09b-4ddd-aad5-7b6d51111695", "file": "/v2/images/7fbaa9c3-c09b-4ddd-aad5-7b6d51111695/file", "checksum": "ee1eca47dc88f4879d8a229cc70a07c6", "owner": "1af03692813f48f49dd0130588c53909", "virtual_size": null, "min_ram": 0, "schema": "/v2/schemas/image"}], "schema": "/v2/schemas/images", "first": "/v2/images?sort_key=name&sort_dir=asc&limit=20"}GET /v2/schemas/image HTTP/1.1
Host: 172.16.90.10:9292
Accept-Encoding: gzip, deflate
Accept: */*
X-Auth-Token: 48dc48e173d246ae89757890dfc56657
Connection: keep-alive
User-Agent: python-glanceclient
Content-Type: application/octet-stream

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 3902
X-Openstack-Request-Id: req-1459fc0b-d282-4b74-b671-83a7b6374351
Date: Thu, 17 Mar 2016 16:00:45 GMT
Connection: keep-alive

{"additionalProperties": {"type": "string"}, "name": "image", "links":
[{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel":
"enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties":
{"status": {"enum": ["queued", "saving", "active", "killed", "deleted",
"pending_delete", "deactivated"], "type": "string", "description":
"Status of the image (READ-ONLY)"}, "tags": {"items": {"type": "string",
"maxLength": 255}, "type": "array", "description": "List of strings
related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0
-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$",
"type": ["null", "string"], "description": "ID of image stored in Glance
that should be used as the kernel when booting an AMI-style image.",
"is_base": false}, "container_format": {"enum": [null, "ami", "ari",
"aki", "bare", "ovf", "ova"], "type": ["null", "string"], "description":
"Format of the container"}, "min_ram": {"type": "integer",
"description": "Amount of ram (in MB) required to boot image."},
"ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-
fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null",
"string"], "description": "ID of image stored in Glance that should be
used as the ramdisk when booting an AMI-style image.", "is_base":
false}, "locations": {"items": {"required": ["url", "metadata"], "type":
"object", "properties": {"url": {"type": "string", "maxLength": 255},
"metadata": {"type": "object"}}}, "type": "array", "description": "A set
of URLs to access the image file kept in external store"}, "visibility":
{"enum": ["public", "private"], "type": "string", "description": "Scope
of image accessibility"}, "updated_at": {"type": "string",
"description": "Date and time of the last image modification (READ-
ONLY)"}, "owner": {"type": ["null", "string"], "description": "Owner of
the image", "maxLength": 255}, "file": {"type": "string", "description":
"(READ-ONLY)"}, "min_disk": {"type": "integer", "description": "Amount
of disk space (in GB) required to boot image."}, "virtual_size":
{"type": ["null", "integer"], "description": "Virtual size of image in
bytes (READ-ONLY)"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-
fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type":
"string", "description": "An identifier for the image"}, "size":
{"type": ["null", "integer"], "description": "Size of image file in
bytes (READ-ONLY)"}, "instance_uuid": {"type": "string", "description":
"ID of instance used to create this image.", "is_base": false},
"os_distro": {"type": "string", "description": "Common name of operating
system distribution as specified in http://docs.openstack.org/trunk
/openstack-compute/admin/content/adding-images.html", "is_base": false},
"name": {"type": ["null", "string"], "description": "Descriptive name
for the image", "maxLength": 255}, "checksum": {"type": ["null",
"string"], "description": "md5 hash of image contents. (READ-ONLY)",
"maxLength": 32}, "created_at": {"type": "string", "description": "Date
and time of image registration (READ-ONLY)"}, "disk_format": {"enum":
[null, "ami", "ari", "aki", "vhd", "vmdk", "raw", "qcow2", "vdi",
"iso"], "type": ["null", "string"], "description": "Format of the
disk"}, "os_version": {"type": "string", "description": "Operating
system version as specified by the distributor", "is_base": false},
"protected": {"type": "boolean", "description": "If true, image will not
be deletable."}, "architecture": {"type": "string", "description":
"Operating system architecture as specified in
http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-
images.html", "is_base": false}, "direct_url": {"type": "string",
"description": "URL to access the image file kept in external store
(READ-ONLY)"}, "self": {"type": "string", "description": "(READ-ONLY)"},
"schema": {"type": "string", "description": "(READ-ONLY)"}}}

nova http error stream
--------------------------
GET /v2/ HTTP/1.1
Host: 172.16.90.10:8774
Connection: keep-alive
User-Agent: python-novaclient
Accept-Encoding: gzip, deflate
Accept: application/json
X-Auth-Token: 391a552d5d9540718e65f1ca09fc0e50

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 377
X-Compute-Request-Id: req-d60364c4-c1e5-4fbf-8b79-17e2be3452d3
Date: Thu, 17 Mar 2016 16:00:53 GMT
Connection: keep-alive

{"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://172.16.90.10:8774/v2/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}GET /v2/1af03692813f48f49dd0130588c53909/images/detail HTTP/1.1
Host: 172.16.90.10:8774
Connection: keep-alive
User-Agent: python-novaclient
Accept-Encoding: gzip, deflate
Accept: application/json
X-Auth-Token: 391a552d5d9540718e65f1ca09fc0e50

HTTP/1.1 500 Internal Server Error
Content-Length: 214
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-d5bbea30-d67b-40bf-845a-d5f1d5ff3949
Date: Thu, 17 Mar 2016 16:00:54 GMT
Connection: keep-alive

{"computeFault": {"message": "Unexpected API Error. Please report this
at http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible.\n<class 'glanceclient.exc.HTTPInternalServerError'>", "code":
500}}

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1558808

Title:
  openstack liberty: glance crash on v1 api called from nova

Status in Glance:
  New

Bug description:
  running 'nova image-list' gives the following crash

  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-d834d6f8-e2fa-4f00-ac53-97efff2f1da7)

  same command called by glance with api v2 works well

  glance image-list
  +--------------------------------------+--------+
  | ID                                   | Name   |
  +--------------------------------------+--------+
  | 7fbaa9c3-c09b-4ddd-aad5-7b6d51111695 | cirros |
  +--------------------------------------+--------+

  the problem seems involved in the glance v1 api

  I'm running centos 7 with the following nova,glance packages:

  python-glanceclient-1.1.0-1.el7.noarch
  python-glance-store-0.9.1-1.el7.noarch
  openstack-glance-11.0.1-2.el7.noarch
  python-glance-11.0.1-2.el7.noarch

  openstack-nova-common-12.0.1-1.el7.noarch
  openstack-nova-api-12.0.1-1.el7.noarch
  python-novaclient-2.30.1-1.el7.noarch
  python-nova-12.0.1-1.el7.noarch
  openstack-nova-cert-12.0.1-1.el7.noarch
  openstack-nova-novncproxy-12.0.1-1.el7.noarch
  openstack-nova-scheduler-12.0.1-1.el7.noarch
  openstack-nova-conductor-12.0.1-1.el7.noarch
  openstack-nova-console-12.0.1-1.el7.noarch

  here follows nova logs, glance logs, and the http flow of the error

  nova
  -----
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions [req-c84a66d7-8801-417e-bbca-5bd46c11182b a96dc90182dd499ebaa3814573336608 1af03692813f48f49dd0130588c53909 - - -] Unexpected exception in API method
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 145, in detail
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     **page_params)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return session.detail(context, **kwargs)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 284, in detail
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     for image in images:
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 254, in list
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     for image in paginate(params, return_request_id):
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 238, in paginate
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     images, resp = self._list(url, "images")
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     resp, body = self.client.get(url)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in get
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     return self._request('GET', url, **kwargs)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in _request
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     resp, body_iter = self._handle_response(resp)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 93, in _handle_response
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions     raise exc.from_response(resp, resp.content)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
  2016-03-17 16:16:41.509 6316 ERROR nova.api.openstack.extensions 

  glance
  --------
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi [req-f63ca99a-8663-4874-8118-cc032b913af1 a96dc90182dd499ebaa3814573336608 1af03692813f48f49dd0130588c53909 - - -] Caught error: [Errno 111] ECONNREFUSED
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi Traceback (most recent call last):
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 879, in __call__
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     request, **action_args)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 907, in dispatch
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return method(*args, **kwargs)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v1/images.py", line 367, in detail
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     images = registry.get_images_detail(req.context, **params)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/api.py", line 155, in get_images_detail
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return c.get_images_detailed(**kwargs)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 150, in get_images_detailed
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     res = self.do_request("GET", "/images/detail", params=params)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 136, in do_request
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     'exc_name': exc_name})
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 121, in do_request
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     **kwargs)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return func(self, *args, **kwargs)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 375, in do_request
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     headers=copy.deepcopy(headers))
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 88, in wrapped
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     return func(self, method, url, body, headers)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 540, in _do_request
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi     raise exception.ClientConnectionError(e)
  2016-03-17 16:13:26.720 10767 ERROR glance.common.wsgi ClientConnectionError: [Errno 111] ECONNREFUSED

  glance http v1 stream
  -------------------------
  GET /v1/images/detail?is_public=none&limit=20 HTTP/1.1
  Host: 172.16.90.10:9292
  Accept-Encoding: gzip, deflate
  X-Tenant-Id: 1af03692813f48f49dd0130588c53909
  Accept: */*
  User-Agent: python-glanceclient
  Connection: keep-alive
  X-User-Id: a96dc90182dd499ebaa3814573336608
  X-Auth-Token: 2e4ae379eead44afad88af328102af8f
  X-Identity-Status: Confirmed
  Content-Type: application/octet-stream
  X-Roles: admin

  HTTP/1.1 500 Internal Server Error
  Content-Length: 228
  Content-Type: text/html; charset=UTF-8
  X-Openstack-Request-Id: req-e9fa43b9-b4a5-4462-b3a4-956fcb56a02e
  Date: Thu, 17 Mar 2016 16:00:48 GMT
  Connection: keep-alive

  <html>
   <head>
    <title>500 Internal Server Error</title>
   </head>
   <body>
    <h1>500 Internal Server Error</h1>
    The server has either erred or is incapable of performing the requested operation.<br /><br />


   </body>
  </html>

  glance http v2 stream
  --------------------------
  GET /v2/images?limit=20&sort_key=name&sort_dir=asc HTTP/1.1
  Host: 172.16.90.10:9292
  Accept-Encoding: gzip, deflate
  Accept: */*
  X-Auth-Token: 48dc48e173d246ae89757890dfc56657
  Connection: keep-alive
  User-Agent: python-glanceclient
  Content-Type: application/octet-stream

  HTTP/1.1 200 OK
  Content-Length: 689
  Content-Type: application/json; charset=UTF-8
  X-Openstack-Request-Id: req-4159a122-4fb0-4495-807e-83f598050409
  Date: Thu, 17 Mar 2016 16:00:45 GMT
  Connection: keep-alive

  {"images": [{"status": "active", "name": "cirros", "tags": [], "container_format": "bare", "created_at": "2016-03-16T17:43:27Z", "size": 13287936, "disk_format": "qcow2", "updated_at": "2016-03-16T17:43:27Z", "visibility": "public", "self": "/v2/images/7fbaa9c3-c09b-4ddd-aad5-7b6d51111695", "min_disk": 0, "protected": false, "id": "7fbaa9c3-c09b-4ddd-aad5-7b6d51111695", "file": "/v2/images/7fbaa9c3-c09b-4ddd-aad5-7b6d51111695/file", "checksum": "ee1eca47dc88f4879d8a229cc70a07c6", "owner": "1af03692813f48f49dd0130588c53909", "virtual_size": null, "min_ram": 0, "schema": "/v2/schemas/image"}], "schema": "/v2/schemas/images", "first": "/v2/images?sort_key=name&sort_dir=asc&limit=20"}GET /v2/schemas/image HTTP/1.1
  Host: 172.16.90.10:9292
  Accept-Encoding: gzip, deflate
  Accept: */*
  X-Auth-Token: 48dc48e173d246ae89757890dfc56657
  Connection: keep-alive
  User-Agent: python-glanceclient
  Content-Type: application/octet-stream

  HTTP/1.1 200 OK
  Content-Type: application/json; charset=UTF-8
  Content-Length: 3902
  X-Openstack-Request-Id: req-1459fc0b-d282-4b74-b671-83a7b6374351
  Date: Thu, 17 Mar 2016 16:00:45 GMT
  Connection: keep-alive

  {"additionalProperties": {"type": "string"}, "name": "image", "links":
  [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel":
  "enclosure"}, {"href": "{schema}", "rel": "describedby"}],
  "properties": {"status": {"enum": ["queued", "saving", "active",
  "killed", "deleted", "pending_delete", "deactivated"], "type":
  "string", "description": "Status of the image (READ-ONLY)"}, "tags":
  {"items": {"type": "string", "maxLength": 255}, "type": "array",
  "description": "List of strings related to the image"}, "kernel_id":
  {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0
  -9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"],
  "description": "ID of image stored in Glance that should be used as
  the kernel when booting an AMI-style image.", "is_base": false},
  "container_format": {"enum": [null, "ami", "ari", "aki", "bare",
  "ovf", "ova"], "type": ["null", "string"], "description": "Format of
  the container"}, "min_ram": {"type": "integer", "description": "Amount
  of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern":
  "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-
  fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"],
  "description": "ID of image stored in Glance that should be used as
  the ramdisk when booting an AMI-style image.", "is_base": false},
  "locations": {"items": {"required": ["url", "metadata"], "type":
  "object", "properties": {"url": {"type": "string", "maxLength": 255},
  "metadata": {"type": "object"}}}, "type": "array", "description": "A
  set of URLs to access the image file kept in external store"},
  "visibility": {"enum": ["public", "private"], "type": "string",
  "description": "Scope of image accessibility"}, "updated_at": {"type":
  "string", "description": "Date and time of the last image modification
  (READ-ONLY)"}, "owner": {"type": ["null", "string"], "description":
  "Owner of the image", "maxLength": 255}, "file": {"type": "string",
  "description": "(READ-ONLY)"}, "min_disk": {"type": "integer",
  "description": "Amount of disk space (in GB) required to boot
  image."}, "virtual_size": {"type": ["null", "integer"], "description":
  "Virtual size of image in bytes (READ-ONLY)"}, "id": {"pattern": "^([0
  -9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0
  -9a-fA-F]){12}$", "type": "string", "description": "An identifier for
  the image"}, "size": {"type": ["null", "integer"], "description":
  "Size of image file in bytes (READ-ONLY)"}, "instance_uuid": {"type":
  "string", "description": "ID of instance used to create this image.",
  "is_base": false}, "os_distro": {"type": "string", "description":
  "Common name of operating system distribution as specified in
  http://docs.openstack.org/trunk/openstack-compute/admin/content
  /adding-images.html", "is_base": false}, "name": {"type": ["null",
  "string"], "description": "Descriptive name for the image",
  "maxLength": 255}, "checksum": {"type": ["null", "string"],
  "description": "md5 hash of image contents. (READ-ONLY)", "maxLength":
  32}, "created_at": {"type": "string", "description": "Date and time of
  image registration (READ-ONLY)"}, "disk_format": {"enum": [null,
  "ami", "ari", "aki", "vhd", "vmdk", "raw", "qcow2", "vdi", "iso"],
  "type": ["null", "string"], "description": "Format of the disk"},
  "os_version": {"type": "string", "description": "Operating system
  version as specified by the distributor", "is_base": false},
  "protected": {"type": "boolean", "description": "If true, image will
  not be deletable."}, "architecture": {"type": "string", "description":
  "Operating system architecture as specified in
  http://docs.openstack.org/trunk/openstack-compute/admin/content
  /adding-images.html", "is_base": false}, "direct_url": {"type":
  "string", "description": "URL to access the image file kept in
  external store (READ-ONLY)"}, "self": {"type": "string",
  "description": "(READ-ONLY)"}, "schema": {"type": "string",
  "description": "(READ-ONLY)"}}}

  nova http error stream
  --------------------------
  GET /v2/ HTTP/1.1
  Host: 172.16.90.10:8774
  Connection: keep-alive
  User-Agent: python-novaclient
  Accept-Encoding: gzip, deflate
  Accept: application/json
  X-Auth-Token: 391a552d5d9540718e65f1ca09fc0e50

  HTTP/1.1 200 OK
  Content-Type: application/json
  Content-Length: 377
  X-Compute-Request-Id: req-d60364c4-c1e5-4fbf-8b79-17e2be3452d3
  Date: Thu, 17 Mar 2016 16:00:53 GMT
  Connection: keep-alive

  {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://172.16.90.10:8774/v2/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}GET /v2/1af03692813f48f49dd0130588c53909/images/detail HTTP/1.1
  Host: 172.16.90.10:8774
  Connection: keep-alive
  User-Agent: python-novaclient
  Accept-Encoding: gzip, deflate
  Accept: application/json
  X-Auth-Token: 391a552d5d9540718e65f1ca09fc0e50

  HTTP/1.1 500 Internal Server Error
  Content-Length: 214
  Content-Type: application/json; charset=UTF-8
  X-Compute-Request-Id: req-d5bbea30-d67b-40bf-845a-d5f1d5ff3949
  Date: Thu, 17 Mar 2016 16:00:54 GMT
  Connection: keep-alive

  {"computeFault": {"message": "Unexpected API Error. Please report this
  at http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible.\n<class 'glanceclient.exc.HTTPInternalServerError'>",
  "code": 500}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1558808/+subscriptions
Follow ups

[Bug 1558808] Re: openstack liberty: glance crash on v1 api called from nova
From: Andrea, 2016-03-18