yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1549852] [NEW] glance image-list is fine, but nova image-list with CEPH backend fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sola <solatour2012@xxxxxxxxx>
Date: Thu, 25 Feb 2016 15:15:47 -0000
Reply-to: Bug 1549852 <1549852@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:


Hi, all 

I've configured HA Openstack Liberty with two controllers and
compute/storage nodes with CEPH backend. OS installed everywhere is
CentOS 7.1

Everything is fine, except one thing - this is output from the nova
image-list command.

Seems that I have the same issue as described here 
https://ask.openstack.org/en/question/86111/glance-image-list-is-fine-but-nova-image-list-error/ and here 
https://ask.openstack.org/en/question/88386/nova-cannot-get-ceph-image-in-liberty/

    [root@vm-controller1 nova]# openstack image list
    +--------------------------------------+--------+
    | ID                                   | Name   |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
    [root@vm-controller1 nova]# glance image-list
    +--------------------------------------+--------+
    | ID                                   | Name   |
    +--------------------------------------+--------+
    | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
    +--------------------------------------+--------+
    [root@vm-controller1 nova]# nova image-list
    ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
    <class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-f0c22157-0d40-4ee7-b0be-4d9bca51f143)

Tried following:

1) On the Controller nodes:
vi /etc/nova/nova.conf

[glance]
api_servers = http://10.7.0.139:9292

2) On the Controller nodes:
vi /etc/glance/glance-api.conf

[DEFAULT]
registry_host = 10.7.0.139

3) On the Controller nodes:
vi /etc/glance/glance-cache.conf

[DEFAULT]
registry_host = 10.7.0.139

4) Double checked all passwords,

5) Changed keystone API from V2 to V3 as described here  -
http://www.cloudkb.net/how-to-change-keystone-api-v2-v3/

Nothing helped and error still persists.
Output from the nova-api.log:

    2016-02-25 17:48:53.265 6274 INFO nova.osapi_compute.wsgi.server [req-f4b90591-3408-40c1-8c3a-8f6ea29f530e b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/ HTTP/1.1" status: 200 len: 583 time: 1.1227880
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] Unexpected exception in API method
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions Traceback (most recent call last):
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 145, in detail
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     **page_params)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return session.detail(context, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 284, in detail
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     for image in images:
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 254, in list
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     for image in paginate(params, return_request_id):
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 238, in paginate
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     images, resp = self._list(url, "images")
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     resp, body = self.client.get(url)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in get
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return self._request('GET', url, **kwargs)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in _request
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     resp, body_iter = self._handle_response(resp)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 93, in _handle_response
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     raise exc.from_response(resp, resp.content)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
    2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions
    2016-02-25 17:48:53.769 6274 INFO nova.api.openstack.wsgi [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
    <class 'glanceclient.exc.HTTPInternalServerError'>
    2016-02-25 17:48:53.771 6274 INFO nova.osapi_compute.wsgi.server [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/292b43fbdc014727a4ec3746f7a7239e/images/detail HTTP/1.1" status: 500 len: 445 time: 0.2422621
    ~
    ~

Please, help, I've run out of any ideas... :(((((((

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1549852

Title:
  glance image-list is fine, but nova image-list with CEPH backend fails

Status in OpenStack Compute (nova):
  New

Bug description:
  
  Hi, all 

  I've configured HA Openstack Liberty with two controllers and
  compute/storage nodes with CEPH backend. OS installed everywhere is
  CentOS 7.1

  Everything is fine, except one thing - this is output from the nova
  image-list command.

  Seems that I have the same issue as described here 
  https://ask.openstack.org/en/question/86111/glance-image-list-is-fine-but-nova-image-list-error/ and here 
  https://ask.openstack.org/en/question/88386/nova-cannot-get-ceph-image-in-liberty/

      [root@vm-controller1 nova]# openstack image list
      +--------------------------------------+--------+
      | ID                                   | Name   |
      +--------------------------------------+--------+
      | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
      +--------------------------------------+--------+
      [root@vm-controller1 nova]# glance image-list
      +--------------------------------------+--------+
      | ID                                   | Name   |
      +--------------------------------------+--------+
      | 365a3e2b-7396-4c6d-a906-e59146ece2d4 | cirros |
      +--------------------------------------+--------+
      [root@vm-controller1 nova]# nova image-list
      ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
      <class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-f0c22157-0d40-4ee7-b0be-4d9bca51f143)

  Tried following:

  1) On the Controller nodes:
  vi /etc/nova/nova.conf

  [glance]
  api_servers = http://10.7.0.139:9292

  2) On the Controller nodes:
  vi /etc/glance/glance-api.conf

  [DEFAULT]
  registry_host = 10.7.0.139

  3) On the Controller nodes:
  vi /etc/glance/glance-cache.conf

  [DEFAULT]
  registry_host = 10.7.0.139

  4) Double checked all passwords,

  5) Changed keystone API from V2 to V3 as described here  -
  http://www.cloudkb.net/how-to-change-keystone-api-v2-v3/

  Nothing helped and error still persists.
  Output from the nova-api.log:

      2016-02-25 17:48:53.265 6274 INFO nova.osapi_compute.wsgi.server [req-f4b90591-3408-40c1-8c3a-8f6ea29f530e b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/ HTTP/1.1" status: 200 len: 583 time: 1.1227880
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] Unexpected exception in API method
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions Traceback (most recent call last):
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 145, in detail
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     **page_params)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return session.detail(context, **kwargs)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 284, in detail
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     for image in images:
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 254, in list
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     for image in paginate(params, return_request_id):
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 238, in paginate
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     images, resp = self._list(url, "images")
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     resp, body = self.client.get(url)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 280, in get
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     return self._request('GET', url, **kwargs)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 272, in _request
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     resp, body_iter = self._handle_response(resp)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 93, in _handle_response
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions     raise exc.from_response(resp, resp.content)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
      2016-02-25 17:48:53.764 6274 ERROR nova.api.openstack.extensions
      2016-02-25 17:48:53.769 6274 INFO nova.api.openstack.wsgi [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
      <class 'glanceclient.exc.HTTPInternalServerError'>
      2016-02-25 17:48:53.771 6274 INFO nova.osapi_compute.wsgi.server [req-f0c22157-0d40-4ee7-b0be-4d9bca51f143 b52d25c57f244c988f04aa6da43dd9f0 292b43fbdc014727a4ec3746f7a7239e - - -] 10.7.0.135 "GET /v2/292b43fbdc014727a4ec3746f7a7239e/images/detail HTTP/1.1" status: 500 len: 445 time: 0.2422621
      ~
      ~

  Please, help, I've run out of any ideas... :(((((((

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1549852/+subscriptions
Follow ups

[Bug 1549852] Re: glance image-list is fine, but nova image-list with CEPH backend fails
From: Sean Dague, 2016-03-10