← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #52610

[Bug 1594859] [NEW] Any nova command throws HTTP 503

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Any nova command I get HTTP 503 error:

$ nova list
ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-db74de5d-b027-4ac0-b628-3c33cf8f7840

$ nova service-list
ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-819d875e-eb26-405e-baad-76989dcbd843)

$ nova --debug image-list
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')                                                                [54/54]
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:35357/v3 -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
DEBUG (connectionpool:393) "GET /v3 HTTP/1.1" 200 267
DEBUG (session:277) RESP: [200] Content-Length: 267 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5 Connection:
Keep-Alive Date: Tue, 21 Jun 2016 14:54:56 GMT Content-Type: application/json x-openstack-request-id: req-608c0e57-54fa-4a9e-9559-713225d459dc
RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.
6", "links": [{"href": "http://cloud.edpsystem.com:35357/v3/";, "rel": "self"}]}}

DEBUG (base:165) Making authentication request to http://cloud.edpsystem.com:35357/v3/auth/tokens
DEBUG (connectionpool:393) "POST /v3/auth/tokens HTTP/1.1" 201 4961
DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:8774/v2.1/8b19e7039bdc43408b0d621f554ce320 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "
X-Auth-Token: {SHA1}98dba7f25486dedb5fb494a94fe6d61e6a35fc8e"
INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
DEBUG (connectionpool:393) "GET /v2.1/8b19e7039bdc43408b0d621f554ce320 HTTP/1.1" 503 100
DEBUG (session:277) RESP: [503] Date: Tue, 21 Jun 2016 14:54:56 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 100 X-Compute-Request-Id: req-ec9edee7-2e4f
-4ca4-9c2f-549da2fce1bf
RESP BODY: 503 Service Unavailable

The server is currently unavailable. Please try again at a later time.


DEBUG (shell:1082) Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 914, in main
    api_version = api_versions.discover_version(self.cs, api_version)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 267, in discover_version
    client)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 248, in _get_server_version_range
    version = client.versions.get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 84, in get_current
    return self._get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 57, in _get_current
    return self._get(url, "version")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 339, in _get
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)


Same situation in the dashboard, I'm getting " Error: Unable to retrieve
images."


However, if I will issue command using openstack/glance and cinder cli
such as:

$ openstack image list
+--------------------------------------+---------------------+--------+
| ID                                   | Name                | Status |
+--------------------------------------+---------------------+--------+
| 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test | active |
| 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      | active |
| 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              | active |

$ glance image-list
+--------------------------------------+---------------------+
| ID                                   | Name                |
+--------------------------------------+---------------------+
| 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      |
| 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test |
| 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              |
+--------------------------------------+---------------------+

$ cinder list
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+
|                  ID                  |   Status  |   Name   | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+
| 08cf8774-d55f-40f5-ae33-644c2d687ff4 | available | test_vol |  20  |      -      |  false   |             |
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+

$ openstack service list
+----------------------------------+------------+-----------------+
| ID                               | Name       | Type            |
+----------------------------------+------------+-----------------+
| 24e4ad54b6ed4c28966312552ad97ba3 | sahara     | data-processing |
| 2a0430637ad94686bf218e8bd7aaef1c | keystone   | identity        |
| 2e70a62cc9934a9796caef5fd39a0f3a | cinder     | volume          |
| 5a893d3a2c4846b3a896e396b94804ec | glance     | image           |
| 6aff85e0f300430d9a23df6cd72f1846 | ceilometer | metering        |
| 71e910325580411c94b161aaaa8437a3 | heat       | orchestration   |
| 8552da111e0542549812984d41b59a48 | neutron    | network         |
| 92be8decffe24c7997da0ae309e90cf8 | cinderv2   | volumev2        |
| a3759bc1639b4739965fe593341a9590 | trove      | database        |
| afee0c29fa5f4a9cb37f35646502a669 | nova       | compute         |
| c0acf687e0074a25811b697a77655981 | swift      | object-store    |

I'm using Mitaka release on CentOS 7.2

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Any nova command I get HTTP 503 error:
  
  $ nova list
  ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-db74de5d-b027-4ac0-b628-3c33cf8f7840
  
  $ nova service-list
  ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-819d875e-eb26-405e-baad-76989dcbd843)
  
- Same situation in the dashboard, I'm getting " Error: Unable to retrieve
- images. "
+ $ nova --debug image-list
+ DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')                                                                [54/54]
+ DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
+ DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
+ DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:35357/v3 -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
+ INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
+ DEBUG (connectionpool:393) "GET /v3 HTTP/1.1" 200 267
+ DEBUG (session:277) RESP: [200] Content-Length: 267 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5 Connection:
+ Keep-Alive Date: Tue, 21 Jun 2016 14:54:56 GMT Content-Type: application/json x-openstack-request-id: req-608c0e57-54fa-4a9e-9559-713225d459dc
+ RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.
+ 6", "links": [{"href": "http://cloud.edpsystem.com:35357/v3/";, "rel": "self"}]}}
  
- But, if I will issue command using openstack cli such as:
+ DEBUG (base:165) Making authentication request to http://cloud.edpsystem.com:35357/v3/auth/tokens
+ DEBUG (connectionpool:393) "POST /v3/auth/tokens HTTP/1.1" 201 4961
+ DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:8774/v2.1/8b19e7039bdc43408b0d621f554ce320 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "
+ X-Auth-Token: {SHA1}98dba7f25486dedb5fb494a94fe6d61e6a35fc8e"
+ INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
+ DEBUG (connectionpool:393) "GET /v2.1/8b19e7039bdc43408b0d621f554ce320 HTTP/1.1" 503 100
+ DEBUG (session:277) RESP: [503] Date: Tue, 21 Jun 2016 14:54:56 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 100 X-Compute-Request-Id: req-ec9edee7-2e4f
+ -4ca4-9c2f-549da2fce1bf
+ RESP BODY: 503 Service Unavailable
+ 
+ The server is currently unavailable. Please try again at a later time.
+ 
+ 
+ DEBUG (shell:1082) Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
+ Traceback (most recent call last):
+   File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
+     OpenStackComputeShell().main(argv)
+   File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 914, in main
+     api_version = api_versions.discover_version(self.cs, api_version)
+   File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 267, in discover_version
+     client)
+   File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 248, in _get_server_version_range
+     version = client.versions.get_current()
+   File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 84, in get_current
+     return self._get_current()
+   File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 57, in _get_current
+     return self._get(url, "version")
+   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 339, in _get
+     resp, body = self.api.client.get(url)
+   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
+     return self.request(url, 'GET', **kwargs)
+   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
+     raise exceptions.from_response(resp, body, url, method)
+ ClientException: Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
+ ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
+ 
+ 
+ 
+ Same situation in the dashboard, I'm getting " Error: Unable to retrieve images."
+ 
+ 
+ However, if I will issue command using openstack/glance and cinder cli
+ such as:
+ 
  $ openstack image list
  +--------------------------------------+---------------------+--------+
  | ID                                   | Name                | Status |
  +--------------------------------------+---------------------+--------+
  | 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test | active |
  | 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      | active |
  | 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              | active |
  
  $ glance image-list
  +--------------------------------------+---------------------+
  | ID                                   | Name                |
  +--------------------------------------+---------------------+
  | 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      |
  | 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test |
  | 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              |
  +--------------------------------------+---------------------+
  
  $ cinder list
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+
  |                  ID                  |   Status  |   Name   | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+
  | 08cf8774-d55f-40f5-ae33-644c2d687ff4 | available | test_vol |  20  |      -      |  false   |             |
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+
  
  $ openstack service list
  +----------------------------------+------------+-----------------+
  | ID                               | Name       | Type            |
  +----------------------------------+------------+-----------------+
  | 24e4ad54b6ed4c28966312552ad97ba3 | sahara     | data-processing |
  | 2a0430637ad94686bf218e8bd7aaef1c | keystone   | identity        |
  | 2e70a62cc9934a9796caef5fd39a0f3a | cinder     | volume          |
  | 5a893d3a2c4846b3a896e396b94804ec | glance     | image           |
  | 6aff85e0f300430d9a23df6cd72f1846 | ceilometer | metering        |
  | 71e910325580411c94b161aaaa8437a3 | heat       | orchestration   |
  | 8552da111e0542549812984d41b59a48 | neutron    | network         |
  | 92be8decffe24c7997da0ae309e90cf8 | cinderv2   | volumev2        |
  | a3759bc1639b4739965fe593341a9590 | trove      | database        |
  | afee0c29fa5f4a9cb37f35646502a669 | nova       | compute         |
  | c0acf687e0074a25811b697a77655981 | swift      | object-store    |
  
- 
  I'm using Mitaka release on CentOS 7.2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1594859

Title:
  Any nova command throws HTTP 503

Status in OpenStack Compute (nova):
  New

Bug description:
  Any nova command I get HTTP 503 error:

  $ nova list
  ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-db74de5d-b027-4ac0-b628-3c33cf8f7840

  $ nova service-list
  ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-819d875e-eb26-405e-baad-76989dcbd843)

  $ nova --debug image-list
  DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')                                                                [54/54]
  DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
  DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
  DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:35357/v3 -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
  INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
  DEBUG (connectionpool:393) "GET /v3 HTTP/1.1" 200 267
  DEBUG (session:277) RESP: [200] Content-Length: 267 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5 Connection:
  Keep-Alive Date: Tue, 21 Jun 2016 14:54:56 GMT Content-Type: application/json x-openstack-request-id: req-608c0e57-54fa-4a9e-9559-713225d459dc
  RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.
  6", "links": [{"href": "http://cloud.edpsystem.com:35357/v3/";, "rel": "self"}]}}

  DEBUG (base:165) Making authentication request to http://cloud.edpsystem.com:35357/v3/auth/tokens
  DEBUG (connectionpool:393) "POST /v3/auth/tokens HTTP/1.1" 201 4961
  DEBUG (session:248) REQ: curl -g -i -X GET http://cloud.edpsystem.com:8774/v2.1/8b19e7039bdc43408b0d621f554ce320 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "
  X-Auth-Token: {SHA1}98dba7f25486dedb5fb494a94fe6d61e6a35fc8e"
  INFO (connectionpool:213) Starting new HTTP connection (1): cloud.edpsystem.com
  DEBUG (connectionpool:393) "GET /v2.1/8b19e7039bdc43408b0d621f554ce320 HTTP/1.1" 503 100
  DEBUG (session:277) RESP: [503] Date: Tue, 21 Jun 2016 14:54:56 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 100 X-Compute-Request-Id: req-ec9edee7-2e4f
  -4ca4-9c2f-549da2fce1bf
  RESP BODY: 503 Service Unavailable

  The server is currently unavailable. Please try again at a later time.


  DEBUG (shell:1082) Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 914, in main
      api_version = api_versions.discover_version(self.cs, api_version)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 267, in discover_version
      client)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 248, in _get_server_version_range
      version = client.versions.get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 84, in get_current
      return self._get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 57, in _get_current
      return self._get(url, "version")
    File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 339, in _get
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)
  ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-ec9edee7-2e4f-4ca4-9c2f-549da2fce1bf)


  
  Same situation in the dashboard, I'm getting " Error: Unable to retrieve images."


  However, if I will issue command using openstack/glance and cinder cli
  such as:

  $ openstack image list
  +--------------------------------------+---------------------+--------+
  | ID                                   | Name                | Status |
  +--------------------------------------+---------------------+--------+
  | 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test | active |
  | 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      | active |
  | 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              | active |

  $ glance image-list
  +--------------------------------------+---------------------+
  | ID                                   | Name                |
  +--------------------------------------+---------------------+
  | 1d151947-cd60-47e9-a773-19c72054dcb6 | centos7-x86_64      |
  | 680a4b72-0ed0-4b11-9f60-9dd9d0fed033 | centos7-x86_64-test |
  | 0522ee9c-e6c0-449f-bd6e-a8ab7386acfe | cirros              |
  +--------------------------------------+---------------------+

  $ cinder list
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+
  |                  ID                  |   Status  |   Name   | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+
  | 08cf8774-d55f-40f5-ae33-644c2d687ff4 | available | test_vol |  20  |      -      |  false   |             |
  +--------------------------------------+-----------+----------+------+-------------+----------+-------------+

  $ openstack service list
  +----------------------------------+------------+-----------------+
  | ID                               | Name       | Type            |
  +----------------------------------+------------+-----------------+
  | 24e4ad54b6ed4c28966312552ad97ba3 | sahara     | data-processing |
  | 2a0430637ad94686bf218e8bd7aaef1c | keystone   | identity        |
  | 2e70a62cc9934a9796caef5fd39a0f3a | cinder     | volume          |
  | 5a893d3a2c4846b3a896e396b94804ec | glance     | image           |
  | 6aff85e0f300430d9a23df6cd72f1846 | ceilometer | metering        |
  | 71e910325580411c94b161aaaa8437a3 | heat       | orchestration   |
  | 8552da111e0542549812984d41b59a48 | neutron    | network         |
  | 92be8decffe24c7997da0ae309e90cf8 | cinderv2   | volumev2        |
  | a3759bc1639b4739965fe593341a9590 | trove      | database        |
  | afee0c29fa5f4a9cb37f35646502a669 | nova       | compute         |
  | c0acf687e0074a25811b697a77655981 | swift      | object-store    |

  I'm using Mitaka release on CentOS 7.2

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1594859/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next