yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1594230] [NEW] nova hypervisor cannot show ironic node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "watanabe.isao" <watanabe_isao@xxxxxxxxxxxxxx>
Date: Mon, 20 Jun 2016 02:54:02 -0000
Reply-to: Bug 1594230 <1594230@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

[[Environment]]
Have enabled n-api n-cond n-cpu n-crt n-obj n-sch.

[[Since]]
This issue is observed to happen since 2016-06-17 07:31:07 by a 3rd party CI.

[[The issue]]
Other nova command (like flavor-show or list) worked fine.
However, showing detail of an Ironic node hypervisor failed.....

$ nova hypervisor-list
+----+--------------------------------------+-------+---------+
| ID | Hypervisor hostname                  | State | Status  |
+----+--------------------------------------+-------+---------+
| 1  | aad1dade-c627-42b0-b2bf-dd7d9925f1bb | up    | enabled |
+----+--------------------------------------+-------+---------+

$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| aad1dade-c627-42b0-b2bf-dd7d9925f1bb | None | None          | power off   | available          | False       |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+

$ nova hypervisor-show aad1dade-c627-42b0-b2bf-dd7d9925f1bb
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-2eab78b2-ac04-454f-b74b-7e7311eeb6d4)

$ nova hypervisor-show 1
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-898eb920-0547-4178-a44b-ddf9c55fbaec)

[[n-api.log]]
2016-06-20 02:16:20.958 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Calling method '<bound method HypervisorsController.show of <nova.api.openstack.compute.hypervisors.HypervisorsController object at 0x7f18ac0898d0>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:702
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Unexpected exception in API method
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 453, in wrapped
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 119, in show
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp, service, True, req))
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 69, in _view_hypervisor
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp_dict['cpu_info'] = jsonutils.loads(hypervisor.cpu_info)
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 235, in loads
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return json.loads(encodeutils.safe_decode(s, encoding), **kwargs)
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return _default_decoder.decode(s)
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     raise ValueError("No JSON object could be decoded")
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions ValueError: No JSON object could be decoded
2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions
2016-06-20 02:16:20.975 12142 INFO nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'>
2016-06-20 02:16:20.976 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1114
2016-06-20 02:16:20.979 12142 INFO nova.osapi_compute.wsgi.server [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] 127.0.0.1 "GET /v2.1/os-hypervisors/1 HTTP/1.1" status: 500 len: 563 time: 0.0255849

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  [[Environment]]
  Have enabled n-api n-cond n-cpu n-crt n-obj n-sch.
+ 
+ [[Since]]
+ This issue is observed to happen since 2016-06-17 07:31:07 by a 3rd party CI.
  
  [[The issue]]
  Other nova command (like flavor-show or list) worked fine.
  However, showing detail of an Ironic node hypervisor failed.....
  
  $ nova hypervisor-list
  +----+--------------------------------------+-------+---------+
  | ID | Hypervisor hostname                  | State | Status  |
  +----+--------------------------------------+-------+---------+
  | 1  | aad1dade-c627-42b0-b2bf-dd7d9925f1bb | up    | enabled |
  +----+--------------------------------------+-------+---------+
  
  $ ironic node-list
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+
  | UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+
  | aad1dade-c627-42b0-b2bf-dd7d9925f1bb | None | None          | power off   | available          | False       |
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+
  
  $ nova hypervisor-show aad1dade-c627-42b0-b2bf-dd7d9925f1bb
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-2eab78b2-ac04-454f-b74b-7e7311eeb6d4)
  
  $ nova hypervisor-show 1
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-898eb920-0547-4178-a44b-ddf9c55fbaec)
  
  [[n-api.log]]
  2016-06-20 02:16:20.958 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Calling method '<bound method HypervisorsController.show of <nova.api.openstack.compute.hypervisors.HypervisorsController object at 0x7f18ac0898d0>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:702
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Unexpected exception in API method
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 453, in wrapped
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 119, in show
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp, service, True, req))
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 69, in _view_hypervisor
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp_dict['cpu_info'] = jsonutils.loads(hypervisor.cpu_info)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 235, in loads
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return json.loads(encodeutils.safe_decode(s, encoding), **kwargs)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return _default_decoder.decode(s)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     raise ValueError("No JSON object could be decoded")
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions ValueError: No JSON object could be decoded
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions
  2016-06-20 02:16:20.975 12142 INFO nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'>
  2016-06-20 02:16:20.976 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1114
  2016-06-20 02:16:20.979 12142 INFO nova.osapi_compute.wsgi.server [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] 127.0.0.1 "GET /v2.1/os-hypervisors/1 HTTP/1.1" status: 500 len: 563 time: 0.0255849

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1594230

Title:
  nova hypervisor cannot show ironic node

Status in OpenStack Compute (nova):
  New

Bug description:
  [[Environment]]
  Have enabled n-api n-cond n-cpu n-crt n-obj n-sch.

  [[Since]]
  This issue is observed to happen since 2016-06-17 07:31:07 by a 3rd party CI.

  [[The issue]]
  Other nova command (like flavor-show or list) worked fine.
  However, showing detail of an Ironic node hypervisor failed.....

  $ nova hypervisor-list
  +----+--------------------------------------+-------+---------+
  | ID | Hypervisor hostname                  | State | Status  |
  +----+--------------------------------------+-------+---------+
  | 1  | aad1dade-c627-42b0-b2bf-dd7d9925f1bb | up    | enabled |
  +----+--------------------------------------+-------+---------+

  $ ironic node-list
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+
  | UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+
  | aad1dade-c627-42b0-b2bf-dd7d9925f1bb | None | None          | power off   | available          | False       |
  +--------------------------------------+------+---------------+-------------+--------------------+-------------+

  $ nova hypervisor-show aad1dade-c627-42b0-b2bf-dd7d9925f1bb
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-2eab78b2-ac04-454f-b74b-7e7311eeb6d4)

  $ nova hypervisor-show 1
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-898eb920-0547-4178-a44b-ddf9c55fbaec)

  [[n-api.log]]
  2016-06-20 02:16:20.958 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Calling method '<bound method HypervisorsController.show of <nova.api.openstack.compute.hypervisors.HypervisorsController object at 0x7f18ac0898d0>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:702
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Unexpected exception in API method
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 453, in wrapped
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 119, in show
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp, service, True, req))
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/hypervisors.py", line 69, in _view_hypervisor
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     hyp_dict['cpu_info'] = jsonutils.loads(hypervisor.cpu_info)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 235, in loads
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return json.loads(encodeutils.safe_decode(s, encoding), **kwargs)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     return _default_decoder.decode(s)
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions     raise ValueError("No JSON object could be decoded")
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions ValueError: No JSON object could be decoded
  2016-06-20 02:16:20.974 12142 ERROR nova.api.openstack.extensions
  2016-06-20 02:16:20.975 12142 INFO nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'>
  2016-06-20 02:16:20.976 12142 DEBUG nova.api.openstack.wsgi [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.ValueError'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1114
  2016-06-20 02:16:20.979 12142 INFO nova.osapi_compute.wsgi.server [req-898eb920-0547-4178-a44b-ddf9c55fbaec admin admin] 127.0.0.1 "GET /v2.1/os-hypervisors/1 HTTP/1.1" status: 500 len: 563 time: 0.0255849

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1594230/+subscriptions
Follow ups

[Bug 1594230] Re: nova hypervisor cannot show ironic node
From: OpenStack Infra, 2016-08-27