yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1611447] [NEW] Nova showCellData request returns an "Unexpected API Error"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Anna Babich <ababich@xxxxxxxxxxxx>
Date: Tue, 09 Aug 2016 16:23:31 -0000
Reply-to: Bug 1611447 <1611447@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It has been reproduced against single-node devstack with cells enabled:
ENABLED_SERVICES=c-api,c-bak,c-sch,c-vol,cinder,dstat,g-api,g-reg,horizon,key,mysql,n-api,n-cell,n-cond,n-cpu,n-crt,n-net,n-obj,n-sch,rabbit,s-account,s-container,s-object,s-proxy,tempest

nova.conf:
[DEFAULT]
...
transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
...
[cells]
name = region
cell_type = api
enable = True

nova-cells.conf:
[DEFAULT]
...
transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
...
[cells]
name = child
cell_type = compute
enable = True

The http://developer.openstack.org/api-ref-compute-v2.1.html#listCells
request has been successful and returned:

stack@ubuntu:~$ TOKEN=$(openstack token issue | awk '/ id /{print $4}')
stack@ubuntu:~$ NOVA_ENDPOINT=$(openstack endpoint show nova | awk '/ publicurl /{print $4}')
stack@ubuntu:~$ TENANT_ID=$(openstack project show admin | awk '/ id /{print $4}')
stack@ubuntu:~$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells
HTTP/1.1 200 OK
Content-Length: 117
Content-Type: application/json
Openstack-Api-Version: compute 2.1
X-Openstack-Nova-Api-Version: 2.1
Vary: OpenStack-API-Version
Vary: X-OpenStack-Nova-API-Version
X-Compute-Request-Id: req-9d5ea58a-256f-42d3-bca6-c388e632ca02
Date: Tue, 09 Aug 2016 15:38:31 GMT

{"cells": [{"username": "stackrabbit", "rpc_host": "127.0.0.1", "type":
"child", "name": "child", "rpc_port": 5672}]}

(It's unexpected, that parent cell is absent both here and in 'cells'
table of Nova DB, but can suggest that single-node configuration may
affect it)

After that I tried to send the http://developer.openstack.org/api-ref-compute-v2.1.html#showCellData request. The UUID of cell expected by this request isn't returned by previous one, and the 'uuid' field is absent in 'cells' Nova DB table:
mysql> select * from cells;
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
| created_at          | updated_at | deleted_at | id | api_url | weight_offset | weight_scale | name  | is_parent | deleted | transport_url                                                                                       |
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
| 2016-08-08 15:26:43 | NULL       | NULL       |  1 | NULL    |             0 |            1 | child |         0 |       0 | rabbit://stackrabbit:secretrabbit@127.0.0.1:5672,stackrabbit:secretrabbit@127.0.0.1:5672/child_cell |
+---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

So, let's try to get cell's info by its name, and the result is:
stack@ubuntu:~/devstack$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells/child
HTTP/1.1 500 Internal Server Error
Openstack-Api-Version: compute 2.1
X-Openstack-Nova-Api-Version: 2.1
Vary: OpenStack-API-Versio
Vary: X-OpenStack-Nova-API-Version
Content-Type: application/json; charset=UTF-8
Content-Length: 216
X-Compute-Request-Id: req-05f0f93a-c9ad-403b-ae3f-639b5f64d659
Date: Tue, 09 Aug 2016 15:40:32 GMT

{"computeFault": {"message": "Unexpected API Error. Please report this
at http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>",
"code": 500}}

And there is n-api.log trace:
2016-08-09 11:39:32.215 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Calling method '<bound method CellsController.show of <nova.api.openstack.compute.cells.CellsController object at 0x7f5ad1410f10>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:636
2016-08-09 11:39:32.217 24100 DEBUG oslo_messaging._drivers.amqpdriver [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] CALL msg_id: b34cb4c869444efba1ba76844208cfba exchange 'nova' topic 'cells' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Unexpected exception in API method
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/common.py", line 530, in inner
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return function(*args, **kwargs)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/cells.py", line 164, in show
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     cell = self.cells_rpcapi.cell_get(context, id)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/cells/rpcapi.py", line 492, in cell_get
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     retry=self.retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 96, in _send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     retry=retry)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID b34cb4c869444efba1ba76844208cfba
2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions 
2016-08-09 11:40:32.223 24100 INFO nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2016-08-09 11:40:32.223 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1044
2016-08-09 11:40:32.226 24100 INFO nova.osapi_compute.wsgi.server [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] 127.0.0.1 "GET /v2.1/2bc2e7102ac94b71b260036c5ec13d10/os-cells/child HTTP/1.1" status: 500 len: 559 time: 60.0148242
 
Expected result:
Information about defined cell is returned without errors.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: cells

** Tags added: cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1611447

Title:
  Nova showCellData request returns an "Unexpected API Error"

Status in OpenStack Compute (nova):
  New

Bug description:
  It has been reproduced against single-node devstack with cells enabled:
  ENABLED_SERVICES=c-api,c-bak,c-sch,c-vol,cinder,dstat,g-api,g-reg,horizon,key,mysql,n-api,n-cell,n-cond,n-cpu,n-crt,n-net,n-obj,n-sch,rabbit,s-account,s-container,s-object,s-proxy,tempest

  nova.conf:
  [DEFAULT]
  ...
  transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
  ...
  [cells]
  name = region
  cell_type = api
  enable = True

  nova-cells.conf:
  [DEFAULT]
  ...
  transport_url = rabbit://stackrabbit:secretrabbit@127.0.0.1:5672/
  ...
  [cells]
  name = child
  cell_type = compute
  enable = True

  The http://developer.openstack.org/api-ref-compute-v2.1.html#listCells
  request has been successful and returned:

  stack@ubuntu:~$ TOKEN=$(openstack token issue | awk '/ id /{print $4}')
  stack@ubuntu:~$ NOVA_ENDPOINT=$(openstack endpoint show nova | awk '/ publicurl /{print $4}')
  stack@ubuntu:~$ TENANT_ID=$(openstack project show admin | awk '/ id /{print $4}')
  stack@ubuntu:~$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells
  HTTP/1.1 200 OK
  Content-Length: 117
  Content-Type: application/json
  Openstack-Api-Version: compute 2.1
  X-Openstack-Nova-Api-Version: 2.1
  Vary: OpenStack-API-Version
  Vary: X-OpenStack-Nova-API-Version
  X-Compute-Request-Id: req-9d5ea58a-256f-42d3-bca6-c388e632ca02
  Date: Tue, 09 Aug 2016 15:38:31 GMT

  {"cells": [{"username": "stackrabbit", "rpc_host": "127.0.0.1",
  "type": "child", "name": "child", "rpc_port": 5672}]}

  (It's unexpected, that parent cell is absent both here and in 'cells'
  table of Nova DB, but can suggest that single-node configuration may
  affect it)

  After that I tried to send the http://developer.openstack.org/api-ref-compute-v2.1.html#showCellData request. The UUID of cell expected by this request isn't returned by previous one, and the 'uuid' field is absent in 'cells' Nova DB table:
  mysql> select * from cells;
  +---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
  | created_at          | updated_at | deleted_at | id | api_url | weight_offset | weight_scale | name  | is_parent | deleted | transport_url                                                                                       |
  +---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
  | 2016-08-08 15:26:43 | NULL       | NULL       |  1 | NULL    |             0 |            1 | child |         0 |       0 | rabbit://stackrabbit:secretrabbit@127.0.0.1:5672,stackrabbit:secretrabbit@127.0.0.1:5672/child_cell |
  +---------------------+------------+------------+----+---------+---------------+--------------+-------+-----------+---------+-----------------------------------------------------------------------------------------------------+
  1 row in set (0.00 sec)

  So, let's try to get cell's info by its name, and the result is:
  stack@ubuntu:~/devstack$ curl --include --request GET -H "X-Auth-Token: $TOKEN" -H "Content-Type: application/json" $NOVA_ENDPOINT/$TENANT_ID/os-cells/child
  HTTP/1.1 500 Internal Server Error
  Openstack-Api-Version: compute 2.1
  X-Openstack-Nova-Api-Version: 2.1
  Vary: OpenStack-API-Versio
  Vary: X-OpenStack-Nova-API-Version
  Content-Type: application/json; charset=UTF-8
  Content-Length: 216
  X-Compute-Request-Id: req-05f0f93a-c9ad-403b-ae3f-639b5f64d659
  Date: Tue, 09 Aug 2016 15:40:32 GMT

  {"computeFault": {"message": "Unexpected API Error. Please report this
  at http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>",
  "code": 500}}

  And there is n-api.log trace:
  2016-08-09 11:39:32.215 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Calling method '<bound method CellsController.show of <nova.api.openstack.compute.cells.CellsController object at 0x7f5ad1410f10>>' _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:636
  2016-08-09 11:39:32.217 24100 DEBUG oslo_messaging._drivers.amqpdriver [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] CALL msg_id: b34cb4c869444efba1ba76844208cfba exchange 'nova' topic 'cells' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Unexpected exception in API method
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/common.py", line 530, in inner
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return function(*args, **kwargs)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/cells.py", line 164, in show
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     cell = self.cells_rpcapi.cell_get(context, id)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/cells/rpcapi.py", line 492, in cell_get
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     retry=self.retry)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 96, in _send
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     retry=retry)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID b34cb4c869444efba1ba76844208cfba
  2016-08-09 11:40:32.223 24100 ERROR nova.api.openstack.extensions 
  2016-08-09 11:40:32.223 24100 INFO nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2016-08-09 11:40:32.223 24100 DEBUG nova.api.openstack.wsgi [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1044
  2016-08-09 11:40:32.226 24100 INFO nova.osapi_compute.wsgi.server [req-05f0f93a-c9ad-403b-ae3f-639b5f64d659 admin admin] 127.0.0.1 "GET /v2.1/2bc2e7102ac94b71b260036c5ec13d10/os-cells/child HTTP/1.1" status: 500 len: 559 time: 60.0148242
   
  Expected result:
  Information about defined cell is returned without errors.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1611447/+subscriptions
Follow ups

[Bug 1611447] Re: Nova showCellData request returns an "Unexpected API Error"
From: melanie witt, 2018-04-05