yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1361186] [NEW] nova service-delete fails for services on non-child (top) cell

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rajesh Tailor <rajesh.tailor@xxxxxxxxxxx>
Date: Mon, 25 Aug 2014 12:30:33 -0000
Reply-to: Bug 1361186 <1361186@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Nova service-delete fails for services on non-child (top) cell.

How to reproduce:

$ nova --os-username admin service-list

+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:56.000000 | -               |
| region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:06:55.000000 | -               |
| region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
| region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:50.000000 | -               |
| region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
| region@2       | nova-cert        | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:58.000000 | -               |
| region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:57.000000 | -               |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

Stop one of the services on top cell (e.g. nova-cert).

$ nova --os-username admin service-list

+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:26.000000 | -               |
| region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:09:25.000000 | -               |
| region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
| region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:20.000000 | -               |
| region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
| region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
| region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:27.000000 | -               |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

Nova service-delete:
$ nova --os-username admin service-delete 'region@2'

Check the request id from nova-api.log:

2014-08-18 15:10:23.491 INFO nova.osapi_compute.wsgi.server [req-
e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] 192.168.101.31 "DELETE
/v2/d66804d2e78549cd8f5efcedd0abecb2/os-services/region@2 HTTP/1.1"
status: 204 len: 179 time: 0.1334069

Error log in n-cell-region service:

2014-08-18 15:10:23.464 ERROR nova.cells.messaging [req-e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] Error locating next hop for message: 'NoneType' object has no attribute 'count'
2014-08-18 15:10:23.464 TRACE nova.cells.messaging Traceback (most recent call last):
2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 406, in process
2014-08-18 15:10:23.464 TRACE nova.cells.messaging     next_hop = self._get_next_hop()
2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 361, in _get_next_hop
2014-08-18 15:10:23.464 TRACE nova.cells.messaging     dest_hops = target_cell.count(_PATH_CELL_SEP)
2014-08-18 15:10:23.464 TRACE nova.cells.messaging AttributeError: 'NoneType' object has no attribute 'count'


Appendix:
In case of services on child cell, no issues.

$ nova --os-username admin service-list

+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:46.000000 | -               |
| region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | down  | 2014-08-18T06:13:15.000000 | -               |
| region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
| region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:50.000000 | -               |
| region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
| region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
| region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:47.000000 | -               |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

Delete child cell service:
$ nova --os-username admin service-delete 'region!child@2'

$ nova --os-username admin service-list

+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:46.000000 | -               |
| region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
| region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:40.000000 | -               |
| region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
| region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
| region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:47.000000 | -               |
+----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

** Affects: nova
     Importance: Undecided
     Assignee: Rajesh Tailor (rajesh-tailor)
         Status: New


** Tags: ntt

** Changed in: nova
     Assignee: (unassigned) => Rajesh Tailor (rajesh-tailor)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1361186

Title:
  nova service-delete fails for services on non-child (top) cell

Status in OpenStack Compute (Nova):
  New

Bug description:
  Nova service-delete fails for services on non-child (top) cell.

  How to reproduce:

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:56.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:06:55.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:50.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:58.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:57.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Stop one of the services on top cell (e.g. nova-cert).

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:26.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:09:25.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:20.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:27.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Nova service-delete:
  $ nova --os-username admin service-delete 'region@2'

  Check the request id from nova-api.log:

  2014-08-18 15:10:23.491 INFO nova.osapi_compute.wsgi.server [req-
  e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] 192.168.101.31
  "DELETE /v2/d66804d2e78549cd8f5efcedd0abecb2/os-services/region@2
  HTTP/1.1" status: 204 len: 179 time: 0.1334069

  Error log in n-cell-region service:

  2014-08-18 15:10:23.464 ERROR nova.cells.messaging [req-e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] Error locating next hop for message: 'NoneType' object has no attribute 'count'
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging Traceback (most recent call last):
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 406, in process
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging     next_hop = self._get_next_hop()
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 361, in _get_next_hop
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging     dest_hops = target_cell.count(_PATH_CELL_SEP)
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging AttributeError: 'NoneType' object has no attribute 'count'

  
  Appendix:
  In case of services on child cell, no issues.

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:46.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | down  | 2014-08-18T06:13:15.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:50.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:47.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Delete child cell service:
  $ nova --os-username admin service-delete 'region!child@2'

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:46.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:40.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:47.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1361186/+subscriptions
Follow ups

[Bug 1361186] Re: nova service-delete fails for services on non-child (top) cell
From: Alan Pevec, 2015-11-19
[Bug 1361186] Re: nova service-delete fails for services on non-child (top) cell
From: Matt Riedemann, 2015-07-17
[Bug 1361186] Re: nova service-delete fails for services on non-child (top) cell
From: Thierry Carrez, 2015-02-05
[Bug 1361186] [NEW] nova service-delete fails for services on non-child (top) cell
From: Rajesh Tailor, 2014-08-25
References

[Bug 1361186] [NEW] nova service-delete fails for services on non-child (top) cell
From: Rajesh Tailor, 2014-08-25