yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1361186] Re: nova service-delete fails for services on non-child (top) cell

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Fri, 17 Jul 2015 03:16:21 -0000
Reply-to: Bug 1361186 <1361186@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/juno
   Importance: Undecided
       Status: New

** Changed in: nova/juno
       Status: New => In Progress

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1361186

Title:
  nova service-delete fails for services on non-child (top) cell

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) juno series:
  In Progress

Bug description:
  Nova service-delete fails for services on non-child (top) cell.

  How to reproduce:

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:56.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:06:55.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:06:50.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:59.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:58.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:06:57.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Stop one of the services on top cell (e.g. nova-cert).

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:26.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | up    | 2014-08-18T06:09:25.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:09:20.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:19.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:09:27.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Nova service-delete:
  $ nova --os-username admin service-delete 'region@2'

  Check the request id from nova-api.log:

  2014-08-18 15:10:23.491 INFO nova.osapi_compute.wsgi.server [req-
  e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] 192.168.101.31
  "DELETE /v2/d66804d2e78549cd8f5efcedd0abecb2/os-services/region@2
  HTTP/1.1" status: 204 len: 179 time: 0.1334069

  Error log in n-cell-region service:

  2014-08-18 15:10:23.464 ERROR nova.cells.messaging [req-e134d915-ad66-41ba-a6f8-33ec51b7daee admin demo] Error locating next hop for message: 'NoneType' object has no attribute 'count'
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging Traceback (most recent call last):
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 406, in process
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging     next_hop = self._get_next_hop()
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 361, in _get_next_hop
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging     dest_hops = target_cell.count(_PATH_CELL_SEP)
  2014-08-18 15:10:23.464 TRACE nova.cells.messaging AttributeError: 'NoneType' object has no attribute 'count'

  
  Appendix:
  In case of services on child cell, no issues.

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:46.000000 | -               |
  | region!child@2 | nova-compute     | region!child@ubuntu | nova     | enabled | down  | 2014-08-18T06:13:15.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:14:50.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:49.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:14:47.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

  Delete child cell service:
  $ nova --os-username admin service-delete 'region!child@2'

  $ nova --os-username admin service-list

  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | Id             | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
  | region!child@1 | nova-conductor   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:46.000000 | -               |
  | region!child@3 | nova-cells       | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
  | region!child@4 | nova-scheduler   | region!child@ubuntu | internal | enabled | up    | 2014-08-18T06:15:40.000000 | -               |
  | region@1       | nova-cells       | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:39.000000 | -               |
  | region@2       | nova-cert        | region@ubuntu       | internal | enabled | down  | 2014-08-18T06:08:28.000000 | -               |
  | region@3       | nova-consoleauth | region@ubuntu       | internal | enabled | up    | 2014-08-18T06:15:47.000000 | -               |
  +----------------+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1361186/+subscriptions
References

[Bug 1361186] [NEW] nova service-delete fails for services on non-child (top) cell
From: Rajesh Tailor, 2014-08-25