← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #59282

[Bug 1646255] [NEW] removing compute node causes ComputeHostNotFound in nova-api

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

trying to remove compute node properly

Steps to reproduce
==================
1) remove all instances from the hypervisor:
(env) vance@zs95k5:~$ nova hypervisor-servers zs93k23
+----+------+---------------+---------------------+
| ID | Name | Hypervisor ID | Hypervisor Hostname |
+----+------+---------------+---------------------+
+----+------+---------------+---------------------+

2) disable the hypervisor:
(env) vance@zs95k5:~$ nova service-list
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
| Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
| 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:34.000000 | -               |
| 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:27.000000 | -               |
| 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:30.000000 | -               |
| 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:13:28.000000 | -               |
| 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
| 17 | nova-compute   | zs93k23             | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
(env) vance@zs95k5:~$ nova service-disable zs93k23 nova-compute
+---------+--------------+----------+
| Host    | Binary       | Status   |
+---------+--------------+----------+
| zs93k23 | nova-compute | disabled |
+---------+--------------+----------+

3) delete the compute service
(env) vance@zs95k5:~$ nova service-delete 17
(env) vance@zs95k5:~$ nova service-list
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
| Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
| 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:54.000000 | -               |
| 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:47.000000 | -               |
| 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:56.000000 | -               |
| 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:14:48.000000 | -               |
| 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:14:53.000000 | -               |
+----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+

4) delete the neutron agent
(env) vance@zs95k5:~$ openstack network agent list
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
| 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
| 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
| 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
| 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
| a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
| aaee3bf0-f8bd-41b7-94ed-f9213f120016 | Open vSwitch agent | zs93k23            | None              | True  | UP    | neutron-openvswitch-agent |
| c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
| cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
| f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
(env) vance@zs95k5:~$ openstack network agent delete aaee3bf0-f8bd-41b7-94ed-f9213f120016
(env) vance@zs95k5:~$ openstack network agent list
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
| 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
| 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
| 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
| 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
| a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
| c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
| cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
| f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
+--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+

5) check hypervisor list
(env) vance@zs95k5:~$ nova hypervisor-list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.ComputeHostNotFound'> (HTTP 500) (Request-ID: req-2b773323-be4f-4c8c-be05-a41144ebed76)


Okay, here is the error from the log:
http://paste.ubuntu.com/23559892/


Expected result
===============
Nova compute node is properly removed from the OS database.


Environment
===========
1. OpenStack nova version 2:13.1.2-0ubuntu2

2. Hypervisor:
   Libvirt+KVM (KVM for IBM z Systems 1.1.3-beta4.3)

3. Networking type: OVS+GRE

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: api db

** Tags added: api

** Tags added: db

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1646255

Title:
  removing compute node causes ComputeHostNotFound in nova-api

Status in OpenStack Compute (nova):
  New

Bug description:
  trying to remove compute node properly

  Steps to reproduce
  ==================
  1) remove all instances from the hypervisor:
  (env) vance@zs95k5:~$ nova hypervisor-servers zs93k23
  +----+------+---------------+---------------------+
  | ID | Name | Hypervisor ID | Hypervisor Hostname |
  +----+------+---------------+---------------------+
  +----+------+---------------+---------------------+

  2) disable the hypervisor:
  (env) vance@zs95k5:~$ nova service-list
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:34.000000 | -               |
  | 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:27.000000 | -               |
  | 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:30.000000 | -               |
  | 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:13:28.000000 | -               |
  | 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
  | 17 | nova-compute   | zs93k23             | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  (env) vance@zs95k5:~$ nova service-disable zs93k23 nova-compute
  +---------+--------------+----------+
  | Host    | Binary       | Status   |
  +---------+--------------+----------+
  | zs93k23 | nova-compute | disabled |
  +---------+--------------+----------+

  3) delete the compute service
  (env) vance@zs95k5:~$ nova service-delete 17
  (env) vance@zs95k5:~$ nova service-list
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:54.000000 | -               |
  | 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:47.000000 | -               |
  | 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:56.000000 | -               |
  | 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:14:48.000000 | -               |
  | 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:14:53.000000 | -               |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+

  4) delete the neutron agent
  (env) vance@zs95k5:~$ openstack network agent list
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
  | 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
  | 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
  | 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
  | a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
  | aaee3bf0-f8bd-41b7-94ed-f9213f120016 | Open vSwitch agent | zs93k23            | None              | True  | UP    | neutron-openvswitch-agent |
  | c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
  | cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
  | f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  (env) vance@zs95k5:~$ openstack network agent delete aaee3bf0-f8bd-41b7-94ed-f9213f120016
  (env) vance@zs95k5:~$ openstack network agent list
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
  | 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
  | 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
  | 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
  | a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
  | c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
  | cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
  | f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+

  5) check hypervisor list
  (env) vance@zs95k5:~$ nova hypervisor-list
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.ComputeHostNotFound'> (HTTP 500) (Request-ID: req-2b773323-be4f-4c8c-be05-a41144ebed76)

  
  Okay, here is the error from the log:
  http://paste.ubuntu.com/23559892/

  
  Expected result
  ===============
  Nova compute node is properly removed from the OS database.

  
  Environment
  ===========
  1. OpenStack nova version 2:13.1.2-0ubuntu2

  2. Hypervisor:
     Libvirt+KVM (KVM for IBM z Systems 1.1.3-beta4.3)

  3. Networking type: OVS+GRE

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1646255/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next