← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #59520

[Bug 1646255] Re: removing compute node causes ComputeHostNotFound in nova-api

  Thread Previous • Date Previous • • Thread Next 
Reviewed:  https://review.openstack.org/406627
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f0d44c5b09f3f3c84038d40b621bb629a1f8110e
Submitter: Jenkins
Branch:    master

commit f0d44c5b09f3f3c84038d40b621bb629a1f8110e
Author: Matt Riedemann <mriedem@xxxxxxxxxx>
Date:   Sun Dec 4 15:08:04 2016 -0500

    Handle ComputeHostNotFound when listing hypervisors
    
    Compute node resources must currently be deleted manually
    in the database, and as such they can reference service
    records which have been deleted via the services delete API.
    Because of this when listing hypervisors (compute nodes), we
    may get a ComputeHostNotFound error when trying to lookup a
    service record for a compute node where the service was
    deleted. This causes the API to fail with a 500 since it's not
    handled.
    
    This change handles the ComputeHostNotFound when looping over
    compute nodes in the hypervisors index and detail methods and
    simply ignores them.
    
    Change-Id: I2717274bb1bd370870acbf58c03dc59cee30cc5e
    Closes-Bug: #1646255


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1646255

Title:
  removing compute node causes ComputeHostNotFound in nova-api

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Confirmed

Bug description:
  trying to remove compute node properly

  Steps to reproduce
  ==================
  1) remove all instances from the hypervisor:
  (env) vance@zs95k5:~$ nova hypervisor-servers zs93k23
  +----+------+---------------+---------------------+
  | ID | Name | Hypervisor ID | Hypervisor Hostname |
  +----+------+---------------+---------------------+
  +----+------+---------------+---------------------+

  2) disable the hypervisor:
  (env) vance@zs95k5:~$ nova service-list
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:34.000000 | -               |
  | 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:27.000000 | -               |
  | 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:13:30.000000 | -               |
  | 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:13:28.000000 | -               |
  | 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
  | 17 | nova-compute   | zs93k23             | nova     | enabled  | up    | 2016-11-30T21:13:33.000000 | -               |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  (env) vance@zs95k5:~$ nova service-disable zs93k23 nova-compute
  +---------+--------------+----------+
  | Host    | Binary       | Status   |
  +---------+--------------+----------+
  | zs93k23 | nova-compute | disabled |
  +---------+--------------+----------+

  3) delete the compute service
  (env) vance@zs95k5:~$ nova service-delete 17
  (env) vance@zs95k5:~$ nova service-list
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | Id | Binary         | Host                | Zone     | Status   | State | Updated_at                 | Disabled Reason |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+
  | 3  | nova-cert      | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:54.000000 | -               |
  | 4  | nova-scheduler | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:47.000000 | -               |
  | 5  | nova-conductor | juju-605709-2-lxd-3 | internal | enabled  | up    | 2016-11-30T21:14:56.000000 | -               |
  | 14 | nova-compute   | u27-maas-machine-1  | nova     | disabled | up    | 2016-11-30T21:14:48.000000 | -               |
  | 16 | nova-compute   | zs95k181            | nova     | enabled  | up    | 2016-11-30T21:14:53.000000 | -               |
  +----+----------------+---------------------+----------+----------+-------+----------------------------+-----------------+

  4) delete the neutron agent
  (env) vance@zs95k5:~$ openstack network agent list
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
  | 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
  | 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
  | 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
  | a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
  | aaee3bf0-f8bd-41b7-94ed-f9213f120016 | Open vSwitch agent | zs93k23            | None              | True  | UP    | neutron-openvswitch-agent |
  | c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
  | cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
  | f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  (env) vance@zs95k5:~$ openstack network agent delete aaee3bf0-f8bd-41b7-94ed-f9213f120016
  (env) vance@zs95k5:~$ openstack network agent list
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+
  | 039e4b7a-3dbe-4e87-a9a5-d4b569d3113d | Open vSwitch agent | u27-maas-machine-2 | None              | True  | UP    | neutron-openvswitch-agent |
  | 2aa13570-0e62-4198-96d9-dfe732d7874d | DHCP agent         | u27-maas-machine-2 | nova              | True  | UP    | neutron-dhcp-agent        |
  | 2ab2320a-69cb-4a7f-8ae4-541d3b2bdd3b | L3 agent           | u27-maas-machine-2 | nova              | True  | UP    | neutron-l3-agent          |
  | 48d6d83b-e459-46c8-945c-eea4197c01ec | Open vSwitch agent | zs95k181           | None              | True  | UP    | neutron-openvswitch-agent |
  | a36eecd5-1fb4-436a-becd-fccc737518fd | Metering agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metering-agent    |
  | c6039c81-ad20-4258-a926-bc4a90dccc96 | Loadbalancer agent | u27-maas-machine-2 | None              | True  | UP    | neutron-lbaas-agent       |
  | cfecc66c-2888-4c3d-8241-1d3fcd018a1f | Metadata agent     | u27-maas-machine-2 | None              | True  | UP    | neutron-metadata-agent    |
  | f60cbf28-f030-43ae-a598-0d0182529804 | Open vSwitch agent | u27-maas-machine-1 | None              | True  | UP    | neutron-openvswitch-agent |
  +--------------------------------------+--------------------+--------------------+-------------------+-------+-------+---------------------------+

  5) check hypervisor list
  (env) vance@zs95k5:~$ nova hypervisor-list
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.ComputeHostNotFound'> (HTTP 500) (Request-ID: req-2b773323-be4f-4c8c-be05-a41144ebed76)

  Okay, here is the error from the log:
  http://paste.ubuntu.com/23559892/

  Expected result
  ===============
  Nova compute node is properly removed from the OS database.

  Environment
  ===========
  1. OpenStack nova version 2:13.1.2-0ubuntu2

  2. Hypervisor:
     Libvirt+KVM (KVM for IBM z Systems)

  3. Networking type: OVS+GRE

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1646255/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next