yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1637368] [NEW] nova-compute got exception cause of residual instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eric Xie <1637368@xxxxxxxxxxxxxxxxxx>
Date: Fri, 28 Oct 2016 02:45:10 -0000
Reply-to: Bug 1637368 <1637368@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Nova-compute got 'KeyError' exception when there is residual instance on the compute node.

Steps to reproduce
==================
* Stop nova-compute service on the compute node
* Then nova delete instance on the controller node
* Then start nova-compute service on the compute node

Expected result
===============
nova-compute service can update its info to DB.

Actual result
=============
nova-compute service got the 'KeyError' exception.

Environment
===========
1. nova version used: stable/mitaka
openstack-nova-scheduler-13.1.0-1.el7.noarch
openstack-nova-compute-13.1.0-1.el7.noarch
openstack-nova-common-13.1.0-1.el7.noarch
openstack-nova-conductor-13.1.0-1.el7.noarch
python-nova-13.1.0-1.el7.noarch
openstack-nova-api-13.1.0-1.el7.noarch
python-novaclient-3.3.1-1.el7.noarch
openstack-nova-console-13.1.0-1.el7.noarch
openstack-nova-novncproxy-13.1.0-1.el7.noarch

2.Libvirt + KVM
libvirt-1.2.17-13.el7_2.5.x86_64
qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64

Logs & Configs
==============
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager Traceback (most recent call last):
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6449, in update_available_resource_for_node
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     rt.update_available_resource(context)
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 508, in update_available_resource
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     resources = self.driver.get_available_resource(self.nodename)
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5407, in get_available_resource
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7070, in _get_disk_over_committed_size_total
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     local_instances[guest.uuid], bdms[guest.uuid])
2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager KeyError: 'ba0fb33a-bc61-4286-8820-5ee6271ff395'

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: mitaka

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1637368

Title:
  nova-compute got exception cause of residual instances

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Nova-compute got 'KeyError' exception when there is residual instance on the compute node.

  Steps to reproduce
  ==================
  * Stop nova-compute service on the compute node
  * Then nova delete instance on the controller node
  * Then start nova-compute service on the compute node

  Expected result
  ===============
  nova-compute service can update its info to DB.

  Actual result
  =============
  nova-compute service got the 'KeyError' exception.

  Environment
  ===========
  1. nova version used: stable/mitaka
  openstack-nova-scheduler-13.1.0-1.el7.noarch
  openstack-nova-compute-13.1.0-1.el7.noarch
  openstack-nova-common-13.1.0-1.el7.noarch
  openstack-nova-conductor-13.1.0-1.el7.noarch
  python-nova-13.1.0-1.el7.noarch
  openstack-nova-api-13.1.0-1.el7.noarch
  python-novaclient-3.3.1-1.el7.noarch
  openstack-nova-console-13.1.0-1.el7.noarch
  openstack-nova-novncproxy-13.1.0-1.el7.noarch

  2.Libvirt + KVM
  libvirt-1.2.17-13.el7_2.5.x86_64
  qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64

  Logs & Configs
  ==============
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager Traceback (most recent call last):
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6449, in update_available_resource_for_node
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     rt.update_available_resource(context)
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 508, in update_available_resource
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     resources = self.driver.get_available_resource(self.nodename)
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5407, in get_available_resource
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7070, in _get_disk_over_committed_size_total
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager     local_instances[guest.uuid], bdms[guest.uuid])
  2016-10-27 16:40:20.936 25310 ERROR nova.compute.manager KeyError: 'ba0fb33a-bc61-4286-8820-5ee6271ff395'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1637368/+subscriptions
Follow ups

[Bug 1637368] Re: nova-compute got exception cause of residual instances
From: Matt Riedemann, 2016-10-31