yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => havana-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1190206

Title:
  Cleanup of deleted instance fails

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  There is a periodic task in the compute manager which identifies and
  (optionally) cleans up instances that are running on the hypervisor
  but set to deleted in the nova database. If the option to destroy
  these instances is enabled (set running_deleted_instance_action to
  ‘reap’ ) the instances are not destroyed and there is an error logged
  (see below).

  This error is raised because the instance data returned from the query
  in nova.compute.manager:_get_instances_on_driver does not include the
  system metadata expected by
  nova.compute.flavors:extract_instance_type. A simple solution is to
  remove the specification of columns_to_join=[] from the calls to
  instance_get_all_by_filters in _get_instances_on_driver as the default
  columns include the expected data.

  Error from compute log:
  2013-06-10 15:39:55.588 25439 INFO nova.compute.manager [-] [instance: 242da1da-cd0c-4c10-b3d7-82aa02c16d04] Destroying instance with name label 'instance-000071d0' which is marked as DELETED but still present on host.
  2013-06-10 15:39:55.589 25439 AUDIT nova.compute.manager [req-e1ab4d31-3cb1-44b7-8214-7bddae7cdbf0 None None] [instance: 242da1da-cd0c-4c10-b3d7-82aa02c16d04] Terminating instance
  2013-06-10 15:39:55.589 25439 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._cleanup_running_deleted_instances: 'instance_type_memory_mb'
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py", line 179, in run_periodic_tasks
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     task(self, context)
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4089, in _cleanup_running_deleted_instances
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     self._shutdown_instance(context, instance, bdms)
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1288, in _shutdown_instance
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     self._notify_about_instance_usage(context, instance, "shutdown.start")
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1211, in _notify_about_instance_usage
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     extra_usage_info=extra_usage_info, host=self.host)
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 246, in notify_about_instance_usage
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     network_info, system_metadata, **extra_usage_info)
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/notifications.py", line 287, in info_from_instance
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     instance_type = flavors.extract_instance_type(instance_ref)
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/flavors.py", line 244, in extract_instance_type
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task     instance_type[key] = type_fn(sys_meta[type_key])
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task KeyError: 'instance_type_memory_mb'
  2013-06-10 15:39:55.589 25439 TRACE nova.openstack.common.periodic_task

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1190206/+subscriptions